Control system, control method and control procedure

Abstract

The speech recognition unit (513v) receives input information (D51v) from the user (1) who is the source of transmission, and inputs it as input information (D51) to the learning model unit (500). The control decision unit (520) specifies the preconditions for the learning model unit (500) to generate response information (D52) in accordance with the input information (D51v) previously input from the user (1) who is the source of transmission. The speech synthesis unit (514v) receives the response information (D52) generated by the learning model unit (500) in accordance with the preconditions specified by the control decision unit (512) and the input information (D51) input through the speech recognition unit (513v), and outputs it as response information (D52v) to the user (1) who is the source of transmission.

Description

Technical Field

[0001] This invention relates to control systems, control methods, and control programs. Background Technology

[0002] In recent years, the effective use of AI (artificial intelligence) has been continuously advancing. In particular, AI known as generative artificial intelligence, which can generate diverse content, is also becoming more widespread, and the effective uses of AI are expected to broaden. The effective uses of AI are not limited to work in the home, but also include work in various facilities such as buildings, factories, stations, schools, hospitals, and commercial facilities, as well as in various outdoor locations and scenarios such as roads, field facilities, and the air or sea.

[0003] For example, Patent Document 1 describes a machining program generation device that uses a large language model to generate a program for controlling machinery.

[0004] Patent Document 1: Japanese Patent Application Publication No. 2021-060806 Summary of the Invention

[0005] To assist in tasks performed by people or things, the approach is not limited to generating control programs for equipment; it also considers having learning models undertake some or all of the tasks involved in the task. Furthermore, "tasks performed by people or things" includes not only physical tasks performed by people or machines, but also data-space tasks such as information processing performed by processors such as CPUs (central processing units).

[0006] As an example of a task performed through an object, the following tasks are given.

[0007] Operations performed using various equipment such as robots, machinery, devices, and sensors. Operations carried out using various mobile vehicles such as cars, trams, buses, aircraft, and ships. This operation can include, for example, operations referred to as control, processing, manipulation, instruction, calculation, input, output, display, communication, testing, manufacturing, transformation, generation, measurement, irradiation, release, inhalation, heat dissipation, heating, cooling, recording, reading out, shaping, driving, moving, transporting, flying, surveying, monitoring, measuring, extraction, etc.

[0008] In addition, as an example of a task performed by a person, the following tasks are given.

[0009] Human-based work involving humans or other living beings Human operations on various equipment This task can include tasks such as conversation, audiovisual, confirmation, operation, monitoring, instruction, coordination, and explanation.

[0010] Furthermore, the above example is merely an example, and the operation of this invention as an auxiliary object is not limited thereto.

[0011] When using a learning model to enable an information processing device to perform part or all of the tasks involved in an operation performed by a person or object, the appropriateness of the model's output can sometimes become a problem. Additionally, the appropriateness of the model's inputs, which sometimes influence the model's output, can also become a problem.

[0012] Furthermore, depending on the device being controlled, proper control may be impossible if the current situation is not understood. In such cases, situation identification can become problematic. At this point, consider the situation where identification is needed not only for the current situation but also for continuous situations including past situations. For example, when deciding on the next control based on past control actions, the accuracy of situation identification can sometimes be problematic in order to ensure the continuity of control.

[0013] In addition, when the control of the equipment requires immediacy, the response time from giving instructions to the learning model to obtaining the result can sometimes become a problem.

[0014] In addition, the maintainability of the model can sometimes become a problem because the model needs to be relearned every time the equipment is changed or added.

[0015] As mentioned above, various problems still exist when utilizing learning models. Depending on the size of the problem, even if a learning model is specifically used to try to improve the efficiency or performance of a task, it may actually reduce the efficiency or performance of the task.

[0016] In particular, the more complex the task that is set as an auxiliary object, the more significant these problems become when using the learning model. In addition, the more advanced the task that is set as an auxiliary object, the more significant these problems become when using the learning model.

[0017] Therefore, the purpose of this invention is to further improve the efficiency or performance of tasks performed by people or things by utilizing a learning model.

[0018] The control system involved in this invention has the following characteristics: The input interface receives input information from the sending source and inputs it into the learning model unit; A control decision unit, which specifies the preconditions for the learning model unit to generate response information in accordance with the input information previously input from the sending source; and The output interface receives response information generated by the learning model unit according to the preconditions specified by the control decision unit and the input information input through the input interface, and outputs it to the sending source.

[0019] The effects of the invention

[0020] According to the present invention, learning models can be used to further improve the efficiency or performance of tasks performed by humans or objects. In particular, it is possible to improve the efficiency or performance of response tasks that react to information transmissions from a sending source. Attached Figure Description

[0021] Figure 1 This is a structural diagram illustrating an example of the control system involved in Implementation Method 1.

[0022] Figure 2 This is an explanatory diagram showing an example of the structure of the learning model department.

[0023] Figure 3 This is an explanatory diagram showing other structural examples of the learning model department.

[0024] Figure 4 This is an explanatory diagram showing other structural examples of the learning model department.

[0025] Figure 5 This is a structural diagram representing an example of an information processing device that includes the operating environment of the control unit, which contains the learning model unit.

[0026] Figure 6 This is an explanatory diagram illustrating an example of model learning in the model generation department.

[0027] Figure 7 This is a flowchart illustrating an example of the operation of the control system involved in Implementation Method 1.

[0028] Figure 8 This is a structural diagram illustrating other examples of the control system involved in Implementation 1.

[0029] Figure 9 This is a structural diagram illustrating other examples of the control system involved in Implementation 1.

[0030] Figure 10 This is a structural diagram illustrating other examples of the control system involved in Implementation 1.

[0031] Figure 11 This is a structural diagram illustrating an example of the control system involved in Implementation Method 2.

[0032] Figure 12 This is a structural diagram illustrating other examples of the control system involved in Embodiment 2.

[0033] Figure 13 This is a flowchart illustrating an example of the operation of the control system involved in Implementation Method 2.

[0034] Figure 14 This is a structural diagram illustrating an example of the control system involved in Implementation Method 3.

[0035] Figure 15 This is a flowchart illustrating an example of the operation of the control system involved in Implementation Method 3.

[0036] Figure 16 This is a structural diagram illustrating other examples of the control system involved in Implementation 3.

[0037] Figure 17 This is a flowchart illustrating the operation example of a variation of the embodiment 3.

[0038] Figure 18 This is a structural diagram illustrating other examples of the control system involved in Implementation 3.

[0039] Figure 19 This is a structural diagram illustrating other examples of the control system involved in Implementation 3.

[0040] Figure 20 This is a structural diagram illustrating other examples of the control system involved in Implementation 3.

[0041] Figure 21 This is a structural diagram illustrating other examples of the control system involved in Implementation 3.

[0042] Figure 22 This is a flowchart illustrating the operation example of a variation of the embodiment 3.

[0043] Figure 23 This is a structural diagram illustrating an example of the control system involved in Implementation Method 4.

[0044] Figure 24 This is a flowchart illustrating an example of the operation of the control system involved in Implementation Method 4.

[0045] Figure 25 This is a structural diagram illustrating other examples of the control system involved in Implementation 4.

[0046] Figure 26 This is a flowchart illustrating the operation example of a variation of implementation method 4.

[0047] Figure 27This is a structural diagram illustrating an example of the control system involved in Implementation 5.

[0048] Figure 28 This is a flowchart illustrating an example of the operation of the control system involved in Implementation 5.

[0049] Figure 29 This is a structural diagram illustrating other examples of the control system involved in Implementation 5.

[0050] Figure 30 This is an explanatory diagram illustrating an example of the judgment method of the correctness determination unit 515.

[0051] Figure 31 This is a structural diagram illustrating other examples of the control system involved in Implementation 5.

[0052] Figure 32 This is a structural diagram illustrating other examples of the control system involved in Implementation 5.

[0053] Figure 33 This is a structural diagram illustrating other examples of the control system involved in Implementation 5.

[0054] Figure 34 This is a structural diagram illustrating other examples of the control system involved in Implementation 5.

[0055] Figure 35 This is a structural diagram illustrating other examples of the control system involved in Implementation 5.

[0056] Figure 36 This is a structural diagram illustrating other examples of the control system involved in Implementation 5. Detailed Implementation

[0057] Hereinafter, in order to provide a more detailed description of the present invention, embodiments for carrying out the invention will be described with reference to the accompanying drawings. The same reference numerals will be used for the same elements, and descriptions will be omitted.

[0058] Implementation Method 1

[0059] In this embodiment, an example is described that uses a learning model to assist the work involved in code generation for an object device.

[0060] Figure 1 This is a structural diagram illustrating an example of the control system 1000 according to Embodiment 1. Figure 1 The control system 1000 shown is a control system for controlling equipment using a learning model, and includes a learning model unit 100, an equipment information storage unit 110 (denoted as equipment information DB in the figure) and an execution code generation unit 120.

[0061] In addition, Figure 1 The diagram shows user 1 and object device 2, but it can also include them and be configured as control system 1000. In this case, "user 1" can also be referred to as "user terminal 1". The same applies to other embodiments.

[0062] If the learning model unit 100 is given input information D11, it outputs control statement D12. If the learning model unit 100 is given input information D11, it outputs control statement D12 based on the model information D102 described later.

[0063] In this embodiment, the learning model unit 100 is a model and its operating environment configured such that if input information D11 is input, it outputs control statement D12 corresponding to the input information D11. Alternatively, the learning model unit 100 may also be a model and its operating environment configured such that if input information D11 is input, it generates and outputs control statement D12 based on the input information D11, device information D13, and / or other information that the learning model unit 100 can refer to (such as model reference information D104 described later).

[0064] In this embodiment, the input information D11 includes information representing the control content requested from the target device 2. The input information D11 may be, for example, text, images, sounds, or combinations thereof representing the control content for the target device 2. The input information D11 may also be, for example, text, images, sounds, or combinations thereof representing multiple control contents for the target device 2. Furthermore, the input information D11 may also include information representing control content performed continuously in time; in this case, it may be time-series data constructed from predetermined data including text, images, sounds, or combinations thereof representing control content. The representation of the control content is based on matching the input format of the model used by the learning model unit 100, but is not limited to this if the learning model unit 100 includes error processing, correction processing, or transformation processing in its preceding stages.

[0065] As an example of representing the control content in input information D11, the following method is used: after determining the control to be performed on the target device 2, the value of the parameter used to perform the control or the state after the control is executed is specified. In this case, input information D11 may, for example, include information determining the control and information indicating the value of the parameter used to perform the control or the state after the control is executed. The value of the parameter used to perform the control may, for example, include values ​​related to the control type (ON / OFF, etc.), orientation, quantity, and time. Examples of control content include "set function X to ON" for a PLC (programmable logic controller), "move the front end to location A" for a robotic arm, and "lower the set temperature by 1 degree" for an air conditioner. Furthermore, as an example of representing the control content in input information D11, a method using various information such as docstrings describing the specifications of functions, specifications, or specifications, design documents, operation commands, control codes, and source code applicable to other models or other equipment is also provided.

[0066] Furthermore, the representation of control content in input information D11 is not limited to explicit representation as described above. For example, it can also be a method of displaying the operation content when control is implemented through a certain operation, thereby displaying the corresponding control content. Additionally, it can be a method of implicit representation, such as the user 1's words and actions associated with a specific control, the result of the target device 2's actions, or the same control command to other models. In other words, input information D11 can include information indirectly represented not only by directly representing control content for the target device 2, but also by using operation content corresponding to that control content, the user 1's words and actions, or images of the target device 2. As an example, as information showing control content related to the temperature control of an air conditioner, it can use the user 1's words such as "It's hot," actions such as wiping sweat, rolling up sleeves, or fanning themselves—actions indicating the user 1's feeling of heat. In this case, input information D11 can use information such as text, sound, or images representing the user 1's speech, or images (videos) representing the user 1's actions. As other examples, information showing control content related to the arm control of a robotic device can include information specifying the robot's posture after control, the destination of movement of a designated part, imitation actions representing robot actions performed by a person or other object (a simulator that performs the robot's simulated actions, including objects on the screen), or instructions to the robot (action instructions issued through gestures such as fingers).

[0067] Here, the form of input information D11 is not particularly limited. For example, this information can also be data described in a specified design language, control statements (including information described in source code and a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), or execution code. Furthermore, this information can be appropriately combined. In addition, in this invention, unless specifically referred to as "text," besides natural language expressed through text, it can also include data that is mechanically discernible, such as data described in a specified design language that is indistinguishable to humans, control statements (including information described in source code and a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), and execution code.

[0068] Control statement D12 contains control-related information described in a prescribed format that can be determined by the subsequent execution code generation unit 120. Control statement D12 may be, for example, source code described in a prescribed programming language. Alternatively, control statement D12 may also be a set of commands described in a format (platform language) used by a prescribed programming platform. Here, the prescribed programming platform may include no-code programming platforms and low-code programming platforms.

[0069] The device information storage unit 110 stores device information D13, which is related to the target device 2. Device information D13 may include, for example, information representing the function, performance, structure, dimensions, operation, and / or control method of the target device 2. Additionally, device information D13 may also include, for example, information related to the program used in controlling the target device 2. Device information D13 may be, for example, information obtained by digitizing the manual or instruction manual of the target device 2. This digitization includes text digitization, image digitization, digitization via voice reading, and combinations thereof. Device information D13 is used, for example, as supplementary information when the learning model unit 100 outputs control description D12.

[0070] Additionally, device information D13 may also include information indicating the state of object device 2. This information includes not only the current state of object device 2 but also information indicating past states. For example, device information D13 may include timing data constructed from specified data indicating the state of object device 2. The information indicating the state of object device 2 may be information output from object device 2 or information input by user 1 or other devices. This information may include various types of information output from object device 2 (e.g., error messages, log messages, notification messages, etc.). Hereinafter, in this embodiment, the information indicating the state of object device 2 will sometimes be referred to as state information D15.

[0071] If the execution code generation unit 120 receives control statement D12, it generates and outputs executable code D14 based on the control statement D12. Executable code D14 may be, for example, a code set written in machine language. Executable code D14 may contain, for example, information used by the object device 2 when actually performing control. Executable code D14 may simply be control-related information written in a form that the object device 2 can discern. The execution code generation unit 120 may be, for example, a compiler that transforms control statement D12 into executable code D14.

[0072] The executable code D14 output from the executable code generation unit 120 is input to the target device 2. As a result, the target device 2 operates according to the executable code D14 output from the executable code generation unit 120. The input of the executable code D14 to the target device 2 can be directly input from the executable code generation unit 120, or indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or a human hand.

[0073] The object device 2 is not particularly limited. Furthermore, it is assumed that the object device 2 is a device capable of receiving and actually executing the execution code D14, but it is not limited to this if there is an interface between the object device 2 and the object device 2, such as a writing device, that allows the object device 2 to read the execution code.

[0074] The target device 2 can be, for example, a PLC, a processing machine, a robot, radar, a sensor, a camera, a projector, or a communication device. Alternatively, the target device 2 can also be, for example, an air conditioner, a refrigerator, a television set, lighting equipment, or a washing machine. Furthermore, the target device 2 can also be, for example, an elevator, a moving body, a conveyor, other machinery, or a control device for controlling such machinery. Additionally, the target device 2 can also be equipment operating in a power generation-transformation-storage workshop, a water treatment workshop, etc., or a control device for controlling other equipment. Furthermore, if the control description D12 is an interpreter language and the target device 2 is a device capable of receiving and directly executing the control description D12, the execution code generation unit 120 is omitted.

[0075] Figure 2 This is an explanatory diagram showing an example of the structure of the learning model unit 100. For example... Figure 2 As shown, the learning model unit 100 may also include: a model control unit 101, which operates on the information processing device 10; and a model information storage unit 11 (denoted as model information DB in the figure), which stores model information D102. Here, the model information storage unit 11 may also consist of multiple databases connected via a network.

[0076] Model information D102 contains information about the model. For example, model information D102 may include information representing the correlation between the model input data D101 and the model output data D103. Additionally, model information D102 may also include information representing candidates for the model output data D103. Furthermore, model information D102 may also include information representing candidates for the model output data D103 and information representing the relationships between these candidates. Additionally, model information D102 may also include information that specifies the behavior of the learning model, such as constraints, weight variables, and evaluation functions—that is, model parameters.

[0077] The model can be, for example, a machine learning model obtained through teacher-assisted learning, reinforcement learning, or unassisted learning. This model can be obtained by performing learning using deep learning, genetic programs, functional logic programs, or other well-known algorithms and methods. Additionally, the model can be, for example, a model called an NN (Neural Network), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), VAE (Variational Autoencoder), GAN (Generative Adversarial Networks), Diffusion model, Transformer model, LLM (Large Language Model), VLM (Visual Language Model), BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), or CLIP (Contrastive Language Image Pre-training). Furthermore, the model can also be a model described by a rule base, which makes judgments based on pre-defined tables or pre-defined conditions to obtain the output result. Furthermore, the aforementioned models are not exclusive; for example, LLM, VLM, BERT, and GPT are included within the Transformer model. Additionally, the Transformer model is included within the Neural Network (NN) model. Moreover, learning algorithms and models can also be obtained by combining multiple types of data. Models also include those obtained by combining multiple different types of data for learning, known as multimodal models.

[0078] If the model control unit 101 receives model input data D101, it outputs model output data D103 corresponding to the model input data D101 based on the model input data D101 and model information D102. If the model control unit 101 receives model input data D101, it outputs model output data D103 corresponding to the model input data D101, for example, using the model represented by the model information D102.

[0079] The model control unit 101 is implemented, for example, by a CPU or the like that operates according to a program provided by the information processing device 10. Hereinafter, the learning model unit 100 is sometimes referred to as an artificial intelligence unit. Here, an artificial intelligence unit refers to an AI with intelligent functions such as inference and judgment, and its operating environment. Therefore, the model control unit 101 may also include an AI with intelligent functions such as inference and judgment, and its operating environment. The model control unit 101 may, for example, be an AI with a learning model as described above, and its operating environment. The model control unit 101 may also be an element (module) of the control unit 104 provided by the information processing device 10.

[0080] In addition, such as Figure 3 As shown, the learning model unit 100 may also include a reference information storage unit 12 (denoted as reference information DB 12 in the figure) for storing model reference information D104. Here, the reference information storage unit 12 may consist of multiple databases connected via a network. The other storage units described later (e.g., device information storage units, etc.) are the same.

[0081] Model reference information D104 is information used by the model control unit 101 for outputting model output data. Model reference information D104 may include the history of past input model data and / or the history of past output model data. Furthermore, model reference information D104 may include information that associates the features contained in past inputs with the features contained in the outputs made for those inputs. Additionally, model reference information D104 may include information obtained from evaluating the results output for past inputs.

[0082] Additionally, model reference information D104 may also include information related to the representations or concepts contained in model input data D101. For example, model reference information D104 may include information linking a specific representation or concept that can be included in model input data D101 with other representations or concepts related to that representation or concept. Here, other representations or concepts related to a particular representation or concept include representations or concepts that further concretize that representation or concept, and other representations or concepts evoked based on that representation or concept. Model reference information D104 may, for example, include information linking a specific representation or concept that can be included in model input data D101 with other representations or concepts related to that representation or concept. As an example, model reference information D104 may, for example, include information linking a specific representation or concept that can be included in model input data D101 with information related to that representation or concept. Model reference information D104 may, for example, include information linking search keywords and values ​​extracted from the representations or concepts that can be included in model input data D101. Model reference information D104 may also include information for so-called grounding. Additionally, the model reference information D104 can also include a so-called knowledge graph that describes real-world entities and the relationships between them. In a knowledge graph, various kinds of information are systematically linked and represented through graphical construction.

[0083] Additionally, the model reference information D104 may also include information for so-called attention. For example, model reference information D104 may include information representing the correlation between the behaviors or concepts that can be included in the model input data D101 and other behaviors or concepts. Furthermore, model reference information D104 may include a feature map, which uses keyword information extracted from the behaviors or concepts that can be included in the model output data D103, which are associated with the behaviors or concepts that can be included in the model input data D101, as features. Additionally, model reference information D104 may also include information that associates the query content extracted from the behaviors or concepts that can be included in the model input data D101 with the search keyword information corresponding to that query content.

[0084] exist Figure 3 In the example shown, if the model control unit 101 receives model input data D101, it outputs model output data D103 based on the model input data D101, model information D102, and model reference information D104.

[0085] Furthermore, the learning model unit 100 can also replace the reference information storage unit 12 by including a search engine for retrieving the model reference information D104 or an interface to that search engine. In this case, the search scope of the search engine can be an external network or a specific network. Here, as one of the external network or the specific network, a database (e.g., device information DB) possessed by the control system of the present invention can be used.

[0086] The term "learning model" sometimes refers to a computer algorithm that produces a certain output based on information learned from input information, or the information itself that has been learned. However, in an operational environment, "learning model" mostly refers to the actual program that runs the aforementioned computer algorithm and its operating environment. In this invention, the latter is used. To distinguish it from the case of simply learning a set of information based on an algorithm or representation, the model that actually performs actions based on the information stored in model information D102 is called a "learning model." In the control system involved in this invention, a learning model unit (particularly model control unit 101) is provided as an equivalent to the aforementioned learning model. Therefore, in the following description of the control system, "learning model" refers to the learning model unit or, in particular, the model control unit 101.

[0087] Figure 4 This is an explanatory diagram showing other structural examples of the learning model unit 100. For example... Figure 4 As shown, the learning model unit 100 may also have an input unit 102, an output unit 103, and a control unit 104.

[0088] Input unit 102 receives model input data D101. Input unit 102 may also receive model input data D101 input by user 1, etc. Input unit 102 may also receive model input data D101 constituting time-series data. In this case, input unit 102 may receive model input data D101 constituting time-series data sequentially, or it may receive model input data D101 that has been buffered to a certain extent. Furthermore, input unit 102 may receive model input data D101 input from multiple input sources. In this case, input unit 102 may receive model input data D101 accompanied by information about the input source (e.g., user identifier, user attribute information, etc.), or it may determine the input source and receive the model input data D101 after including the input source information, or it may receive it without doing anything. Input unit 102 may be implemented, for example, by various input devices (e.g., pointing devices, keyboards, voice input devices, image input devices, data reading devices, data input devices corresponding to various communication interfaces, etc.) included in information processing device 10. Alternatively, the input unit 102 can be implemented via an external device of the information processing device 10. In this case, the information processing device 10 only needs to include an interface with the input unit 102.

[0089] Output unit 103 outputs the object generated by control unit 104. Here, the object includes model output data D103 or data generated based on the model output data D103. Furthermore, output unit 103 can also output the object to multiple output targets if the object generated by control unit 104 contains information for multiple output targets. In this case, output unit 103 can output the same data to multiple output targets or output different data for each output target. Output unit 103 can be implemented, for example, through various output devices (e.g., display devices, sound output devices, image output devices, data writing devices, data output devices corresponding to various communication interfaces, etc.) included in information processing device 10. Alternatively, output unit 103 can be implemented through an external device of information processing device 10. In this case, information processing device 10 only needs to include an interface with output unit 103.

[0090] The control unit 104 operates on the information processing device 10 and includes a preprocessing unit 105 and a postprocessing unit 106 in addition to the model control unit 101 described above.

[0091] The preprocessing unit 105 performs processing to improve the accuracy of the objects generated by the control unit 104. For example, the preprocessing unit 105 can add, modify, or delete features, or transform (including processing) the model input data D101.

[0092] For example, if the input unit 102 receives model input data D101, the preprocessing unit 105 may modify (including add and delete) features or transform (including process) the data based on the model input data D101. Modification of features or transformation of data (including processing) includes not only changes in data format but also changes in the representation or concept expressed by the data. The modified data, processed by the preprocessing unit 105, is then input to the subsequent model control unit 101 as model input data D101. The processing performed by the preprocessing unit 105 includes what is called "prompt shaping" for the model control unit 101.

[0093] The preprocessing unit 105 can, for example, decompose the model input data D101 into specified unit data. Additionally, the preprocessing unit 105 can, for example, merge multiple model input data D101. Furthermore, the preprocessing unit 105 can perform feature changes or data transformations after decomposing the model input data D101 into specified unit data, or it can perform feature changes or data transformations after merging multiple model input data D101.

[0094] The post-processing unit 106 corrects the object if, for example, a problem exists in the object generated by the control unit 104 (especially the model control unit 101). The post-processing unit 106 can, for example, use the aforementioned knowledge graph to determine if a problem exists in the object. For example, it can compare the similarity between the relationships represented by the knowledge graph, the relationships between the representations or concepts contained in the model input data and the representations or concepts contained in the model output data, and / or the relationships between the representations or concepts contained in the model output data. If the similarity is greater than a predetermined distance from the relationships represented by the knowledge graph, it is determined that the object has a problem.

[0095] Furthermore, the structural elements mentioned above, except for the model control unit 101, are not essential and can be selected appropriately for installation.

[0096] In addition, the model information D102 and other information used by the learning model can be prepared in advance or obtained as needed via a communication network.

[0097] Figure 5 This is a structural diagram showing another example of an information processing device 10, which is an operating environment including a control unit 104 and the like, including a learning model unit 100. Figure 5 The information processing device 10 shown may also have a control unit 104a, which includes a learning model unit 100 (in particular a model control unit 101), an input processing unit 201, an output confirmation unit 202, and a correction confirmation unit 203.

[0098] The input processing unit 201 receives input information D11 from input source 1a such as user 1. Furthermore, the input processing unit 201 outputs the received input information D11 as model input data D101 to the learning model unit 100.

[0099] At this time, the input processing unit 201 can, for example, output data after modifying the elements or transforming the data of the input information D11 as model input data D101. The input processing unit 201 can, for example, remove noise from the input information D11. Furthermore, if the input information D11 contains qualitative information, the input processing unit 201 can transform that information into quantitative information. Additionally, if the input information D11 contains quantitative information, the input processing unit 201 can, for example, correct the quantity according to the device or its operating environment that is the requesting entity of the input information D11. Furthermore, the input processing unit 201 can, for example, perform so-called grounding processing, that is, change the representation or concept expressed by the input information D11 into a further concretized representation or concept.

[0100] Furthermore, the input processing unit 201 can also query the input source when the input information D11 contains ambiguous or uncertain information. The input processing unit 201 can output messages such as confirming the input content, proposing amendments to the input information D11, or requesting re-input of the input information D11 with different statuses or expressions as queries. Additionally, amendments to the input information D11 can be generated by the correction confirmation unit 203, which will be described later. Hereinafter, information regarding the correction, addition, or cancellation of the input-output data displayed on the learning model after input-output is sometimes referred to as supplementary information D18. An amendment is an example of supplementary information D18.

[0101] The output confirmation unit 202 performs a simulation of the control and state of the target device 2 based on the model output data D103 output from the learning model unit 100. The output confirmation unit 202 can also perform the simulation after transforming the model output data D103 into control information that matches a specified simulator (not shown) capable of simulating the control and state of the target device 2. The output confirmation unit 202 may also have simulator functionality. The output confirmation unit 202 may also utilize information obtained from the output target 2a of the model output data D103 during simulation. Here, the output target 2a includes output targets generated based on the information from the model output data D103. The information obtained from the output target 2a may include, for example, the state information D15 and / or feedback information D16, described later.

[0102] The output verification unit 202 can verify, for example, the state of the object device 2, the state of the system containing the object device 2, and / or the state of the workpieces possessed by the object device 2. Furthermore, before performing action verification, the output verification unit 202 can generate and display human-understandable intermediate products based on the model output data D103 or the information generated therefrom. Examples of intermediate products include source code for the control program and operation images of the controller of the object device 2 for operation commands to the object device 2. Additionally, the output verification unit 202 can also display the simulation results along with the reliability indicators of the learned model.

[0103] Here are examples of reliability metrics for learning models. For instance, an evaluation network could be included. This network accumulates evaluations of human inputs whenever they are fed into the learning model, learning from both the inputs and the evaluations. When using the learning model, the inputs to the learning model are also fed into this evaluation network, and its output is used as the reliability metric.

[0104] Alternatively, a learner can be included, which clusters the output of the learning model during pre-learning, and when using the learning model, the output of the learning model is also input into the learner, with the clustering result used as a reliability indicator.

[0105] Alternatively, an evaluation network can be included, which accumulates the results of human evaluation of the results whenever an input is given to the learning model during prior learning, and learns the feature values ​​of the inputs with high evaluation results. When using the learning model, the input of the learning model is also input into the aforementioned evaluation network, and the similarity between the feature values ​​of its output result and the feature values ​​of the learning result is used as a reliability index.

[0106] Alternatively, a learner may be included, which, during prior learning, accumulates the results obtained by human evaluation whenever an input is given to the learning model. The learner then clusters the inputs of the learning model with high evaluation results. When using the learning model, the input of the learning model is also input into the learner, and the clustering results are used as a reliability index.

[0107] The correction and verification unit 203 uses the simulation results implemented by the output verification unit 202 to determine the rationality of the model output data D103 and / or model input data D101. For example, the correction and verification unit 203 can compare the state of the object device 2 represented by the simulation results with the state of the object device 2 determined by the input information D11, model output data D103, and / or model input data D101 to determine whether correct control has been performed, thereby determining the rationality of the model output data D103 and / or model input data D101. The correction and verification unit 203 can also determine that correct control has been performed if the state of the object device 2 represented by the simulation results is consistent with the state of the object device 2 determined by the model output data D103 and / or model input data D101. The state of the object device 2 compared here is not limited to one.

[0108] In addition, the correction and confirmation unit 203 can, for example, confirm whether the state of the object device 2 or the control trajectory represented by the simulation results is consistent with the control represented by the input information D11, or whether it does not contain content that has been prohibited in advance, thereby judging the rationality of the model output data D103 and / or the model input data D101.

[0109] In addition, the correction and confirmation unit 203 can also prompt the simulation results to the input source 1a of the input information D11, so that the input source 1a can answer whether the desired control has been performed, thereby judging the rationality of the model output data D103 and / or the model input data D101.

[0110] The correction and confirmation unit 203 can also correct the model input data D101 if it determines that the model output data D103 and / or the model input data D101 are incorrect. Alternatively, the correction and confirmation unit 203 can replace the correction of the model input data D101 by generating supplementary information D18 for the input information D11 and inputting it to the input source 1a.

[0111] The control system 1000 can, for example, serve as the operating environment for the learning model unit 100. Figures 1 to 5 The structure shown in either of the above. Similar to the learning model unit 100, in this case, part or all of the structure can be an internal structure of the control system 1000 or an external structure.

[0112] Furthermore, the aforementioned learning model unit 100 and its surrounding structure are merely illustrative; not all structural elements are essential. Installation can be selected appropriately based on the desired functionality.

[0113] Figure 6 This is an illustrative diagram representing an example of model learning. For example... Figure 6As shown, the model information D102 may be generated by the model generation unit 107 using the model learning data D105 to perform machine learning.

[0114] The model generation unit 107 is a processing unit that generates or updates model information D102 based on the input model learning data D105 according to a prescribed algorithm. The model generation unit 107 is implemented, for example, by a CPU or similar device included in the information processing apparatus 20 that operates according to a program. Here, the algorithm used by the model generation unit 107 can be a machine learning algorithm corresponding to the learning model, such as teacher-assisted learning, reinforcement learning, or unassisted learning, or it can be deep learning, genetic programming, functional logic programming, or other known algorithms.

[0115] Furthermore, the model generation unit 107 can also generate or update model information D102 based on the model reference information D104 for the input model learning data D105. Additionally, the model generation unit 107 can also generate and update model information D102 based on the model output data D103 from the model control unit 101 for the input model learning data D105.

[0116] The model learning data D105 is not particularly limited. For example, in the case of teacher-guided learning used as a learning algorithm, the model learning data D105 may include candidate input model data D101 and corresponding candidate model output data D103. Alternatively, the model learning data D105 may also include the actual input model input data D101 and / or the actual output model output data D103. By appropriately using the actual input model data D101 and / or output model data D103, feedback control can be performed. Furthermore, the model learning data D105 may also include information obtained from the equipment or processing unit of the system that actually operates the learning model.

[0117] The model information D102 generated or updated by the model generation unit 107 is stored in the model information storage unit 11 and then provided to the model control unit 101. Alternatively, the model generation unit 107 may also directly output the model information D102 to the model control unit 101.

[0118] The model generation unit 107 can generate model information D102 by learning in advance, using the input model learning data D105 before the model control unit 101 uses the model information D102, and store it in the model information storage unit 11.

[0119] The update of model information D102 performed by the model generation unit 107 can also be a process known as FineTune.

[0120] Furthermore, the model generation unit 107 may be included in the control system 1000 or in other systems different from the control system 1000.

[0121] In addition, Figure 1 The learning model unit 100, device information storage unit 110, and device information D13 are shown separately, but the device information storage unit 110 and device information D13 may also be part of the learning model unit 100. That is, the learning model unit 100 may include the device information storage unit 110 and device information D13. For example, the learning model unit 100 may have the device information storage unit 110 as one of the reference information storage units 12 described later. Alternatively, the device information D13 may be pre-added to the model by using the device information D13 during the model learning stage when learning the model used by the learning model unit 100. In this case, the device information storage unit 110 may be omitted.

[0122] Furthermore, the learning model unit 100 may be part or all of the internal structure of the control system 1000 or the external structure of the control system 1000. In the case of being an external structure of the control system 1000, the control system 1000 may replace part or all of the learning model unit 1000 and have an interface capable of exchanging information with the external system that has that part or all of it. For example, the control system 1000 may make the model information storage unit 11, referred to as the core of the learning model, an external structure. Alternatively, the control system 1000 may also make the model information storage unit 11, referred to as the core of the learning model, and the model control unit 101 responsible for the model's algorithm, external structures.

[0123] In the control system 1000, to distinguish between the model control unit 101 responsible for the learning model algorithm and the part that performs the processing of issuing requests to the model control unit 101 and receiving responses, the part that performs the latter processing is sometimes referred to as the "model processing unit". More specifically, the model processing unit is equivalent to the part of the information processing device 10, control unit 104, or control unit 104a other than the model control unit 101. Furthermore, the model processing unit may be implemented, for example, in the case where the model control unit 101 exists in an internal environment, by means of an OS (Operating System) running on the information processing device 10 that calls the learning model application, a prompting application (and a control unit as its operating environment). Furthermore, the model processing unit may be implemented, for example, in the case where the model control unit 101 exists in an external environment, by means of a browser running on the information processing device 10, a client application (and a control unit as its operating environment).

[0124] Furthermore, the structure of the aforementioned learning model and the information processing device that serves as its operating environment, as well as the relationship between the learning model and the control system having the learning model, are the same in other embodiments.

[0125] In this embodiment, the input information D11 corresponds to the model input data D101. Furthermore, the control statement D12 corresponds to the model output data D103. The learning model unit 100 (especially the model control unit 101) can be configured, for example, to output the control statement D12 corresponding to the input information D11 based on the model information D102 and, as needed, based on the model reference information D104, if the input information D11 is received.

[0126] Furthermore, in the above-described case, the model generation unit 107, corresponding to the learning model unit 100, can, for example, use model learning data D105, which includes candidates for input information D11 that can be input to the model control unit 101, to perform machine learning and generate or update model information D102. Alternatively, the model generation unit 107 can also, for example, use model learning data D105, which includes candidates for input information D11 that can be input to the model control unit 101 and candidates for corresponding control descriptions D12, to perform machine learning and generate or update model information D102.

[0127] In this embodiment, the learning model unit 100 may be, for example, a language learning model such as LLM (Language Learning Model) that takes natural language as input and obtains output results, and its operating environment. Alternatively, the learning model unit 100 may be, for example, an image learning model such as VLM (Visual Learning Model) that takes an image as input and obtains output results, and its operating environment. Furthermore, the learning model unit 100 may be, for example, a multi-model that takes natural language and an image as input and obtains output results, and its operating environment. In this case, the input information D11 may also be input in the form of text data, image data, a combination of text data and image data, or data forms that can be transformed into them (such as audio data, a combination of audio data and image data, i.e., video). Moreover, the learning model used in this embodiment is not limited to the models described above.

[0128] In this embodiment, the input information D11 received by the control system 1000 can be considered as information related to the operating environment, specifically, the requirements (in this case, the control content requested from the object device 2) in the environment where the operation is performed. Therefore, the input information D11 received by the control system 1000 can be considered an example of first information representing the requirements in the operating environment. Furthermore, corresponding to the aforementioned input information D11, the control statement D12 and execution code D14 can be considered as information used in this operation (the operation involved in controlling the object device 2). Hereinafter, the control statement D12, which outputs from the operating environment of the learning model based on the model input data input from the input information D11 to a specified output target, is sometimes referred to as second information.

[0129] Here, in the relationship between input information D11 and model input data, in the case of model input data based on input information D11, it can include input information D11 itself, information after transforming input information D11 into a form matching the input of the learning model, and information supplementing input information D11. Similarly, in the relationship between model output data and the second information, in the case of second information based on model output data, it can include model output data itself, data after transforming model output data into a form matching the input of the output target, and data supplementing model output data. The relationship between input / output information and model input / output data in other embodiments is the same.

[0130] Next, the operation of the control system 1000 of this embodiment will be described. Figure 7 This is a flowchart representing an example of the operation of control system 1000.

[0131] exist Figure 7 In the example shown, firstly, the control system 1000 receives input information D11 (step S110). For example, the input unit 102 or the input processing unit 201 described above can receive input information D11. The received input information D11 is input to the learning model unit 100 as model input data D101.

[0132] In step S110, the control system 1000 can also receive multiple input information D11. Additionally, the control system 1000 can also communicate with the user 1 through dialogue, that is, repeatedly inputting and outputting information related to the input information D11 with the user 1 while receiving input information D11 that better matches the user 1's needs.

[0133] Next, the control system 1000 performs a process for generating control statements D12 using the learning model unit 100 (step S111). In step S111, the learning model unit 100 generates and outputs control statements D12 corresponding to the input information D11. For example, the learning model unit 100 (more specifically, the model control unit 101) outputs control statements D12 corresponding to the input information D11 based on model information D102, the input information D11, and, as needed, model reference information D104 including device information D13. For example, the learning model unit 100 can use a learning model capable of generating text data to generate control statements D12 of text data based on the input information D11.

[0134] In step S111, the learning model unit 100 (more specifically, the preprocessing unit 105 or the input processing unit 201) may further, before the processing by the model control unit 101, add, modify, or delete elements, or transform (including process) data on the input information D11 to improve the accuracy of the control record D12. Additionally, in step S111, the learning model unit 100 (more specifically, the postprocessing unit 106) may also, after the processing by the model control unit 101, determine whether there are problems with the control record D12, and if problems are determined to exist, perform correction processing on the control record D12.

[0135] The control statement D12 output from the learning model unit 100 is input to the execution code generation unit 120. If the control statement D12 is input, the execution code generation unit 120 generates execution code D14 based on the input control statement D12 (step S112).

[0136] Next, the execution code D14 generated by the execution code generation unit 120 is input to the target device 2 (step S113). As already explained, the input of the execution code D14 to the target device 2 can be directly input from the control system 1000 (more specifically, the execution code generation unit 120), or indirectly input via a communication network, other devices (servers, various conversion devices, etc.) or by a person.

[0137] Therefore, the object device 2 performs actions according to the input execution code D14.

[0138] Alternatively, if execution code D14 is output, and the state of the target device 2 changes as a result, such as through control of the target device 2, then the control system 1000 acquires state information D15 (step S114). The acquired state information D15 is stored in the device information storage unit 110, for example, as part of the device information D13. The control system 1000 can, for example, use the acquired state information D15 to update the device information D13 stored in the device information storage unit 110. Furthermore, the control system 1000 may also output the acquired state information D15 as information representing the control result to the user 1, the learning model unit 100, or other devices not shown. Moreover, if the control system 1000 does not use the state information D15, the processing in step S114 can be omitted.

[0139] The control system 1000 may also repeat the processing of steps S110 to S114 multiple times (for example, until the desired control is completed for the target device 2).

[0140] In addition, the control system 1000 can also output the control record D12 to the user 1's operation terminal, etc., and after the user 1 confirms the content, the subsequent processing (such as code generation in the code generation unit 120) is executed through the user 1's operation.

[0141] The status information D15 input to the learning model unit 100 is used, for example, for additional learning by the learning model unit 100. The learning model unit 100 may, for example, update the model information D102 and / or the model reference information D104 based on the input status information D15.

[0142] As described above, according to this embodiment, there is no need for user 1 to create control description D12, and execution code D14 can be generated based on input information D11 input from user 1, thus enabling efficient operation of controlling the target device 2.

[0143] In addition, in this embodiment, the input information D11 may be text, image, sound or a combination thereof that explicitly or implicitly represents the control content for the target device 2. Therefore, the input workload of the input information D11 can be further suppressed, and the operation of controlling the target device 2 can be made more efficient.

[0144] Furthermore, according to this embodiment, a control statement D12 can be generated based on the input information D11 using a learning model. Therefore, even if the user 1 is unaware of the detailed specifications of the target device 2 or the specifications of the control statement D12, or other information used to control the target device 2, a control statement D12 corresponding to the input information D11 can still be generated. Thus, high-performance operation of controlling the target device 2 can be achieved. Here, high-performance operation of controlling the target device 2 also includes high precision in controlling the target device 2.

[0145] Furthermore, in this embodiment, the status information D15 obtained after controlling the target device 2 based on the input information D11 can be used for the generation of the next control description D12, thus further improving the performance of the operation of controlling the target device 2.

[0146] Furthermore, in the above example, only one object device 2 is shown, but the object device 2 controlled by the control system 1000 can be multiple. In the above case, for example, the input information D11 may contain information that can determine the object device 2, or the input side (input unit 102, preprocessing unit 105, input processing unit 201) toward the learning model unit 100 may perform the processing of determining the object device 2 based on the input information D11, or the learning model unit 100 may output the control content obtained by determining the object device 2 as the learning result.

[0147] Variation Example 1-1

[0148] Next, a variation of the control system 1000 will be described. Figure 8 This is a structural diagram showing a modified example of the control system 1000 according to this embodiment, namely, control system 1000a. Furthermore, elements identical to those in the control system 1000 are labeled with the same reference numerals and their descriptions are omitted.

[0149] exist Figure 8 In the control system 1000a shown, after user 1 confirms, the output from the learning model unit 100 is input to the subsequent execution code generation unit 120.

[0150] In this embodiment, user 1 can confirm the control statement D12 output from the learning model unit 100 and input input information D11 based on the confirmation result. Alternatively, user 1 can also confirm the feedback information D16 from the execution code generation unit 120 and / or the target device 2, based on the control statement D12 output from the learning model unit 100, and input input information D11 based on their confirmation results. In this case, user 1 can input not only new content input information D11, but also input information D11 indicating correction, addition, or cancellation of already input content. In this case, the input information D11 can include instructions for the learning model unit 100. For example, user 1 can input instructions for removing problems contained in the input information D11 or problems contained in the output control statement D12, along with the feedback information D16, as input information D11. Here, the instructions for removing problems also include input for searching for the cause of the problem and its solution.

[0151] When a control request is made to a processing unit that is a level later than the learning model unit 100, the feedback information D16 may also include a response to the request returned by that processing unit. Alternatively, the feedback information D16 may include information obtained from that processing unit after a control request has been made to it. For example, the feedback information D16 may include a response returned by the execution code generation unit 120 to the request when the generation of control statement D12 is requested by inputting control statement D12 into the execution code generation unit 120. Additionally, the feedback information D16 may include a response returned by the target device 2 to the request when the execution code D14 is requested by inputting execution code D14 into the target device 2. The feedback information D16 may include status information D15. The feedback information D16 may be output directly to the user 1, or it may be output to the user 1 via the execution code generation unit 120 or an output device (not shown) provided by the control system 1000.

[0152] Furthermore, the feedback information D16 may include information used to determine whether the control requested from a processing unit that is later than the learning model unit 100 has been correctly executed in that processing unit. This information is not limited to information obtained directly from that processing unit. For example, it may also be information obtained from other people, devices, networks, or AI (none shown). The feedback information D16 may, for example, include parsing information such as execution time or control trajectory information used to determine whether the execution code D14 can correctly execute the target control. User 1 may also, for example, instruct the learning model unit 100 to control the timing of the process in the control description D12 or adjust the delivery cycle based on the information contained in the feedback information D16.

[0153] Additionally, user 1 can, for example, exchange feedback information D16 with the learning model unit 100 multiple times, judging the rationality (whether there are any problems) of the output control statement D12 each time. User 1 can also output the control statement D12 to the execution code generation unit 120 if it is determined that the control statement D12 is not problematic.

[0154] The feedback information D16 can be obtained, for example, in step S114 above.

[0155] In addition, Figure 8 The example shown is of user 1 inputting control description D12 into execution code generation unit 120, but the input of control description D12 into execution code generation unit 120 can also be performed by learning model unit 100 that receives instructions from user 1.

[0156] In this example, control descriptor D12 may contain descriptors corresponding to low-code or no-code.

[0157] In this example, the exchange of information between user 1 and learning model unit 100 can be carried out, for example, through the terminal owned by user 1, or through the user interface (e.g., input unit 102) of the information processing device 10 that enables the learning model unit 100 to operate.

[0158] Alternatively, the update of input information D11 in this example can be performed not by user 1, but by the control system 1000 side (e.g., correction confirmation unit 203).

[0159] Additionally, feedback information D16 can also be input to the learning model unit 100. The feedback information D16 input to the learning model unit 100 can be used, for example, for additional learning by the learning model unit 100. The learning model unit 100 can also update the model information D102 and / or the model reference information D104 based on the input feedback information D16.

[0160] Other aspects can be the same as other control systems involved in this embodiment.

[0161] As described above, in this modified example, user 1 can simultaneously confirm the control statement D12 output from the learning model unit 100 and exchange additional instructions, error inquiries, etc., with the learning model unit 100, while correcting the input information D11. Therefore, high precision of the output control statement D12 can be achieved. As a result, the operation of controlling the target device 2 can be made more efficient and perform better.

[0162] Variations 1-2

[0163] Next, a second variation of the control system 1000 will be described. Figure 9 This is a structural diagram showing a variation of control system 1000, namely control system 1000b. Furthermore, elements identical to those in control system 1000 and control system 1000a are labeled with the same reference numerals and their descriptions are omitted.

[0164] exist Figure 9 In the control system 1000b shown, the difference lies in that the learning model unit 100 sends query D17 back to user 1. Examples of query D17 include following up on unclear or uncertain input information D11, requesting a solution, or requesting re-input from someone whose state or performance has changed. Alternatively, as a follow-up on unclear or uncertain input information D11, the learning model unit 100 may suggest a reference point and output query D17 requesting more specific information to user 1. Alternatively, as a request for a solution, the learning model unit 100 may suggest a reference point and output query D17 providing candidate solutions as options to user 1. Alternatively, as a request for a solution, the learning model unit 100 may suggest a reference point and output information about the most likely solution along with query D17 indicating its correctness to user 1. In addition, the learning model unit 100 can also temporarily generate intermediate control statements that are easy for humans to understand, i.e., intermediate control statements, and output the generated intermediate control statements and the query D17 that asks whether they are correct to the user 1.

[0165] The output of D17 can be queried, for example, after step S110 above.

[0166] Alternatively, if the learning model unit 100 receives a response from user 1 to query D17, it may update the input information D11 or determine the interpretation (explanation) of the input information D11.

[0167] The processing of the learning model unit 100 described above in this example can also be installed as a function of the input unit 102 or preprocessing unit 105 of the learning model unit 100, or as part of the input processing unit 201 (not shown) of the information processing device 10.

[0168] As described above, in this modified example, in response to the input information D11, a query D17 is output to user 1, and the input information D11 is updated or its interpretation is determined based on the user's response, thus eliminating the uncertainty of the input information D11. As a result, the output control description D12 can be made more accurate, and the operation of controlling the target device 2 can be made more efficient and performant.

[0169] Variations 1-3

[0170] Next, a third variation of the control system 1000 will be described. Figure 10 This is a structural diagram representing a variant of control system 1000, namely control system 1000c. Furthermore, elements identical to those in control systems 1000, 1000a, and 1000b are labeled with the same reference numerals and their descriptions are omitted.

[0171] Figure 10 The control system 1000c shown also includes a status acquisition unit 130. The status acquisition unit 130 acquires feedback information D16 indicating the processing result or status information D15 indicating the processed device status from the processing unit that processes the control description D12 output from the learning model unit 100 and the execution code D14 generated therefrom. Here, the feedback information D16 or the status information D15 may include information such as execution time or control trajectory information used to determine whether the execution code D14 correctly executed the target control.

[0172] The status acquisition unit 130 can, for example, input the acquired information to the learning model unit 100. Additionally, the status acquisition unit 130 can, for example, update the device information D13 based on the acquired information. Furthermore, the status acquisition unit 130 can, for example, generate supplementary information (including addition, correction, and cancellation) to the input information D11 based on the acquired information, and input this supplementary information D18 to the learning model unit 100. Additionally, the status acquisition unit 130 can, for example, generate supplementary information (including addition, correction, and cancellation) to the control description D12 based on the acquired information, and input this supplementary information D18 to the learning model unit 100.

[0173] The status acquisition unit 130 can, for example, generate control instructions for new content or instructions to add, modify, or cancel content shown by the input information D11, and input these as supplementary information D18 to the learning model unit 100. Alternatively, the status acquisition unit 130 can, for example, input instructions for removing problems contained in the input information D11 or problems contained in the output control description D12, together with the acquired information, as supplementary information D18 to the learning model unit 100.

[0174] The status acquisition unit 130 may, for example, determine whether the acquired information indicates normal processing or normal status of the processor. If not, it inputs supplementary information D18, which indicates the correction, addition, or cancellation of the content shown by the input information D11, together with the acquired information, into the learning model unit 100.

[0175] The learning model unit 100 can, for example, update the model information D102 and / or the model reference information D104 based on the input information (status information D15, feedback information D16, supplementary information D18, etc.).

[0176] The generation of supplementary information D18 can be performed, for example, in step S115 described above. Furthermore, the output target of supplementary information D18 can also include entities other than the learning model unit 1000. The control system 1000 can, for example, output the supplementary information D18 generated by the status acquisition unit 130 to user 1 or other devices not shown.

[0177] Alternatively, the status acquisition unit 130 can acquire the action result obtained through the simulator (not shown) of the object device 2 or the action result in the debug mode of the object device 2 without actually causing the object device 2 to operate. The debug mode of the object device 2 refers to the mode in which the executable code is executed on the control board of the object device 2, but actual device control is not executed, and only the internal state is updated; it is also called the idle mode. By utilizing the debug mode, it is possible to safely attempt to execute code D14 on the object device 2 in a state close to actual control.

[0178] Not limited to this variation, as a method for judging the rationality of the control statement D12 output from the learning model unit 100 and the execution code D14 generated based on the control statement D12 without actually causing the target device 2 to operate, the execution code generation unit 120 can be connected to the target device 2 or a simulator in a switchable manner as the output target of the execution code D14. The simulator includes an operator that makes the icon of the target device 2 move through augmented reality space. In addition, the execution code generation unit 120 can also provide information indicating whether the execution is in normal mode or debug mode when outputting the execution code D14 to the target device 2.

[0179] The processing of the status acquisition unit 130 in this example can also be installed as part of the input unit 102, preprocessing unit 105 and postprocessing unit 106 of the learning model unit 100 or the input processing unit 201, output confirmation unit 202 and correction confirmation unit 203 (all not shown) of the information processing device 10.

[0180] Other aspects can be the same as other control systems involved in this embodiment.

[0181] As described above, in this modified example, in response to the input information D11, the status acquisition unit 130 obtains feedback information D16 indicating the processing result or status information D15 indicating the processed device status from the target device 2, which is the output data D103 of the model and / or the information generated based thereon, or from the execution code generation unit 120. Based on the obtained information, it appropriately issues supplementary information D18 to the learning model unit 100. This enables high precision in the control record D12, and even higher efficiency and performance in controlling the target device 2.

[0182] Furthermore, in this modified example, for example, cooperation between a human and a machine (state acquisition unit 130) can improve the accuracy of the input to the learning model unit 100, thus contributing to the reduction of the workload of user 1.

[0183] Implementation Method 2

[0184] Next, this second embodiment will be described. In this embodiment, an example will be described of assisting the operation involved in controlling the target device using a learning model.

[0185] Below, for example, consider controlling various control devices within a factory, such as PLCs, machining centers, robots, sensors, conveyors, and other mechanical control devices. While skilled operators may be familiar with a wide variety of control devices and complex control methods, changes in configurations sometimes require less experienced operators to control these devices. Furthermore, if new control devices (including version upgrades) are introduced, all operators must understand the control methods corresponding to the new devices; insufficient understanding may lead to errors.

[0186] In the above situation, if the desired control can be reliably performed even without knowing the specific control method, such as control commands, control signals, control codes, or commands to the controller corresponding to the control device, then the operation will be more efficient and perform better, and therefore preferred.

[0187] Furthermore, the scenarios for controlling the equipment are not limited to within a factory, and the effective application scenarios of this implementation method are not limited to within a factory.

[0188] Figure 11 This is a structural diagram illustrating an example of the control system 2000 according to Embodiment 2. Figure 11 The control system 2000 shown is a control system for controlling equipment using a learning model, and has a learning model unit 200 and an equipment information storage unit 210 (denoted as equipment information DB in the figure).

[0189] If input information D21 is input to the learning model unit 200, then control command D22 is output. For example, if input information D21 is input to the learning model unit 200, then control command D22 is output based on model information D102. The structure of the learning model unit 200 can be substantially the same as that of the learning model unit 100 in Embodiment 1.

[0190] In this embodiment, the learning model unit 200 is a model and its operating environment configured such that if input information D21 is input, it outputs a control command D22 corresponding to the input information D21. Alternatively, the learning model unit 200 may also be a model and its operating environment configured such that if input information D21 is input, it generates and outputs a control command D22 based on the input information D21, the device information D23, and other information that the learning model unit 200 can refer to.

[0191] In this embodiment, the input information D21 includes information representing control content for the target device 2. The input information D21 may be, for example, text, images, sounds, or combinations thereof representing control content for the target device 2. The input information D21 may also be, for example, text, images, sounds, or combinations thereof representing multiple control contents for the target device 2. Furthermore, the input information D21 may also include information representing control content that occurs continuously in time; in this case, it may be time-series data constructed from predetermined data including text, images, sounds, or combinations thereof representing control content as described above. The representation of the control content is based on matching the input format of the model used by the learning model unit 200, but is not limited to this if the learning model unit 200 includes error processing, correction processing, or transformation processing in its preceding stages.

[0192] The representation of the control content in the input information D21 can be the same as in Embodiment 1. For example, after determining the control to be performed on the target device 2, the values ​​of the parameters used to perform the control and the state after the control can be specified. In this case, the input information D21 may include, for example, information on determining the control and information indicating the values ​​of the parameters used to perform the control or the state after the control. In addition, the input information D21 may include not only information directly representing the control content for the target device 2, but also information indirectly represented by operation content corresponding to the control content, the user 1's words and actions, the image of the target device 2, or the same control commands from other models.

[0193] Control instruction D22 contains information related to the control of object device 2, expressed in a prescribed form that can be determined by object device 2 or an interface requesting control from object device 2. Control instruction D22 may also contain information indicating a control request to object device 2. Control instruction D22 may be, for example, a control command, control signal, or control code for object device 2. Alternatively, control instruction D22 may also be a command written in a form used by a prescribed controller corresponding to object device 2.

[0194] The device information storage unit 210 stores device information D23, which is related to the target device 2. The processing of the device information storage unit 210 and the device information D23 is essentially the same as that of the device information storage unit 110 and the device information D13 in Embodiment 1. Furthermore, the device information D23 in this embodiment may, for example, include information used in the control of the target device 2. The device information D23 is used, for example, as supplementary information when the learning model unit 200 outputs control command D22. Hereinafter, in this embodiment, in particular, the information representing the state of the target device 2 is sometimes referred to as state information D25.

[0195] In this embodiment, the learning model unit 200 may be, for example, a language learning model such as LLM (Language Learning Model) that takes natural language as input and obtains output results, and its operating environment. Alternatively, the learning model unit 200 may be, for example, an image learning model such as VLM (Visual Learning Model) that takes an image as input and obtains output results, and its operating environment. Furthermore, the learning model unit 200 may be, for example, a multi-model that takes natural language and an image as input and obtains output results, and its operating environment. In this case, the input information D21 may also be input in the form of text data, image data, a combination of text data and image data, or data forms that can be transformed into them (such as audio data, a combination of audio data and image data, i.e., video). Moreover, the learning model used in this embodiment is not limited to the models described above.

[0196] In this embodiment, for the sake of simplicity, the reference numerals of the structural elements corresponding to the learning model unit 100 are sometimes used directly to describe the structural elements corresponding to the learning model unit 200. However, it should be noted that these elements are only provided corresponding to the learning model unit 200. The same applies in other embodiments.

[0197] In this embodiment, the input information D21 corresponds to the model input data D101. Furthermore, the control command D22 corresponds to the model output data D103. The learning model unit 200 (especially the model control unit 101) can be configured, for example, to output the control command D22 corresponding to the input information D21 based on the model information D102 and, as needed, based on the model reference information D104, if the input information D21 is received.

[0198] Furthermore, in the above-described case, the model generation unit 107, corresponding to the learning model unit 200, can, for example, use model learning data D105, which includes candidates for input information D21 that can be input to the model control unit 101, to perform machine learning and generate or update model information D102. Alternatively, the model generation unit 107 can also, for example, use model learning data D105, which includes candidates for input information D21 that can be input to the model control unit 101 and candidates for corresponding control commands D22, to perform machine learning and generate or update model information D102.

[0199] Reference numeral D26 indicates feedback information representing the control result in object device 2. In this embodiment, status information D25 and / or feedback information D26 may also be obtained from the model output data D103 and / or the output target based on the information generated therefrom. The control system 2000 may, for example, output the obtained status information D25 and / or feedback information D26 as information representing the control result to user 1, learning model unit 200, or other devices not shown. In addition, the control system 2000 may also generate supplementary information D28 for the input and output data of the learning model unit 200 based on the obtained status information D25 and / or feedback information D26, and send it to user 1, learning model unit 200, or other devices not shown. Furthermore, the control system 2000 may also be configured to send an inquiry D27 back to user 1 when the input information D21 contains ambiguous or uncertain information. The processing of inquiry D27 is the same as that of inquiry D17 in embodiment 1.

[0200] Figure 12 These are block diagrams representing other examples of the Control System 2000. For example... Figure 12 As shown, the control system 2000 may also include a status acquisition unit 230, which acquires status information D25 and / or feedback information D26 and issues supplementary information D28. The status acquisition unit 230 is the same as the status acquisition unit 130 in Embodiment 1.

[0201] In this embodiment, the target device 2 is not particularly limited. Furthermore, while it is assumed that the target device 2 is a device capable of receiving control command D22 and actually performing control, it is not limited to this if a conversion device such as a controller or converter is included between the target device 2 and the device. In this case, the conversion device can receive control command D22 and control the target device 2.

[0202] In this embodiment, the input information D21 received by the control system 2000 can be considered as information related to the operating environment, specifically, the requirements (in this case, the control content requested from the object device 2) in the environment where the operation of the object device 2 takes place. Therefore, the input information D21 received by the control system 2000 can be considered an example of first information representing the requirements of the operating environment. Furthermore, the control instruction D22, corresponding to the aforementioned input information D21, can be considered as information used in this operation (the operation involved in controlling the object device 2). Hereinafter, the control instruction D22, which outputs from the operating environment of the learning model based on the model input data input from the input information D21 to the specified output target, will be referred to as second information.

[0203] Next, the operation of the control system 2000 of this embodiment will be described. Figure 13 This is a flowchart illustrating an example of the operation of the control system 2000.

[0204] exist Figure 13 In the example shown, firstly, the control system 2000 receives input information D21 (step S210). For example, the input unit 102 or the input processing unit 201 described above can receive input information D21. The received input information D21 is input to the learning model unit 200 as model input data D101.

[0205] Next, the control system 2000 performs the generation process of control command D22 using the learning model unit 200 (step S211). In step S211, the learning model unit 200 (more specifically the model control unit 101) outputs control command D22 corresponding to the input information D21 based on the model information D102, the input information D21, and the model reference information D104 including the device information D23 as needed.

[0206] In step S211, the learning model unit 200 may, for example, use a learning model capable of generating binary data to generate control instructions D22 for binary data based on the input information D21. Alternatively, the learning model unit 200 may also use a learning model capable of generating text data to generate control instructions D22 for text data based on the input information D21. Furthermore, the learning model unit 200 may also use a learning model capable of generating image data to generate control instructions D22 for image data based on the input information D21. Additionally, the learning model unit 200 may also use a learning model capable of generating sound data to generate control instructions D22 for sound data based on the input information D21.

[0207] In step S211, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 200 may further perform the above-mentioned processing.

[0208] The control command D22 output from the learning model unit 200 is input to the target device 2, for example (step S212). The input of the control command D22 to the target device 2 can be directly input from the control system 2000 (more specifically the learning model unit 200 or the information processing device 10 that serves as its operating environment), or indirectly input via a communication network or other devices (servers, various conversion devices, etc.).

[0209] Therefore, the object device 2 performs actions according to the input control command D22.

[0210] The control system 2000 can also output control command D22, and as a result, control the target device 2, etc. If the state of the target device 2 changes and there is feedback from the target device 2, it can obtain state information D25 and feedback information D26 (step S213). In addition, the processing of step S213 is not necessary and can be omitted appropriately.

[0211] The control system 2000 may also repeat the processing of steps S210 to S213 multiple times (for example, until the desired control is completed for the target device 2).

[0212] As described above, according to this embodiment, even if user 1 is unaware of the specific control method for the target device 2, control command D22 can be generated based on the input information D21 input from user 1, and the target device 2 can be controlled based on the generated control command D22. Therefore, the operation involved in the control of the target device 2 can be made more efficient and advanced.

[0213] Furthermore, according to this embodiment, even with ambiguous information, the device can be controlled to an appropriate state.

[0214] Implementation Method 3

[0215] Next, this third embodiment will be described. In this embodiment, an example will be described of using a learning model to assist in the operation of the target device.

[0216] Next, consider operating various devices such as air conditioners, refrigerators, televisions, lighting, washing machines, projectors, various sensors, and communication equipment in a home or building. In recent years, even these consumer-facing devices have offered more advanced functions and more complex controls. To simplify complex control, user interfaces, remote controls, and other controllers have been improved; however, even so, it is difficult to remember all the operations, and although the desired functions are provided, they are sometimes not easily accessible.

[0217] In addition, even for the same type of function, the function name, detailed function, or control method often differ depending on the model. In scenarios where different models are introduced due to replacement or purchase, it is troublesome to have to remember their differences again.

[0218] In addition, depending on the equipment, there are also cases where past operating history is recorded, the operating environment is understood, and automatic control is performed to the appropriate state. However, the appropriate state varies from person to person. In scenarios where many people are gathered, or even when there is only one person, the appropriate state varies depending on changes in physical condition, it is sometimes difficult to achieve proper control.

[0219] In the above situation, even if the specific operating method is unknown, or even if the operator does not have a proper understanding of the state, if the operation to set the desired state can be performed simply, the efficiency and performance of the operation can be improved, and therefore it is preferred.

[0220] Furthermore, the scenarios for operating the device are not limited to inside a home or building, and the effective use scenarios of this embodiment are not limited to inside a home or building.

[0221] Figure 14 This is a structural diagram illustrating an example of the control system 3000 involved in Embodiment 3. Figure 14 The control system 3000 shown is a control system for operating equipment using a learning model. It has a learning model unit 300, an equipment information storage unit 310 (referred to as equipment information DB in the figure), an input interface 311 (referred to as input IF in the figure), and an output interface 312 (referred to as output IF in the figure).

[0222] If input information D31 is input to the learning model unit 300, then operation instruction D32 is output. For example, if input information D31 is input to the learning model unit 300, then operation instruction D32 is output based on model information D102. The structure of the learning model unit 300 can be substantially the same as that of the learning model unit 100 in Embodiment 1.

[0223] In this embodiment, the learning model unit 300 is a model and its operating environment configured such that if input information D31 is input, it outputs an operation instruction D32 corresponding to the input information D31. Alternatively, the learning model unit 300 may also be a model and its operating environment configured such that if input information D31 is input, it generates and outputs the operation instruction D32 based on the input information D31, the device information D33, and other information that the learning model unit 300 can refer to.

[0224] In this embodiment, the input information D31 includes information representing the operation content requested for the target device 2. The input information D31 may be, for example, text, images, sounds, or combinations thereof representing the operation content for the target device 2. The input information D31 may also be, for example, text, images, sounds, or combinations thereof representing multiple operation contents for the target device 2. Furthermore, the input information D31 may also include information representing operation contents performed sequentially in time; in this case, it may be time-series data constructed from predetermined data including text, images, sounds, or combinations thereof representing control content as described above. The representation of the control content is based on matching the input form of the model used by the learning model unit 300, but is not limited to this if the learning model unit 300 includes error processing, correction processing, or transformation processing in its preceding stages.

[0225] As an example of how the operation content in the input information D31 is represented, after determining the operation to be performed on the target device 2, the values ​​of the parameters used to perform the operation and the state after the operation can also be specified. In this case, the input information D31 may include, for example, information indicating that the operation has been determined and information indicating the values ​​of the parameters used to perform the operation or the state after the operation. The values ​​of the parameters used to perform the operation may include, for example, values ​​related to the type of operation (ON / OFF, etc.), orientation, amount, and time. In addition, the input information D31 may include not only information directly indicating the operation content on the target device 2, but also information indirectly represented by control content corresponding to the operation content, the user 1's words and actions, the image of the target device 2, or the same operation instructions from other models.

[0226] Operation instruction D32 contains operation-related information about the object device 2, expressed in a prescribed form that can be discerned by the object device 2 or the interface (including a human) requesting control from the object device 2. Operation instruction D32 may also contain information indicating an operation request or control request to the object device 2. Operation instruction D32 may be, for example, an operation command, operation signal, operation code, control command, control signal, or control code for the object device 2. Alternatively, operation instruction D32 may also be a command written in a form used by a prescribed controller corresponding to the object device 2. Operation instruction D32 can be considered as adding operation-related information to the aforementioned control instruction D22. Furthermore, if the interface is a human, i.e., when control is requested from the object device 2 by a human, operation instruction D32 may be information indicating the operation method of the object device 2 in a form that can be discerned by a human.

[0227] The device information storage unit 310 stores device information D33, which is related to the target device 2. The processing of the device information storage unit 310 and the device information D33 is essentially the same as that of the device information storage unit 110 and the device information D13 in Embodiment 1. Furthermore, the device information D33 in this embodiment may, for example, include information used in the operation of the target device 2. For example, the device information D33 may include information showing the flow of the actual operation performed on the target device 2 for the operation content. Additionally, the device information D33 may, for example, include commands, signals, codes, etc., issued to the target device 2. The device information D33 may, for example, be used as additional information when the learning model unit 300 outputs the operation instruction D32. Hereinafter, in this embodiment, in particular, the information representing the state of the target device 2 is sometimes referred to as state information D35.

[0228] Input interface 311 is an interface that receives input information D31 from user 1 and inputs it to the learning model unit 300. For example, input interface 311 may be an interface that transforms the input information D31 from user 1 into data that matches the input of the learning model unit 300 and outputs it. Furthermore, input interface 311 may be provided as an example of the input unit 102 described above.

[0229] Output interface 312 is an interface that receives operation instructions D32 from the learning model unit 300 and outputs them to a specified output target. Furthermore, output interface 312 can be, for example, an example of the output unit 103 described above. Output interface 312 can also be an interface that converts the operation instructions D32 output from the learning model unit 300 into data matching the specified output target and outputs it. In this embodiment, the output target of output interface 312 can include object device 2, controller 4 (not shown), specified display 7 (not shown), and user 1's operation terminal (not shown).

[0230] In this embodiment, the learning model unit 300 may be, for example, a language learning model such as LLM (Language Learning Model) that takes natural language as input and obtains output results, and its operating environment. Alternatively, the learning model unit 300 may be, for example, an image learning model such as VLM (Visual Learning Model) that takes an image as input and obtains output results, and its operating environment. Furthermore, the learning model unit 300 may be, for example, a multi-model that takes natural language and an image as input and obtains output results, and its operating environment. In this case, the input information D31 may also be input in the form of text data, image data, a combination of text data and image data, or data forms that can be transformed into them (such as audio data, a combination of audio data and image data, i.e., video). Moreover, the learning model used in this embodiment is not limited to the models described above.

[0231] In this embodiment, the input information D31 corresponds to the model input data D101. Furthermore, the operation instruction D32 corresponds to the model output data D103. The learning model unit 300 (particularly the model control unit 101) can be configured, for example, to output the operation instruction D32 corresponding to the input information D31 based on the model information D102 and, as needed, based on the model reference information D104, if the input information D31 is received.

[0232] Furthermore, in the above-described case, the model generation unit 107, corresponding to the learning model unit 300, may, for example, use model learning data D105, which includes candidates of input information D31 that can be input to the model control unit 101, to perform machine learning and generate or update model information D102. Additionally, the model generation unit 107 may, for example, use model learning data D105, which includes candidates of input information D31 that can be input to the model control unit 101 and candidates of corresponding operation instructions D32, to perform machine learning and generate or update model information D102.

[0233] Although figures are omitted, in this embodiment, status information D35 and / or feedback information D36 can be obtained from the model output data D103 of the learning model unit 300 and / or the output target based on the information generated therefrom. The control system 3000 can, for example, output the obtained status information D35 and / or feedback information D36 as information indicating a response result to the user 1, the learning model unit 300, or other devices not shown. Alternatively, the control system 3000 can be configured to send an inquiry D37 back to the user 1 if the input information D31 contains ambiguous or uncertain information. Furthermore, the control system 3000 can also generate supplementary information D38 for the input / output data of the learning model unit 300 based on the obtained status information D35 and / or feedback information D36, and send it to the user 1, the learning model unit 300, or other devices not shown. The processing of status information D35, feedback information D36, inquiry D37, and supplementary information D38 can be substantially the same as in Embodiment 1.

[0234] Additionally, the control system 3000 may also include a status acquisition unit 330 (not shown), which acquires status information D35 and / or feedback information D36, and issues supplementary information D38 as needed. The status acquisition unit 330 is the same as the status acquisition unit 130 in Embodiment 1.

[0235] In this embodiment, the target device 2 is not particularly limited. Furthermore, while the target device 2 is assumed to be a device capable of receiving the operation command D32 and performing control corresponding to the operation content expressed by the operation command D32, it is not limited to this if a controller 4 or a converter or other conversion device that transforms various signals is included between the target device 2 and the target device 2. In this case, the conversion device or operator can simply receive the operation command D32 and operate the target device 2.

[0236] In this embodiment, the input information D31 received by the control system 3000 can be considered as information related to the working environment, specifically, the requirements (in this case, the operation requested from the target device) in the environment where the operation is performed on the target device 2. Therefore, the input information D31 received by the control system 3000 can be considered an example of first information representing the requirements of the working environment. Furthermore, the operation instruction D32, corresponding to the aforementioned input information D31, can be considered as information used in the operation (the operation involved in the operation of the target device 2). Hereinafter, the operation instruction D32, which outputs from the operating environment of the learning model based on the model input data input from the input information D31 to a specified output target, is sometimes referred to as second information.

[0237] Next, the operation of the control system 3000 in this embodiment will be described. Figure 15This is a flowchart illustrating an example of the operation of the control system 3000.

[0238] exist Figure 15 In the example shown, firstly, the input interface 311 of the control system 3000 receives input information D31 (step S310). The received input information D31 is input to the learning model unit 300 as model input data D101.

[0239] Next, the control system 3000 performs the process of generating operation instructions D32 using the learning model unit 300 (step S311). In step S311, the learning model unit 300 (more specifically, the model control unit 101) generates and outputs operation instructions D32 corresponding to the input information D31 based on the model information D102 and the input information D31, and as needed based on the model reference information D104 containing the device information D33.

[0240] In step S311, the learning model unit 300 may, for example, use a learning model capable of generating binary data, and generate operation instructions D32 for binary data based on the input information D31. Alternatively, the learning model unit 300 may also use a learning model capable of generating text data, and generate operation instructions D32 for text data based on the input information D31. Furthermore, the learning model unit 300 may also use a learning model capable of generating image data, and generate operation instructions D32 for image data based on the input information D31. Additionally, the learning model unit 300 may also use a learning model capable of generating sound data, and generate operation instructions D32 for sound data based on the input information D31.

[0241] In step S311, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 300 may further perform the above-mentioned processing.

[0242] The operation instruction D32 output from the learning model unit 300 is output to a designated output target via the output interface 312, for example. Here, the designated output target may be the object device 2, the controller 4, the designated display 7, or the user 1's operation terminal (not shown). If the operation instruction D32 is input to the designated output target, the object device 2 is operated according to the input operation instruction D32 (step S312).

[0243] For example, output interface 312 can output operation command D32 to the target device 2. In this case, the target device 2, which receives operation command D32 (e.g., operation command, operation signal, operation code, control command, control signal, or control code), can also perform actual control according to operation command D32. Alternatively, output interface 312 can also output operation command D32 to the controller 4 corresponding to the target device 2. In this case, the controller 4, which receives operation command D32 (e.g., commands, operation commands, operation signals, operation codes, etc., which are indirect control information for the target device 2), can also operate the target device 2 according to operation command D32. The controller 4 can also output direct control information such as control codes to the target device 2 based on the control information represented by the received operation command D32, thereby operating the target device 2. Here, controller 4 can be, for example, an operation panel provided by the target device 2, or a remote control corresponding to the target device 2 directly operated by the user. Controller 4 includes controllers inherent to the target device 2 and general-purpose controllers. Additionally, output interface 312 can also output operation command D32 to the user 1's operating terminal or a specified display. In this case, the user 1's operating terminal or display, which receives the operation instruction D32 (such as information indicating the operation method), displays the operation instruction D32. Furthermore, the user 1 can also operate the target device 2 or the controller 4 by referring to the displayed operation instruction D32.

[0244] The input of the operation command D32 to the output target can be directly input from the control system 3000 (more specifically the learning model unit 300 or the information processing device 10 that serves as its operating environment), or indirectly input via a communication network or other devices (servers, various conversion devices, etc.).

[0245] Therefore, the object device 2 performs an action according to the operation instruction D32.

[0246] The control system 3000 can also output operation command D32, and as a result, control the target device 2, etc. If the state of the target device 2 changes and there is feedback from the target device 2, it can obtain state information D35 and feedback information D36 (step S313). In addition, the processing of step S313 is not necessary and can be omitted appropriately.

[0247] The control system 3000 may also repeat the processing of steps S310 to S313 multiple times (for example, until the desired operation is completed for the target device 2).

[0248] As described above, according to this embodiment, even if user 1 is unaware of the specific operation method for the target device 2, operation instructions D32 can be generated based on the input information D31 input by user 1, and the target device 2 can be operated based on the generated operation instructions D32. Therefore, the operation of the target device 2 can be made more efficient and advanced.

[0249] Furthermore, according to this embodiment, even with ambiguous information, the device can be operated in an appropriate state. Additionally, according to this embodiment, the device can be operated in an appropriate state without relying on the device itself, even without learning the device's operating method.

[0250] Variation Example 3-1

[0251] Next, a variation of the control system 3000 will be described. Figure 16 This is a structural diagram illustrating a modified example of the control system 3000 according to this embodiment, namely, control system 3000a. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0252] Figure 16 The control system 3000a shown also includes an input judgment unit 31. The input judgment unit 31 is a unit that, if it receives input information D31, parses the input information D31 and switches the controlled object corresponding to the input information D31. In this modified example, the input judgment unit 31 switches the controlled object corresponding to the input information D31 between the learning model unit 300 and the output interface 312.

[0253] The input determination unit 31 can, for example, switch the controlled object corresponding to the input information D31 based on whether the input information D31 matches the command rules for the operation of the target device 2. Alternatively, the input determination unit 31 can directly input the input information D31 to the output interface 312 if the input information D31 matches the command rules for the operation of the target device 2. On the other hand, the input determination unit 31 can also input the input information D31 to the learning model unit 300 if the input information D31 does not match the command rules for the operation of the target device 2.

[0254] Whether it conforms to the command rules of the operation can be determined, for example, using a model described by a rule base. Here, the input judgment unit 31 can also be a relatively lightweight learning model compared to the learning model unit 300.

[0255] In order to distinguish the input information in the output interface 312, the operation instruction D32 output from the learning model unit 300 is sometimes called operation instruction D32a, and the input information D31 output to the output interface 312 is called operation instruction D32b.

[0256] In this example, output interface 312 is simply an interface that receives operation command D32a or operation command D32b and outputs it to the specified output target.

[0257] Next, the operation of the control system 3000a in this modified example will be explained. Figure 17 This is a flowchart illustrating an example of the operation of the control system 3000a.

[0258] exist Figure 17 In the example shown, firstly, the input interface 311 of the control system 3000a receives input information D31 (step S310). The received input information D31 is then input to the input determination unit 31.

[0259] Next, the input determination unit 31 determines whether the input information D31 matches the command rules for the operation of the target device 2 (step S321). Here, if it is determined that the input information D31 matches the command rules for the operation of the target device 2 (Yes in step S321), the input information D31 is input to the output interface 312 (proceeding to step S322). On the other hand, if it is determined that the input information D31 does not match the command rules for the operation of the target device 2 (No in step S321), the input information D31 is input to the learning model unit 300 (proceeding to step S311).

[0260] Processing of steps S311 to S313 Figure 15 The example shown is the same.

[0261] In step S322, the output interface 312 outputs the incoming input information D31 as an operation command D32b to the designated output target. Thus, the target device 2 performs an operation according to the operation command D32b.

[0262] Other aspects can be the same as other control systems involved in this embodiment.

[0263] As described above, according to this variation, when the input from user 1 matches the command rules for operating the target device 2, the target device 2 can be operated according to the input; conversely, when they do not match, the target device 2 can be operated using a learning model. Therefore, the efficiency of the operations involved in operating the target device 2 can be further improved.

[0264] Variation Example 3-2

[0265] Next, other variations of the control system 3000 will be described. In this variation, a learning model is used to generate coordinated operating instructions that include multiple inputs.

[0266] Figure 18 This is a structural diagram illustrating a modified example of the control system 3000 according to this embodiment, namely, control system 3000b. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0267] exist Figure 18 In the control system 3000b shown, the input interface 311 receives input information D31 from multiple users 1.

[0268] Input interface 311 receives input information D31 from multiple users 1 and inputs it into the learning model unit 300. At this time, input interface 311 may receive input information D31 with information of the user 1 who is the input source, or it may receive input information D31 after the input interface 311 has identified the user 1 who is the input source and added information of the input source, or it may receive input information without doing anything.

[0269] The learning model unit 300 can be any model and its operating environment configured such that, if input information D31 is received from the input interface 311, it outputs an operation instruction D32 corresponding to the input information D31. Alternatively, the learning model unit 300 can be a model and its operating environment configured such that, if input information D31 is received, it generates and outputs the operation instruction D32 based on the input information D31, device information D33, and other information that the learning model unit 300 can refer to.

[0270] For example, the learning model unit 300 can use a language learning model such as LLM, which takes natural language as input and outputs the result, to extract the optimal solution in the language space (more specifically, in the feature vector space containing information of the language space), thereby generating and outputting operation instructions D32 that represent a compromise solution for different operation contents represented by the input information D31. At this time, the learning model unit 300 can also refer to the history of the input information D31 for each user 1 as the input source and / or the history of the operation instructions D32 for each user 1 as the input source.

[0271] Other aspects can be the same as other control systems involved in this embodiment.

[0272] As described above, according to this modified example, even when multiple users input information related to different operation contents, the learning model unit 300 can be used to generate more appropriate operation instructions D32 after coordinating their contents, thus further realizing the high functionality of the operations involved in the operation of the object device 2.

[0273] Variation Example 3-3

[0274] Next, other variations of the control system 3000 will be described. In this variation, a learning model is used to generate the user interface for the operation screen.

[0275] Figure 19 This is a structural diagram illustrating a modified example of the control system 3000 according to this embodiment, namely, control system 3000c. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0276] Figure 19 The control system 3000c shown also has an operation screen user interface 3 (referred to as operation screen UI in the figure). In addition, the learning model unit 300 generates an operation screen as operation instruction D32 for actually performing the operation content corresponding to the input information D31 on the target device 2.

[0277] The operation screen generated by the learning model unit 300 can be, for example, an application programming interface (API) that receives user input along with a description of the operation content and outputs control commands D34, such as control codes, corresponding to the received operation input. Here, the output of control codes, etc., corresponding to the operation input also includes a method where multiple control commands D34 are output sequentially corresponding to one operation input. Alternatively, the operation screen can also be an API that has operation descriptions, operation input reception, and control command output corresponding to two or more different operation contents. For example, the learning model unit 300 can extract operation information representing two or more different operation contents as operation commands D32 corresponding to input information D31, and generate an API that receives operation inputs and outputs control commands corresponding to each operation information.

[0278] In addition, the operation screen generated by the learning model unit 300 can also be an operation screen that changes the display method of an existing operation screen by emphasizing the operation part corresponding to the corresponding operation content, displaying the operation function with restrictions, and changing the position and form (shape, size, color, etc.) of the UI components on the screen.

[0279] The operation screen user interface 3 displays the operation screen for the target device 2, and also receives user input related to the operation on this operation screen. The operation screen user interface 3 can be implemented, for example, by a controller having a touch panel display, operation buttons, and a display unit. Alternatively, the operation screen user interface 3 can also be implemented by a display device such as a monitor that is linked to an operation input device such as a mouse.

[0280] In addition, the output interface 312 in this variant will output the operation instruction D32 (operation screen) output from the learning model unit 300 to the operation screen user interface 3.

[0281] Furthermore, in this modified example, the learning model unit 300 may also have the function of confirming the operation expected by user 1 through dialogue. In the above case, for example, after prompting the operation screen as operation command D32, if the learning model unit 300 receives information requesting to retrieve operation command D32 again, it may retrieve operation command D32 again after changing a part of the input information, a part of the model parameters, or the reference target of the reference information.

[0282] As described above, in this modified example, the learning model unit 300 can generate operation screens that implement the desired operations (such as screen API construction or display mode changes) in a way that allows for simple or easy understanding, thus further improving the efficiency of operations related to the operation of the target device 2. Furthermore, according to this modified example, the user 1 can confirm the explanation of the operation instructions generated by the learning model unit 300 while performing the actual operation, thus enabling error-free operation.

[0283] Variations 3-4

[0284] Next, other variations of the control system 3000 will be described. In this variation, the learning model also uses environmental information to generate operating commands.

[0285] Figure 20 This is a structural diagram illustrating a modified example of the control system 3000 according to this embodiment, namely, the control system 3000d. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0286] Figure 20 The control system 3000d shown also has an environmental information storage unit 313 (referred to as environmental information DB in the figure).

[0287] The environmental information storage unit 313 stores information about the environment at the operating location of the object device 2, namely environmental information D33a. Environmental information D33a may also include information related to the space where the object device 2 operates. In this modified example, information related to objects or people existing in the space where the object device 2 operates, as well as user 1, the operator of the object device 2, are also part of the environment. Therefore, environmental information D33a may also include information related to those objects, people, or user 1.

[0288] Environmental information D33a may include, for example, information related to a person such as attributes, temperature, location, posture, and heart rate. Additionally, environmental information D33a may include, for example, information related to the location, temperature, humidity, and brightness of the space. Furthermore, environmental information D33a can also save information indicating changes in the aforementioned space- or person-related information. Here, information indicating changes is also referred to as time-series data or historical information. Environmental information D33a may, for example, be part of the model reference information D104 of the learning model.

[0289] Environmental information D33a can be obtained, for example, by sensors not shown.

[0290] The learning model unit 300 is, for example, a model and its operating environment configured in such a way that if input information D31 is input, an operation instruction D32 is generated and output based on the input information D31, device information D33, environment information D33a and other information that the learning model unit 300 can refer to.

[0291] As described above, according to this variant, the learning model can generate operation instructions D32 using environmental information D33a related to the space in which the object device 2 operates, thus enabling higher functionality of the operations involved in the operation of the object device 2.

[0292] Variations 3-5.

[0293] Next, other variations of the control system 3000 will be described. In this variation, two learning models are combined to generate operating instructions. Figure 21 This is a structural diagram illustrating a variant of the control system 4000 according to this embodiment, namely, the control system 3000e. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0294] exist Figure 21 In the control system 3000e shown, replacing Figure 20 The learning model unit 300 shown has a learning model unit 300a as a first learning model unit 300 and a learning model unit 300b as a second learning model unit 300.

[0295] If the learning model unit 300a is given input information D31, it outputs operation information D320. The learning model unit 300a may also be a model and its operating environment configured such that, if it is given input information D31, it generates and outputs operation information D320 based at least on the input information D31 and the environment information D33a.

[0296] Here, the operation information D320 includes information related to the operation of the target device 2, expressed in a prescribed form that can be determined by the subsequent learning model unit 330b. Here, the operation information D320 may also be information that supplements (including adding, modifying, or canceling) the operation content represented by the input information D31 in accordance with the environment information D33a. The operation information D320 may also be information that changes the operation content or its manifestation represented by the input information D31 to match the state of the space in which the target device 2 is driven. The learning model unit 300a may also be a model primarily grounded in relation to the input information D31.

[0297] For example, even if the desired operation is the same, differences can arise in language expression and cognitive approaches to phenomena depending on the environment in which the object device 2 is driven. For instance, the operation represented by input information D31 may sometimes differ depending on dialect or wording habits, the use of company or family language, or differences in senses such as hot / cold.

[0298] The learning model unit 300a, for example, has the function of absorbing and modifying the differences in language performance and / or differences in cognition regarding phenomena into more general or more specific content. The learning model unit 300a may also be a local learning model that obtains output results based on local information such as limitations in the database of the reference target.

[0299] The operation information D320 generated by the learning model unit 300a is input to the learning model unit 300b.

[0300] The learning model unit 300b can be basically the same as the learning model unit 300 described above. However, instead of the input information D31, the operation information D320 generated by the learning model unit 300a is input.

[0301] If operation information D320 is input, the learning model unit 300b outputs operation instruction D32. The learning model unit 300b can also be a model and its operating environment configured such that, if operation information D320 is input, it generates and outputs operation instruction D32 based on the operation information D320, device information D33, and other information that the learning model unit 300b can refer to. The learning model unit 300b can also be a global learning model that obtains output results based on global information that can freely access external networks, etc.

[0302] Figure 22 This is a flowchart illustrating the action example of this variation. Figure 22 In the example shown, if the input interface 311 of the control system 3000e receives input information D31 in step S310, it inputs the input information D31 to the learning model unit 300a.

[0303] Next, the control system 3000e performs the processing of generating operation information D320 using the learning model unit 300a (step S331). In step S331, the learning model unit 300a (more specifically, the model control unit 101) generates and outputs operation information D320 corresponding to the input information D31 based on the model information D102, the input information D31, and the model reference information D104 including the environmental information D33a as needed. The operation information D320 output from the learning model unit 300a is input to the learning model unit 300b.

[0304] Next, the control system 3000e performs the process of generating the operation instruction D32 using the learning model unit 300b (step S332). In step S332, the learning model unit 300b (more specifically, the model control unit 101) generates and outputs the operation instruction D32 corresponding to the operation information D320 based on the model information D102, the input operation information D320, and the model reference information D104 including the device information D33 as needed.

[0305] The subsequent processing can be the same as that of other control systems involved in this embodiment.

[0306] As described above, according to this modified example, operation instructions D32 can be generated after absorbing the differences in language expression and / or the differences in cognition of phenomena in the input information D31 input from user 1 and changing it into more general or more specific content. Therefore, it is possible to further realize the high functionality of the operation involved in the operation of object device 2.

[0307] Furthermore, even in the structure shown in variations 3-4, the learning model unit 300 can generate operation instructions D32 that equalize the differences in language expression and / or the differences in cognition of phenomena, based on model reference information D104, including environmental information D33a, device information D33, and past operation history. However, according to this variation, the role of the learning model can be clearly distinguished in terms of absorbing differences in expression and transforming them into operation instructions. Therefore, the learning model can focus on learning in this way, and a compact design can be achieved by suppressing the learning scale.

[0308] Implementation Method 4

[0309] Next, this fourth embodiment will be described. In this embodiment, an example will be described of a task involving the monitoring of a certain task status using a learning model.

[0310] For example, consider monitoring anomalies in a factory's Factory Automation (FA) system, which includes control equipment such as robots and PLCs. While existing monitoring algorithms based on rule bases can handle obvious setup errors in the workpiece being controlled, there are also scenarios where minor setup errors could trigger anomalies in subsequent processes. In such cases, even analysis triggered by anomaly detection may not accurately capture the situation or provide solutions.

[0311] In this embodiment, by assisting the work involved in monitoring the work environment, the monitoring work can be made more efficient and higher efficient. In this work environment, it is conceivable that a small problem may escalate into a large anomaly, or that the situation may not conform to existing rules and the cause may be difficult to find.

[0312] Figure 23 This is a structural diagram illustrating an example of the control system 4000 involved in Embodiment 4. Figure 23 The control system 4000 shown is a control system for monitoring specific operating conditions using a learning model. It includes a sensor 5, a learning model unit 400a, a learning model unit 400b, an equipment information storage unit 410 (referred to as equipment information DB in the figure), a model interface 6 (referred to as model IF in the figure), and a display 7.

[0313] Sensor 5 acquires data indicating the status of the operation being monitored. Hereinafter, the data acquired by sensor 5 will be referred to as sensor data. Sensor data may, for example, be image data obtained by capturing images of the operation being monitored. Alternatively, sensor data may also be audio data obtained by recording audio of the operation being monitored. Furthermore, sensor data may also be measurement data obtained by measuring the position and other states of the person or object performing the operation being monitored.

[0314] The acquisition of sensor data performed by sensor 5 is always performed, but it can also be based on a trigger provided by a person or other monitoring system. The sensor data acquired by sensor 5 is input as input information D41 to the learning model unit 400a. Alternatively, the sensor data itself, which is set as input information D41, can be provided from a person or other monitoring system. In the above case, sensor 5 can be omitted.

[0315] If the learning model unit 400a is given input information D41, it outputs the parsing result D42a. For example, if the learning model unit 400a is given input information D41, it outputs the parsing result D42a based on the model information D102. The structure of the learning model unit 400a can be substantially the same as that of the learning model unit 100 in Embodiment 1.

[0316] In this embodiment, the learning model unit 400a is a model and its operating environment configured such that if input information D41 is input, it outputs a parsing result D42a corresponding to the input information D41. Alternatively, the learning model unit 400a may also be a model and its operating environment configured such that if input information D41 is input, it generates and outputs a parsing result D42a based on the input information D41, the equipment information D43, and other information that the learning model unit 400a can refer to (such as model reference information D104). Here, the learning model unit 400a may also refer to and utilize information related to the operation of the monitored object as model reference information D104. Information related to the operation of the monitored object may include, for example, information indicating the location, personnel, objects, process, and conditions of the operation. The learning model unit 400a may, for example, use information digitized from a manual that records the conditions, setup environment, and operation process of the equipment used in the operation as model reference information D104.

[0317] In this embodiment, the input information D41 includes information indicating the status of the tasks being monitored. Here, the tasks being monitored include one or more tasks performed by a person or device. The input information D41 may be, for example, a measurement, image, sound, or a combination thereof indicating the status of the tasks being monitored. The input information D41 may also be, for example, a measurement, image, sound, or a combination thereof indicating the status of multiple tasks being monitored. Additionally, the input information D41 may include information indicating the status of tasks performed continuously in time. In this case, it may be time-series data constructed from predetermined data including the aforementioned measurement, image, sound, or combination thereof indicating the status. The way the task status is represented is based on matching the input format of the model used by the learning model unit 400a, but is not limited to this if the learning model unit 400a includes error processing, correction processing, or transformation processing in its preceding stages.

[0318] The parsing result D42a contains information representing the result of parsing the work situation represented by the input information D41. This information can also represent objects (environment) present in the work situation represented by the input information D41 and / or the phenomena they cause. The information representing the parsing result can be described as an explanation of the work situation represented by the input information D41. For example, the parsing result D42a can be text representing an explanation of the work situation represented by the input information D41. Alternatively, the parsing result D42a can also be text explaining a part of the work situation represented by the input information D41 that differs from the normal situation. Furthermore, the form of the parsing result D42a can be other than text. If the parsing result D42a is described in a prescribed form that can be determined by the subsequent learning model unit 400b, the form is not particularly limited; for example, it can be text, an image, sound, or a combination of these.

[0319] As an example of interpreting a task situation, examples include representing objects existing in that task situation using their attributes, representing phenomena occurring in that task situation using prescribed grammatical forms such as 5W1H or 7W1H, or further creating a summary after the above-mentioned concretization. In addition, examples include decomposing and interpreting the task in that task situation from multiple perspectives and then representing it for each perspective; and when the task in that task situation contains multiple sub-tasks or steps, decomposing the task into sub-task units or step units and explaining each sub-task or step. The parsing result D42a can be said to be the result of further adding a prescribed form of representation when concretizing, refining, and / or extracting singularities from the task situation represented by the input information D11, as described above. As mentioned above, in the parsing result D42a, the task situation is easy to understand and is represented in a sorted state.

[0320] If the learning model unit 400b is input with the analysis result D42a, it outputs the analysis result D42b. For example, if the learning model unit 400b is input with the analysis result D42a, it outputs the analysis result D42b based on the model information D102. The structure of the learning model unit 400b can be substantially the same as that of the learning model unit 100 in Embodiment 1.

[0321] In this embodiment, the learning model unit 400b is a model and its operating environment configured such that if a parsing result D42a is input, it outputs a parsing result D42b corresponding to the parsing result D42a. Alternatively, the learning model unit 400b may also be a model and its operating environment configured such that if a parsing result D42a is input, it generates and outputs a parsing result D42b based on the parsing result D42a, device information D43, and / or information that can be referenced in the learning model unit (such as model reference information D104).

[0322] The analysis result D42b contains information representing methods for improving the work status, derived from the analysis results of the work status of the learning model unit 400a. This information can be a recovery method for restoring an abnormal state to normal, or it can be information representing solutions to problems that occur in the environment (work environment) where the work being monitored is being performed, such as when a person is fatigued or equipment stops.

[0323] Information representing the improvement method can be, for example, text, images, or sounds representing the method, or control instructions (e.g., commands, control signals, control codes, etc.) for the device (object device 2) that is designated as the implementer of the method, flow documents describing the method, timing diagrams, source code, executable code, or controller commands used to cause the controller to execute the method. Information representing the improvement method can also be text, images, sounds, data described in a specified design language, control descriptions (including information described in source code and a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), executable code, and combinations of two or more of these elements. Specified design languages ​​include, for example, UML (Unified Modeling Language), but are not limited to these.

[0324] Hereinafter, the learning model unit 400a is sometimes referred to as the first learning model unit 400, and the analysis result D42a is sometimes referred to as the first analysis result D42. In addition, the learning model unit 400b is sometimes referred to as the second learning model unit 400, and the analysis result D42b is sometimes referred to as the second analysis result D42.

[0325] As already explained, the parsing result D42a contains information representing the status parsing result of the work status expressed by the input information D41. Therefore, the learning model unit 400b can also be a model and its operating environment configured in such a way that it outputs the parsing result D42b corresponding to the status parsing result expressed by the parsing result D42a. Here, when the information representing the status parsing result is text explaining the work status expressed by the input information D41, the learning model unit 400b can also be a model and its operating environment configured in such a way that it outputs the parsing result D42b corresponding to the text explaining the work status.

[0326] The processing of equipment information storage unit 410 and equipment information D43 is basically the same as that of equipment information storage unit 110 and equipment information D13 in embodiment 1. Furthermore, in this embodiment, equipment information storage unit 410 stores information about equipment associated with the operation being monitored, i.e., equipment information D43, as object equipment 2. Here, equipment associated with the operation broadly includes equipment required for deriving the aforementioned situation analysis and improvement method. More specifically, this includes not only equipment used in the operation but also equipment that affects the person or equipment performing the operation. Equipment that affects the person or equipment performing the operation can more specifically include equipment that directly or indirectly causes changes to the person or equipment performing the operation. Examples include equipment directly used in the operation (including various machines such as processing machines and conveyors, as well as tools such as workbenches and cutting tools), equipment that controls the equipment directly used in the operation (power supplies, relays, switches, controllers, etc.), and equipment that changes the working environment (lighting equipment, air conditioning equipment, vacuum cleaners, purifiers, etc.).

[0327] Device information D43 is used, for example, as supplementary information when the learning model unit 400a and / or the learning model unit 400b outputs model output data D103 (parse result D42a, parsing result D42b). Hereinafter, in this embodiment, in particular, the information indicating the state of the object device 2 is sometimes referred to as state information D45.

[0328] In this embodiment, the learning model unit 400a may be an image learning model such as VLM (Visual Learning Model) that takes an image as input and obtains an output result, and its operating environment. Alternatively, the learning model unit 400a may be a multi-model that takes natural language and an image as input and obtains an output result, and its operating environment. Similarly, the learning model unit 400b may be a language learning model such as LLM (Language Learning Model) that takes natural language as input and obtains an output result, and its operating environment. In this case, the input information D41 may be input as text data, image data, a combination of text data and image data, or data forms that can be transformed into them (sound data, a combination of sound data and image data, i.e., video, etc.). Furthermore, the learning model used in this embodiment is not limited to the models described above.

[0329] Model interface 6 is an interface that outputs model output data (parsed result D42a and parsed result D42b) to a specified output target if model output data (parsed result D42a and parsed result D42b) is received from learning model unit 400a and learning model unit 400b. Model interface 6 may also be an interface that transforms the model output data output from learning model unit 400a and learning model unit 400b into data that matches the specified output target. Model interface 6 may also be provided as an example of the output unit 103 described above. In this embodiment, the object device 2 and the display 7 are included as the output targets of model interface 6.

[0330] For example, model interface 6 can also output the result information D44a, representing the situation analysis result contained in analysis result D42a and the improvement method contained in analysis result D42b, to display 7, and output the result information D44b, representing the improvement method contained in analysis result D42b, to object device 2. In this case, model interface 6 can also extract a portion of data from analysis result D42a and / or analysis result D42b, transform it into a data format matching the output target, and output it as result information D44a and result information D44b.

[0331] In addition, Figure 23 In the example shown, the object device 2 and the display 7 are shown as the output targets of the model interface 6, but the output targets of the model output data are not limited to the above. For example, if the method for improving the state of the model output data set as the output target includes information indicating control for the object device 2, the model interface 6, for example, can output the model output data or the information indicating the method directly to the object device 2, which is the implementer of the method, and also output the model output data or the information indicating the method to a conversion device (not shown) that can convert information acceptable to the object device 2. This conversion device may be, for example, the control system 1000 of Embodiment 1, which converts the input information into control descriptions or execution code that the object device 2 can determine.

[0332] Furthermore, the model interface 6 itself may also function as a transformation device. For example, the model interface 6 may not only control the output of model output data, but also transform the improvement method output by the learning model unit 400b into code that can be executed by an interpreter, output the transformed code, or control the device based on the code. Additionally, the model interface 6 may control the processing flow by immediately executing processes with high urgency, such as those included in the improvement method. Furthermore, the model interface 6 may transmit prompts input via the display 7, such as responses to suggestions on the methods displayed on the display 7, to the learning model unit 400b.

[0333] Furthermore, the model interface 6 may also have the functions of the output confirmation unit 202 and the correction confirmation unit 203 described above. For example, the model interface 6 can determine the urgency of the parsed situation. If the urgency is determined to be low, it can confirm the appropriateness of the improvement method by querying the monitor or using a simulator. If the improvement method is inappropriate, it can transmit this message to the learning model unit 400b and prompt the output of the improvement method again. At this time, the model interface 6 can also issue supplementary information D48 based on the model input data of the object's learning model unit.

[0334] In this embodiment, input information D41 corresponds to model input data D101 of learning model unit 400a. Furthermore, parsing result D42a corresponds to model output data D103 of learning model unit 400a. Additionally, parsing result D42a corresponds to model input data D101 of learning model unit 400b. Furthermore, parsing result D42b corresponds to model output data D103 of learning model unit 400b. Learning model unit 400a (especially model control unit 101) can, for example, be configured to, upon receiving input information D41, output parsing result D42a corresponding to input information D41 based on model information D102 and, as needed, based on model reference information D104. Similarly, learning model unit 400b (especially model control unit 101) can, for example, be configured to, upon receiving parsing result D42a, output parsing result D42b corresponding to parsing result D42a based on model information D102 and, as needed, based on model reference information D104.

[0335] Furthermore, in the above-described case, the model generation unit 107 corresponding to the learning model unit 400a can, for example, use model learning data D105 containing candidates of input information D41 that can be input to the model control unit 101 to perform machine learning and generate or update model information D102. Alternatively, it can use model learning data D105 containing candidates of input information D41 that can be input to the model control unit 101 and candidates of corresponding parsing results D42a to perform machine learning and generate or update model information D102. Similarly, the model generation unit 107 corresponding to the learning model unit 400b can, for example, use model learning data D105 containing candidates of parsing results D42a that can be input to the model control unit 101 to perform machine learning and generate or update model information D102. It can also use model learning data D105 containing candidates of parsing results D42a that can be input to the model control unit 101 and candidates of corresponding parsing results D42b to perform machine learning and generate or update model information D102.

[0336] Although figures are omitted, in this embodiment, status information D45 and / or feedback information D46 can be obtained from the model output data D103 of learning model units 400a and 400b and / or the output target based on the information generated therefrom. The control system 4000 can, for example, output the obtained status information D45 and / or feedback information D46 as information representing the control result to the user, learning model units 400a, 400b, or other devices not shown. Furthermore, the control system 4000 can also be configured to send an inquiry D47 back to the user if the input information D41 contains ambiguous or uncertain information. Additionally, the control system 4000 can also generate supplementary information D48 for the input and output data of learning model units 400a and 400b based on the obtained status information D45 and / or feedback information D46, and send it to the user, learning model units 400a, 400b, or other devices not shown. The processing of status information D45, feedback information D46, inquiry D47, and supplementary information D48 can be substantially the same as in embodiment 1. Here, information output to the user can be performed, for example, via the display 7 or the input / output interface of the information processing device 10 (not shown).

[0337] Additionally, the control system 4000 may also include a status acquisition unit 430 (not shown), which acquires status information D45 and / or feedback information D46, and issues supplementary information D48 as needed. The status acquisition unit 430 is the same as the status acquisition unit 130 in Embodiment 1.

[0338] In this embodiment, the target device 2 is not particularly limited. Furthermore, it is assumed that the target device 2 is a device capable of receiving the parsing result D42b and actually performing control, but it is not limited to this if the above-described conversion device is included between the target device 2 and the target device 2.

[0339] In this embodiment, the input information D41 received by the control system 4000 can be considered as information related to the conditions in the working environment (here, the conditions in the environment where the monitoring operation is performed). Therefore, the input information D41 received by the control system 4000 can be considered as an example of first information representing the conditions in the working environment. Furthermore, the parsing results D42a and D42b, corresponding to the aforementioned input information D41, can be considered as information used in this operation (monitoring operation). Hereinafter, the parsing results D42a and / or D42b, output from the operating environment of the learning model, which is input based on the model input data of the input information D41, to the specified output target, will be referred to as second information.

[0340] Next, the operation of the control system 4000 in this embodiment will be explained. Figure 24 This is a flowchart illustrating an example of the operation of the control system 4000.

[0341] exist Figure 24 In the example shown, firstly, the control system 4000 receives input information D41 (step S410). For example, the input unit 102 or the input processing unit 201 described above can receive input information D41. The received input information D41 is input to the learning model unit 400a as model input data D101.

[0342] Next, the control system 4000 performs a process to generate the parsing result D42a using the learning model unit 400a (step S411). In step S411, the learning model unit 400a (more specifically, the model control unit 101) outputs the parsing result D42a corresponding to the input information D41 based on the model information D102, the input information D41, and, as needed, the model reference information D104 including the device information D43. For example, the learning model unit 400a can use a learning model capable of generating text data to generate the parsing result D42a of text data based on the input information D41.

[0343] In step S411, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 400a may further perform the above-mentioned processing.

[0344] The parsing result D42a output from the learning model unit 400a is input to the learning model unit 400b. Additionally, the parsing result D42a output from the learning model unit 400a is input to both the learning model unit 400b and the model interface 6. The parsing result D42a output from the learning model unit 400a can also be input to the model interface 6 via the learning model unit 400b. In this case, the learning model unit 400b can also output model output data D103 containing both the parsing result D42a and the parsing result D42b.

[0345] Next, the control system 4000 performs a process to generate the parsing result D42b using the learning model unit 400b (step S412). In step S412, the learning model unit 400b (more specifically, the model control unit 101) outputs the parsing result D42b corresponding to the parsing result D42a based on the model information D102, the input parsing result D42a, and, as needed, the model reference information D104 including device information D43. For example, the learning model unit 400b can use a learning model capable of generating text data to generate the parsing result D42b of binary data based on the input parsing result D42a. Alternatively, the learning model unit 400b can also use a learning model capable of generating both text data and binary data to generate the parsing result D42b of both text data and binary data based on the input parsing result D42a.

[0346] In step S412, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 400b may further perform the above-mentioned processing.

[0347] The parsing result D42b output from the learning model unit 400b is, for example, input to the model interface 6.

[0348] Model interface 6 controls the target device 2 and / or causes the display 7 to display information based on the analysis results obtained by learning model unit 400a and learning model unit 400b (step S413). In step S413, for example, model interface 6 outputs information based on analysis results D42a and D42b to a predetermined output target. For example, based on analysis results D42a and D42b, model interface 6 outputs the result information D44a of the representation status analysis result and the improvement method to the display 7, and outputs the result information D44b of the representation improvement method based on analysis result D42b to the target device 2.

[0349] Result information D44a can, for example, express the situation occurring in the work environment and the improvement methods through text and sound. Additionally, result information D44b can, for example, express the improvement methods through text or control signals.

[0350] Therefore, based on the result information D44a, the display 7 shows the status analysis result represented by the analysis result D42a and the improvement method represented by the analysis result D42b, and the object device 2 implements the improvement method represented by the analysis result D42b based on the result information D44b. The information input to the display 7 and the object device 2 can be directly input from the control system 4000 (more specifically, the model interface 6), or indirectly input via a communication network, other devices (servers, various conversion devices, etc.) or by human touch.

[0351] The control system 4000 can also acquire status information D45 and feedback information D46 (step S414) when controlling the target device 2, etc., and when the state of the target device 2 changes and there is feedback from the output target. In addition, the processing of step S414 is not necessary and can be omitted appropriately.

[0352] The control system 4000 may also repeat the processing of steps S410 to S414 multiple times (for example, until the desired state is achieved in the object's operating environment).

[0353] As described above, according to this embodiment, different learning models are used in two stages to grasp the situation and obtain improvement methods, thereby improving the accuracy of the final product. As a result, the efficiency of the work involved in monitoring the work status can be improved.

[0354] For example, in scenarios where the situation is being assessed, it becomes important to broadly detect abnormal conditions in the work environment, such as "what strange thing has happened." On the other hand, in scenarios where improvement methods are being developed, specific information, such as "stop the machine, move the workpiece to location A, and restart the machine after restoring it to state B," becomes important.

[0355] When the level of abstraction of the information to be extracted (i.e., the target information) varies, attempting to learn and extract it all using a single learning model may reduce the accuracy of the output. In particular, when seeking improvement methods, specific approaches based on knowledge and information about the operational environment are required. In the above cases, providing appropriate industry knowledge (environmental information) by separating the learning models can reliably improve output accuracy.

[0356] Furthermore, the hallucination problem becomes significant when attempting to obtain solutions for different tasks—state mastery and improvement method acquisition—using a single learning model. This is because the ability to adjust the solution for one task (state mastery) in a way that makes the solution for one task (improvement method acquisition) appear correct may implicitly operate within the model algorithm. According to this embodiment, the hallucination problem is also addressed. That is, by separating the learning models corresponding to the two tasks of state mastery and improvement method acquisition, modalities incorporated into each learning model can be suppressed. As a result, the magnitude of the hallucination can be reduced, thus improving the accuracy of the final product.

[0357] Furthermore, in this embodiment, the output results of the learning model unit 400a and the learning model unit 400b, namely the analysis results D42a and D42b, can be verbally displayed on the display 7. Therefore, by confirming the content, a person can suppress hallucinations and more reliably implement the method for improving the situation.

[0358] Furthermore, the control system 4000 of this embodiment can be applied not only to the monitoring of the control system of the equipment in the factory, but also, for example, to the monitoring of logistics objects in a logistics system.

[0359] Variation Example 4-1

[0360] Next, a variation of the control system 4000 will be described. Figure 25 This is a structural diagram showing a modified example of the control system 4000 according to this embodiment, namely, control system 4000a. Furthermore, elements identical to those in the control system 4000 are labeled with the same reference numerals and their descriptions are omitted.

[0361] exist Figure 25 The control system 4000a shown differs from the control system 4000 in that it has two analysis units that analyze and improve the situation using different methods, and the analysis unit used is switched appropriately according to the situation that occurs.

[0362] exist Figure 25 The control system 4000a shown includes a first analysis unit 41-1, which uses the learning model unit 400a and the learning model unit 400b to obtain the method for analyzing and improving the situation. It also includes a second analysis unit 41-2, a switching unit 42, and an output switching switch 43.

[0363] The second parsing unit 41-2 is not particularly limited; it can be any unit that analyzes the situation and obtains an improvement method for the input information D41 using a different method than the first parsing unit 41-1. As an example, the second parsing unit 41-2 could be a unit that analyzes the situation and obtains an improvement method using a rule base. The second parsing unit 41-2 could also, for example, determine whether the input information D41 matches a predetermined pattern of an anomaly if it is input, and if it matches a pattern of an anomaly, obtain an improvement method corresponding to that pattern. The second parsing unit 41-2 outputs a parsing result D42c that includes at least the situation improvement method.

[0364] The analysis result D42c may, for example, contain information equivalent to the aforementioned result information D44a and information equivalent to result information D44b. In this modified example, the analysis result D42c at least contains result information D44b representing the improved method obtained through the second analysis unit 41-2.

[0365] In this modified example, the second analysis unit 41-2 can be implemented as an internal execution module, for example, by installing a PLC, information processing device, or the like in the operating environment.

[0366] The switching unit 42 is a unit that switches the controlled object corresponding to the input information D41 according to predetermined conditions. In this modified example, the switching unit 42 switches the controlled object corresponding to the input information D41 between the first analysis unit 41-1 and the second analysis unit 41-2. For example, the switching unit 42 can switch the controlled object corresponding to the input information D41 based on whether the input information D41 matches an existing rule. In this case, the switching unit 42 can also switch the output target of the input information D41 to the second analysis unit 41-2 if the input information D41 matches an existing rule, and switch the output target of the input information D41 to the first analysis unit 41-1 if they do not match, thereby switching the controlled object.

[0367] The switching unit 42 can, for example, switch the controlled object corresponding to input information D41 according to instructions from the monitor. Alternatively, the switching unit 42 can switch the controlled object corresponding to input information D41 based on factors such as time, work content, or the presence or absence of a monitor. Furthermore, the switching unit 42 can also switch the controlled object corresponding to input information D41 based on whether an anomaly has occurred in the work environment. Here, whether an anomaly has occurred in the work environment can be determined, for example, by whether an anomaly signal has been generated. For example, in the event of an anomaly, the switching unit 42 can switch the controlled object corresponding to input information D41 to the first analysis unit 41-1. Additionally, the switching unit 42 can also switch the controlled object corresponding to input information D41 based on the degree or urgency of the anomaly occurring in the work environment.

[0368] In addition, the switching unit 42 can also control the output switching switch 43 in conjunction with the switching of the controlled object corresponding to the input information D41. The output switching switch 43 switches the connection path (circuit or communication line, etc.) that connects the output of the first analysis unit 41-1 or the output of the second analysis unit 41-2 to the object device 2 and the display 7 that are set as the output target of the analysis result.

[0369] For example, when switching the controlled object corresponding to the input information D41 to the first analysis unit 41-1, the switching unit 42 can control the output switching switch 43 to connect the output of the first analysis unit 41-1 to the object device 2 and the display 7, and disconnect the output of the second analysis unit 41-2 from the object device 2 and the display 7. Similarly, when switching the controlled object corresponding to the input information D41 to the second analysis unit 41-2, the switching unit 42 can control the output switching switch 43 to connect the output of the second analysis unit 41-2 to the object device 2 and the display 7, and disconnect the output of the first analysis unit 41-1 from the object device 2 and the display 7.

[0370] Figure 26 This is a flowchart illustrating the action example of this variation. Figure 26 In the example shown, if the control system 4000a receives input information D41 in step S410, the switching unit 42 switches the controlled object corresponding to the input information D41 according to predetermined conditions (step S421). Figure 26 In the example shown, the switching unit 42 determines whether the input information D41 matches an existing rule. If it determines that the input information D41 does not match (No in step S421), it proceeds to the first parsing process (step S422). On the other hand, if it determines that the input information D41 matches an existing rule (Yes in step S421), it proceeds to the second parsing process (step S423).

[0371] In the first analysis process of step S422, the learning model unit 400a and learning model unit 400b, which are the first analysis units 41-1, analyze the situation and obtain the improvement method. The learning model unit 400a and learning model unit 400b output the analysis result D42a, which includes the analysis result of the situation, and the analysis result D42b, which includes the improvement method of the situation, as the result of the first analysis process.

[0372] In the second analysis process of step S423, the second analysis unit 41-2 analyzes the situation and obtains the improvement method according to existing rules. For example, the second analysis unit 41-2 outputs an analysis result D42c, which includes at least the situation improvement method, as the result of the first analysis process.

[0373] If the result of the parsing process obtained by the first parsing unit 41-1 or the second parsing unit 41-2 is output, the target device 2 is controlled and / or the display 7 is made to display information based on the result of the arbitrary parsing process, corresponding to the state of the output switching switch 43 (step S424).

[0374] In this example, when the first parsing unit 41-1 performs parsing processing, the connection path connecting the output of the first parsing unit 41-1 to the target device 2 and the display 7 is activated. In this case, the model interface 6 can, for example, output the result information D44a representing the state parsing result and the improvement method to the display 7 based on the parsing result D42a and the parsing result D42b, and output the result information D44b representing the improvement method based on the parsing result D42b to the target device 2. On the other hand, when the second parsing unit 41-2 performs parsing processing, the connection path connecting the output of the second parsing unit 41-2 to the target device 2 and the display 7 is activated. In this case, the result information D44a representing the state parsing result and the improvement method can also be output to the display 7 based on the parsing result D42c output from the second parsing unit 41-2, and / or the result information D44b representing the improvement method can be output to the target device 2.

[0375] The result information D44b can be displayed on the monitor 7 in a manner that allows the operator to confirm it. In this case, the operator can also refer to the result information D44b displayed on the monitor 7 to confirm the improvement method shown in the result information D44b and implement the work involved in that method. Alternatively, the operator can also confirm the improvement method shown in the result information D44b and judge its appropriateness. At this time, if the improvement method shown in the result information D44b is inappropriate, the operator can prompt the learning model unit 400b to obtain other improvement methods (re-acquiring the model output data). For example, if the learning model unit 400b receives information requesting the re-acquisition of model output data, it can re-acquire the model output data after making changes to a portion of the input information, a portion of the model parameters, or the reference target of the reference information.

[0376] The subsequent processing can be the same as that of other control systems involved in this embodiment.

[0377] As described above, in this modified example, the system is configured with multiple analysis units that analyze the situation and obtain improvement methods using different approaches, and these analysis units are switched according to the situation. Therefore, control that is more closely matched to the situation can be achieved. For example, for problems with clear causes, the second analysis unit, which has a high processing load, can immediately analyze the situation and suggest and execute improvement methods; for problems with unclear causes, the first analysis unit, which uses a learning model, can analyze complex situations and suggest and execute better improvement methods.

[0378] Furthermore, while the above example illustrates how the first analysis unit 41-1 uses two learning models to analyze the situation and obtain an improvement method, the structure of the first analysis unit 41-1 is not limited to this example. For instance, if situation analysis is not required, the learning model unit 400a can be omitted. Similarly, if obtaining an improvement method is not required, the learning model unit 400b can be omitted. Alternatively, situation analysis and the acquisition of an improvement method can also be performed using only one learning model unit.

[0379] For example, if the first analysis unit 41-1 includes a learning model unit 400b that acquires a method for improving the situation based on input information D41, the switching unit 42 can switch the controlled object corresponding to the input information D41 to the first analysis unit 41-1 in case of an anomaly. In this case, the learning model unit 400b of the first analysis unit 41-1 is configured to output information indicating the improvement method corresponding to the anomaly situation indicated by the input information D41 if the input information D41 is input. At this time, the learning model unit 400b can also output information indicating the improvement method corresponding to the situation by referring to the device information storage unit 410 accessible by the control system.

[0380] Implementation Method 5

[0381] Next, this fifth embodiment will be described. In this embodiment, an example will be described of using a learning model to assist in response operations in call centers or product websites to information posted by users. Here, the information posted by users may include inquiries or opinions related to a certain service, information, phenomenon, or item.

[0382] Figure 27 This is a structural diagram illustrating an example of the control system 5000 according to Embodiment 5. Figure 27The control system 5000 shown includes a learning model unit 500, a reference information storage unit 12, a database retrieval unit 511 (denoted as DB retrieval unit in the figure), a control generation unit 512, a speech recognition unit 513v, and a speech synthesis unit 514v. Here, the reference information storage unit 12, the database retrieval unit 511, and the control generation unit 512 can also be equipped as part of the learning model unit 500.

[0383] If the learning model unit 500 receives input information D51, it outputs response information D52 representing the response content. For example, if the learning model unit 500 receives input information D51, it outputs response information D52 based on model information D102. The structure of the learning model unit 500 can be substantially the same as that of the learning model unit 100 in Embodiment 1.

[0384] In this embodiment, the learning model unit 500 is a model and its operating environment configured such that if input information D51 is input, it outputs response information D52 corresponding to the input information D51. Alternatively, the learning model unit 500 may also be a model and its operating environment configured such that if input information D51 is input, it generates and outputs response information D52 based on the input information D51 and other information that the learning model unit 500 can refer to.

[0385] In this embodiment, input information D51 includes information representing content sent from user 1, etc. Input information D51 may also include information representing content requesting a response in the working environment. Input information D51 may, for example, be text, images, sounds, or combinations thereof representing inquiries or opinions related to a service, information, phenomenon, or item. Input information D51 may also, for example, be text, images, sounds, or combinations thereof representing multiple inquiries or opinions related to a service, information, phenomenon, or item. Furthermore, input information D51 may also include information representing content sent consecutively in time; in this case, it may be time-series data containing prescribed data structures including text, images, sounds, or combinations thereof representing sent content as described above. The representation of the sent content is based on matching the input form of the model used by the learning model unit 500, but is not limited to this if the learning model unit 500 has error handling, correction processing, or transformation processing in its preceding stages.

[0386] Response information D52 contains information indicating a response to the content sent in input information D51. Response information D52 may, for example, be information indicating a response to an inquiry or comment related to a service, information, phenomenon, or item as shown in the content sent in input information D51.

[0387] The reference information storage unit 12 stores model reference information D104 referenced by the model control unit 101 of the learning model unit 500 for outputting response information D52. Model reference information D104 may include, for example, information related to services, information, phenomena, or items that can be included in the input information D51. Here, the reference information storage unit 12 may also specifically store information related to a particular service, information, phenomenon, or item as model reference information D104. Model reference information D104 may, for example, include information digitized from the response manual. Additionally, model reference information D104 may, for example, include the history of previously input input information D51 or the transmission content it contains. In this case, the reference information storage unit 12 may also store information about the sending source user 1 (e.g., user identifier, user attribute information, etc.) and previously input input information D51 or history information indicating the transmission content it contains as model reference information D104. Hereinafter, in this embodiment, information indicating the state of the sending source user 1 is sometimes referred to as state information D55.

[0388] The database retrieval unit 511 is a retrieval engine that references the information storage unit 12 and other databases. In response to requests from the model control unit 101 of the learning model unit 500, the database retrieval unit 511 searches for databases accessible to it and outputs retrieval results. At this time, the database retrieval unit 511 can also restrict the databases that can be accessed.

[0389] The control generation unit 512 is an interface used to set the preconditions for generating model output data by the learning model unit 500 (especially the model control unit 101). The control generation unit 512 may be, for example, an interface used by the learning model unit 500 to identify information as a control object and / or to set the output tendency. Here, control object information refers to information representing the object that becomes the control focus in the model control unit 101. The model control unit 101 may be configured to generate model output data D103 based on the control object information represented by the control generation unit 512, according to the model input data D101. The control generation unit 512 may, for example, identify a portion of the model input data input by the user as control object information, identify information generated by the model control unit 101 as control object information, or identify information generated by the model control unit 101 and corrected by other control units as control object information. The control object information and / or the output tendency setting may be specified by the user, by an external processing unit, or by the control generation unit 512 according to a predetermined algorithm.

[0390] When the input from user 1 includes audio input information D51v, the speech recognition unit 513v recognizes the speech represented by the input information D51v, transforms it into a form that matches the data form of the learning model unit 500, and outputs it. For example, the speech recognition unit 513v can transform the audio input information D51v into text input information D51. The speech recognition unit 513v receives the input information D51v from user 1 (the sending source) and inputs it as input information D51 to the learning model unit 500; therefore, it can be considered an example of the aforementioned input interface 311.

[0391] The speech synthesis unit 514v converts the content represented by the response information D52 into sound form and outputs it. For example, if the response information D52 from the learning model unit 500 contains data in a form other than sound, the speech synthesis unit 514v converts that part of the content represented by the response information D52 into sound form and outputs it. For example, if the response information D52 contains a specified data structure in a data form, the speech synthesis unit 514v can also convert data elements specified in the data structure in the data form into sound form and output them. For example, the speech synthesis unit 514v can also convert the response information D52 in text form into response information D52v in sound form. The speech synthesis unit 514v receives the response information D52 generated by the learning model unit 500 and outputs it as response information D52v to the user 1, which is the sending source, thus it can be considered an example of the output interface 312 described above.

[0392] Furthermore, in the above example, an example of using audio data in the input and output of user 1 was shown, but the data format used in the input and output of user 1 is not limited to audio. In this case, it is sufficient to replace the speech recognition unit 513v and the speech synthesis unit 514v with a processing unit that converts the data format used for input from user 1 into the data format used for input to the learning model unit 500, and a processing unit that converts the data format used for output from the learning model unit 500 into the data format used for input to user 1.

[0393] Furthermore, if the learning model unit 500 can accept the data format used for input from user 1, the speech recognition unit 513v can be omitted. Furthermore, if user 1 can accept the data format used for output from the learning model unit 500, the speech synthesis unit 514v can be omitted.

[0394] Furthermore, if the structure of the speech recognition unit 513v can be omitted, the input information received from the user 1, which is the sending source, is directly input to the learning model unit 500. Therefore, the input interface 311 can be considered to replace the speech recognition unit 513v. Similarly, if the structure of the speech synthesis unit 514v can be omitted, the response information generated by the learning model unit 500 is directly output to the user 1, which is the sending source. Therefore, the output interface 312 can be considered to replace the speech synthesis unit 514v.

[0395] In this embodiment, the input information D51 corresponds to the model input data D101. Furthermore, the response information D52 corresponds to the model output data D103. The learning model unit 500 (particularly the model control unit 101) can be configured, for example, to output the response information D52 corresponding to the input information D51 based on the model information D102 and, as needed, based on the model reference information D104, if the input information D51 is received.

[0396] Furthermore, in the above-described case, the model generation unit 107, corresponding to the learning model unit 500, may, for example, use model learning data D105, which includes candidates for input information D51 that can be input to the model control unit 101, to perform machine learning and generate or update model information D102. Additionally, the model generation unit 107 may, for example, use model learning data D105, which includes candidates for input information D21 that can be input to the model control unit 101 and corresponding candidates for response information D52, to perform machine learning and generate or update model information D102.

[0397] Although figures are omitted, in this embodiment, status information D55 and / or feedback information D56 can be obtained from the model output data D103 of the learning model unit 500 and / or the output target based on the information generated therefrom. The control system 5000 can, for example, output the obtained status information D55 and / or feedback information D56 as information indicating a response result to a designated supervisor, the learning model unit 500, or other devices not shown. Alternatively, the control system 5000 can be configured to send an inquiry D57 back to the user 1 if the input information D51 contains ambiguous or uncertain information. Furthermore, the control system 5000 can also generate supplementary information D58 for the input / output data of the learning model unit 500 based on the obtained status information D55 and / or feedback information D56, and send it to the user 1, the designated supervisor, the learning model unit 500, or other devices not shown. The processing of status information D55, feedback information D56, inquiry D57, and supplementary information D58 can be substantially the same as in Embodiment 1.

[0398] Additionally, the control system 5000 may also include a status acquisition unit 530 (not shown), which acquires status information D55 and / or feedback information D56, and issues supplementary information D58 as needed. The status acquisition unit 530 is the same as the status acquisition unit 130 in Embodiment 1.

[0399] In this embodiment, the input information D51 received by the control system 5000 can be considered as information related to a request in the work environment (here, the content of a response sent in the environment of responding to an inquiry). Therefore, the input information D51 received by the control system 5000 can be considered as an example of first information representing a request in the work environment. Furthermore, the response information D52, corresponding to the aforementioned input information D51, can be considered as information used in this operation (response operation). Hereinafter, the response information D52, which outputs from the operating environment of the learning model, which is input with model input data based on the input information D51, to a predetermined output target, will be referred to as second information.

[0400] Next, the operation of the control system 5000 in this embodiment will be explained. Figure 28 This is a flowchart illustrating an example of the operation of the control system 5000.

[0401] exist Figure 28 In the example shown, firstly, the control system 5000 receives input information D51v (step S510). For example, the input unit 102 or the input processing unit 201 described above can receive input information D51v. The received input information D51v is then input to the speech recognition unit 513v.

[0402] If the speech recognition unit 513v receives input information D51v, it recognizes the speech contained in the input information D51v and transforms it into input information D51 that matches the data format input to the learning model unit 500 (step S511). The transformed input information D51 is then input to the learning model unit 500 as model input data D101.

[0403] The speech recognition unit 513v can also segment the long input information D51 into multiple segments and input each segmented input information D51 into the learning model unit 500. This is because if the input information D51 is long, the learning model unit 500 is more likely to fail to recognize the correct meaning. This is particularly effective when inputting via voice, as user 1 can easily input long sentences.

[0404] Furthermore, if the speech recognition unit 513v is omitted, the received input information D51v can also be input to the learning model unit 500 as model input data D101.

[0405] Next, the control system 5000 performs the process of generating response information D52 using the learning model unit 500 (step S512). In step S512, the learning model unit 500 (more specifically, the model control unit 101) outputs response information D52 corresponding to the input information D51 based on the model information D102, the input information D51, and the model reference information D104 as needed.

[0406] In step S512, the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 500 may further perform the above-mentioned processing.

[0407] The response information D52 output from the learning model unit 500 is input to the speech synthesis unit 514v, for example (step S513). The input of the response information D52 to the speech synthesis unit 514v can be directly input from the control system 5000 (more specifically, the learning model unit 500 or the information processing device 10, which is its operating environment), or indirectly input via a communication network, other devices (servers, various conversion devices, etc.) or by human hands.

[0408] Next, the speech synthesis unit 514v transforms the input response information D52 into sound-based response information D52v and outputs it (step S514). For example, the speech synthesis unit 514v can synthesize speech that reads out the response content represented by the response information D52 in a data form other than sound, thereby generating response information D52v. Response information D52v is then output to user 1, which is the source of the input information D51v (step S515).

[0409] Furthermore, even if the speech synthesis unit 514v is omitted, the response information D52 output from the learning model unit 500 can also be output to the user 1, which is the source of the input information D51v.

[0410] As described above, in this embodiment, even without preparing an operator or a website with pre-embedded response content for the information sent from user 1, the learning model unit 500 can dynamically generate response information D52 to reply to the user of the sending source, thus achieving high efficiency and high performance in the response operation.

[0411] Variation Example 5-1

[0412] Next, a variation of the control system 5000 will be described. Figure 29 This is a structural diagram illustrating a modified example of the control system 5000 according to this embodiment, namely, control system 5000a. Furthermore, elements identical to those in control system 5000 are labeled with the same reference numerals and their descriptions are omitted.

[0413] exist Figure 29 The control system 5000a shown differs from the control system 5000 in that it has a correctness judgment unit 515.

[0414] The error judgment unit 515 determines whether the content shown in the response information D52, the output from the learning model unit 500, is correct. For example, the error judgment unit 515 may output the response information D52 to the user 1 or update the content of the reference information storage unit 12 only if it determines that the content shown in the response information D52 is correct.

[0415] Furthermore, the error judgment unit 515 may, for example, cause the learning model unit 500 to obtain other response information D52 (re-acquisition of model output data) if it determines that the content shown in the response information D52 is not correct. The error judgment unit 515 may be provided as an example of the post-processing unit 106 described above.

[0416] The error judgment unit 515 determines, for example, whether the content shown in the response information D52 is correct using the following method.

[0417] The accuracy judgment unit 515 generates a knowledge graph in advance based on information related to the object sent by user 1, and then databases it as a graph database. For example, if the object is a product, the information related to the object might be the product manual, etc. Alternatively, this operation can also be performed manually. For example, ... Figure 30 In the same way as (A), a graph database is generated to represent the relationships between statements that relate to objects.

[0418] The correctness judgment unit 515 generates a knowledge graph based on the response information D52 output from the learning model unit 500. This determines the relationships between the statements contained in the response information D52. For example, ... Figure 30 The correlation is determined in the way that (B) does.

[0419] Furthermore, the accuracy determination unit 515 uses query languages ​​such as SPARQL to search the pre-generated graph database and determines whether the relationships between the identified statements are contained within the graph database. SPARQL is an abbreviation for SPARQL Protocol and RDF Query Language. If the accuracy determination unit 515 determines that the relationships between the identified statements are contained within the graph database, it determines that the content shown in response information D52 is correct. On the other hand, if the accuracy determination unit 515 determines that the relationships between the identified statements are not contained within the graph database, it determines that the content shown in response information D52 is incorrect. Figure 30 In the example, Figure 30 The relevance of (B) is not included in Figure 30 The graph database of (A) is therefore determined to be incorrect, as indicated by response information D52.

[0420] Other aspects can be the same as other control systems involved in this embodiment.

[0421] As described above, according to this modified example, it is determined whether the content shown in the output of the learning model unit 500, i.e. the response information, is correct. Based on the result, it is performed to determine whether there is output to the user, to obtain the response information again, and to update the reference information. Therefore, it is possible to further improve the performance of the response operation.

[0422] Variation Example 5-2

[0423] Next, a second variation of the control system 5000 will be described. Figure 31 This is a structural diagram showing a modified example of the control system 5000 according to this embodiment, namely, control system 5000b. Furthermore, elements identical to those in control system 5000 and control system 5000a are labeled with the same reference numerals and their descriptions are omitted.

[0424] like Figure 31 As shown, the control system 5000b may also have an emotion determination unit 516.

[0425] The emotion determination unit 516 uses input information D51 and other information to determine the emotion of the sending source, user 1. Additionally, the emotion determination unit 516 can also determine the emotion of user 1 after the response information D52 from the learning model unit 500 is output to user 1.

[0426] The emotion of user 1 determined by the emotion determination unit 516 can be input into the learning model unit 500 as the state information D55 contained in the model reference information D104, or it can be recorded as a history in the reference information storage unit 12 along with the input and output data of the model.

[0427] As a method for recording in the reference information storage unit 12, the control system 5000b may further include a registration determination unit 518, which determines whether to record in the reference information storage unit 12 based on the determination result of the user 1's emotion obtained by the emotion determination unit 516. Here, the determination result is the determination result of the user 1's emotion after the response information D52 from the learning model unit 500 is output to the user 1. That is, the determination result here is envisioned as the determination result of the emotion determined based on the input information D51 received after the output response information D52.

[0428] For example, if the determined emotion of User 1 is positive, the registration and determination unit 518 may record the model's input and output data as history information in the reference information storage unit 12 as a good example. In this case, the registration and determination unit 518 may also record the model's input and output data, including the emotion information before and after the response, in the reference information storage unit 12, if the determination result of User 1's emotion is available before the output of the response information D52 from the learning model unit 500.

[0429] Alternatively, for example, if the determined emotion of User 1 is negative, the registration and determination unit 518 may record the model's input and output data as historical information in the reference information storage unit 12 as a negative example. Alternatively, the model's input and output data may be recorded as historical information in the reference information storage unit 12. In this case, the registration and determination unit 518 may also record the model's input and output data, including the emotion information before and after the response, in the reference information storage unit 12, if the determination result of User 1's emotion before the output of the response information D52 from the learning model unit 500 is available.

[0430] Furthermore, the control system 5000b may also include an additional learning unit 519, which, when updating the content of the reference information storage unit 12, reconstructs (additionally learns) the model reference information D104 stored in the reference information storage unit 12 and other information referenced by the model control unit 101 based on the updated information. That is, the additional learning unit 519 may also enable the learning model unit 500 to perform additional learning based on the input information, the response information, and the evaluation of the response information by the user 1, which is the sending source.

[0431] Alternatively, the control system 5000b may replace the emotion determination unit 516 or have an evaluation acquisition unit 517 in addition to the emotion determination unit 516.

[0432] The evaluation acquisition unit 517 requests the evaluation of the response information D52 from user 1, and obtains the evaluation information D59 as its answer. The evaluation information D59, for example, can be used for updating the information referenced by the model, supplementary learning, etc., in the same way as the emotion of user 1 mentioned above.

[0433] In addition, the control system 5000b may also have a control decision unit 520.

[0434] The control decision unit 520, based on the speech recognition results, emotion determination results, and / or the evaluation results of the response information D52 on the input information from user 1, as well as instructions from maintenance personnel (not shown), specifies the control object information and / or the output tendency settings for the control generation unit 512. Here, the speech recognition results on the input information from user 1 can include information such as user 1's attributes, emotions, region, language, past usage history, and usage frequency. Furthermore, the control decision unit 520 can also set the synthesized speech for the speech synthesis unit 514v based on the speech recognition results, emotion determination results, and / or the evaluation results of the response information D52 on the input information from user 1, as well as instructions from maintenance personnel (not shown).

[0435] For example, as an example of setting the output preference, the control decision unit 520 can specify the difficulty of the explanation in the response, the speaking style (tone, pitch), the level of language and grammar, politeness, the speaker's standing position, and the ending point of the topic. It can also specify the gender, tone, and pitch of the synthesized speech. Furthermore, as an example of setting the synthesized speech, the control decision unit 520 can specify the gender, speaking style, level of language and grammar, politeness, etc., of the synthesized speech. The control decision unit 520 can, for example, perform these settings based on predetermined setting rules.

[0436] The control generation unit 512 sets the preconditions specified by the control decision unit 520 for the learning model unit 500. As a result, the learning model unit 500 generates response information D52 corresponding to the input information D51 input through the speech recognition unit 513v, according to the preconditions specified by the control decision unit 512.

[0437] If the control decision unit 520 presumes that user 1's emotion is anger, it considers and determines the output tendency to be an apology. Furthermore, if the control decision unit 520 presumes that user 1's attribute is child, it considers and determines the output tendency to be simple and easy to understand. Additionally, the control decision unit 520 changes the gender of the text of response information D52 and the gender of the voice of response information D52v based on the presumed gender of user 1. Furthermore, the control decision unit 520 considers changing the tendency of response information D52 based on the presumed nationality or language used by user 1.

[0438] also, Figure 31 The elements of the control system 5000b shown can be appropriately selected according to the desired functions.

[0439] Other aspects can be the same as other control systems involved in this embodiment.

[0440] As described above, according to this modified example, the control decision unit 520 specifies the controlled object information and / or the output preference setting based on information obtainable from the control system 5000b, thus enabling the generation of response information D52 that easily matches the requirements of the sending source. Therefore, it is possible to further achieve high-performance response operations to the user.

[0441] Variation Example 5-3

[0442] Next, the third variation of the control system 5000 will be explained. Figure 32 This is a structural diagram illustrating a variant of the control system 5000 according to this embodiment, namely, control system 5000c. Furthermore, elements identical to those in control system 5000, control system 5000a, and control system 5000b are labeled with the same reference numerals and their descriptions are omitted.

[0443] like Figure 32 As shown, the control system 5000c may also include an image resolution unit 513i, an image generation unit 514i, and a program generation unit 514p.

[0444] When the input from user 1 includes input information D51i in the form of an image (input image), the image parsing unit 513i parses the image represented by the input information D51i, transforms it into a form that matches the data form of the learning model unit 500, and outputs it. For example, the image parsing unit 513i can transform the input information D51i in the form of an image into input information D51 in the form of text.

[0445] For example, when the input from user 1 includes an input image captured from the operation screen of the product held by user 1, the image analysis unit 513i can analyze the image, determine which product's operation screen it is, and what operation state it is in, and then output it as explanatory text. Alternatively, when the input from user 1 includes an input image captured from a shopping website that user 1 is browsing, the image analysis unit 513i can analyze the image, determine which website's operation screen it is, and what operation state it is in, and then output it as explanatory text.

[0446] The image generation unit 514i generates and outputs a response image based on the response information D52. For example, if the output from the learning model unit 500, i.e., the response information D52, contains data in a form other than an image, the image generation unit 514i can generate and output a response image representing the content of that portion represented by the response information D52. For example, if the response information D52 is a specified data structure containing data in a data form, the image generation unit 514i can also transform data elements specified in the data structure as image form into image form and output them. For example, the image generation unit 514i can also generate response information D52v in image form based on the response information D52 in text form. For example, the image generation unit 514i can also perform a synthesis process that appends the content shown in the response information D52 in text form to the input image contained in the input information D51 as annotation. Additionally, based on the response information D52 in text form, the image generation unit 514i can also perform a process that emphasizes a portion of the input image contained in the input information D51. The image generation unit 514i can also use a learning model to generate a response image based on the input information (response information D52 and, as needed, input information D51).

[0447] The program generation unit 514p transforms the content represented by the response information D52 into a prescribed program data format and outputs it. For example, if the output from the learning model unit 500, i.e., the response information D52, contains a data format other than the prescribed program data format, the program generation unit 514p transforms the content represented by the response information D52 into the prescribed program data format and outputs it. For example, if the response information D52 contains a specified data structure with a specified data format, the program generation unit 514p can also transform data elements that specify the prescribed program data format in that specification into the prescribed program data format and output it. For example, the program generation unit 514p can also transform the text-format response information D52 into the prescribed program data format response information D52p. The program generation unit 514p can also use a learning model to generate a prescribed program based on the input information.

[0448] Image analysis processing performed by image analysis unit 513i is performed, for example, in step S511 described above. Furthermore, image generation processing performed by image generation unit 514i and program generation processing performed by program generation unit 514p are performed, for example, in step S514 described above.

[0449] Furthermore, sometimes the response is given by speech via the speech synthesis unit 514, and by image via the image generation unit 514i. In this case, the speech recognition unit 513v can also divide the text-based input information D51 into input information D51 for speech responses and input information D51 for image responses, and input them to the learning model unit 500. Similarly, the image parsing unit 513i can also divide the text-based input information D51 into input information D51 for speech responses and input information D51 for image responses, and input them to the learning model unit 500.

[0450] Other aspects can be the same as other control systems involved in this embodiment.

[0451] As described above, according to this variation, inquiries and responses can be made not only by sound, but also by a combination of sound and images. Therefore, responses can be made more effectively, for example, to inquiries about the operation screen. Furthermore, according to this variation, the program can also be provided as response information to the sending source in addition to sound and images, thus enabling more effective responses to inquiries such as problem-solving strategies.

[0452] Variation Example 5-4

[0453] Next, the fourth variation of the control system 5000 will be explained. Figure 33 This is a structural diagram showing a modified example of the control system 5000 according to this embodiment, namely, control system 5000d. Furthermore, elements identical to those in control systems 5000 to 5000c are labeled with the same reference numerals and their descriptions are omitted.

[0454] In this variant, the following function is provided: based on the query content from user 1 and / or the output results from the learning model, the response is switched to the response made by the operations and maintenance personnel 8 or the response made by other learning models.

[0455] like Figure 33 As shown, the control system 5000d may also include a call confirmation unit 531 and an output selection unit 532.

[0456] Here, the control system 5000d has a learning model unit 500a as a first response function, and an operator 8 and a communication channel with the operator 8 as a second response function. Additionally, the control system 5000d may also have another learning model unit 500b with a different algorithm or data than the learning model unit 500a as a third response function. Furthermore, as a second response function, another learning model unit 500b with a different algorithm or data than the learning model unit 500a may also be included. In this case, as a third response function, the operator 8 and a communication channel with the operator 8 may also be included. Moreover, the type and number of response functions are not particularly limited. For example, the target switching response function may also be a response system that does not use a learning model.

[0457] In this example, we will take the case where the learning model unit 500a with the first response function is the aforementioned learning model unit 500, the second response function is the operation and maintenance personnel 8 and the communication channel with the operation and maintenance personnel 8, and the third response function is another learning model unit 500b whose algorithm or the data used is different from that of the learning model unit 500a as an example.

[0458] Here, the learning model unit 500a can be a local learning model that obtains output results based on local information such as the limitation of the database of the reference target, and the learning model unit 500b can be a global learning model that obtains output results based on global information such as the ability to freely access external networks.

[0459] The confirmation unit 531 switches the processing unit to handle the response based on the query content from user 1 and / or the output results from the learning model.

[0460] The call confirmation unit 531 can also call the maintenance personnel 8 as a second response function, based on the query content from user 1 and / or the output result from the learning model, i.e., the response information D52. For example, if it is determined that the accuracy of the output obtained through the first response function is unlikely to be expected, the call confirmation unit 531 can also call the maintenance personnel 8 using the communication channel with the maintenance personnel 8 and input the input information D51 to the operation device of the maintenance personnel 8. Alternatively, the call confirmation unit 531 can also call the maintenance personnel 8 using the communication channel with the maintenance personnel 8 and input the input information D51 to the operation terminal (not shown) of the maintenance personnel 8.

[0461] Furthermore, the call confirmation unit 531 can also call the learning model unit 500b as a third response function when it is impossible to call the second response function or when it is determined that the accuracy of the output is difficult to expect. For example, the call confirmation unit 531 can also use the interface with the learning model unit 500b to input input information D51 to the learning model unit 500b, thereby calling the learning model unit 500b.

[0462] Here, the accuracy of the output can be determined using, for example, the evaluation value or likelihood output by the response function itself, or by the reliability evaluation mentioned above. Furthermore, if the response function itself outputs a message indicating that it does not understand the main idea or the idea of ​​delegating a call to other functions, the presence or absence of such a message can also be used to determine the accuracy.

[0463] The output selection unit 532 selects the response information D52 to be output to user 1 based on the switching result of the response processing performed by the call confirmation unit 531. If the switching result of the response processing performed by the call confirmation unit 531 is that the execution body of the response processing is set to the first response function, the output selection unit 532 outputs the response information D52a, which is set to be the output of the first response function, to user 1. If the switching result of the response processing performed by the call confirmation unit 531 is that the execution body of the response processing is set to the second response function, the output selection unit 532 outputs the response information D52b, which is set to be the output of the second response function, to user 1. If the switching result of the response processing performed by the call confirmation unit 531 is that the execution body of the response processing is set to the third response function, the output selection unit 532 outputs the response information D52c, which is set to be the output of the third response function, to user 1.

[0464] The output selection unit 532 can also be controlled by an output switching switch (not shown) to output the output from the selected response function to the user 1. The output switching switch switches the connection path (circuit or communication link, etc.) connected to the response function that is set as the execution subject and the user 1 that is set as the output target.

[0465] Here, the connection path between the response function and user 1 can include various conversion devices and specified interfaces, such as the aforementioned speech synthesis unit, image generation unit, and program generation unit, as needed.

[0466] For example, if the information obtained by the maintenance personnel 8 through text input using the operating terminal is output as response information D52b, the connection path between the response function and user 1 may not include a speech synthesis unit that converts text into speech. Furthermore, the output selection unit 532 can also receive information that has been corrected from the response information D52a output through the first response function as output for the second response function, etc. In this case, the operating terminal of the maintenance personnel 8 includes a text display unit and a text input unit, and the control system 5000d can, for example, receive response information D52b that has been corrected from a portion of the response information D52a output from the operating terminal of the maintenance personnel 8.

[0467] Other aspects can be the same as other control systems involved in this embodiment.

[0468] As described above, according to this variation, in addition to generating a response using the learning model unit 500 described above, it is also possible to generate a response using response generation performed by maintenance personnel, other learning models (such as a serial construction model obtained by connecting multiple models, a multi-model model, or a model obtained by learning focused on a specific device or service), thereby further achieving high performance in the response operation to the user.

[0469] Variation Example 5-5

[0470] Next, the fifth variation of the control system 5000 will be explained. Figure 34 This is a structural diagram showing a modified example of the control system 5000 according to this embodiment, namely, control system 5000e. Furthermore, elements identical to those in control systems 5000 to 5000d are labeled with the same reference numerals and their descriptions are omitted.

[0471] exist Figure 34 The control system 5000e shown has a control decision unit 520, similar to that of control system 5000b, which is different from control system 5000. Figure 34 The control system 5000e shown has a consultation website 541, an engineering design tool 542, and a history database 543 (denoted as history DB in the figure), which is different from the control system 5000.

[0472] The control decision unit 520 specifies preconditions to the control generation unit 512 based on past input information D51 from the user 1, which is the sending source. That is, the control decision unit 520 specifies the control object information and / or the output preference setting to the control generation unit 512 based on the past input information D51 from the user 1.

[0473] The past input information D51 from User 1 here is not limited to information representing query content input to the learning model unit 500 via the speech recognition unit 513v, but may also include information such as query content input via the consultation website 541 and information such as programs input using the engineering design tool 542. The past input information D51 from User 1 is stored in the resume database 543.

[0474] The control decision unit 520 obtains past input information D51 from user 1, which is the source of the input, by referring to the history database 543. The history database 543 stores the input information D51 in association with user 1's identification information. The control decision unit 520 filters the input information D51 stored in the history database 543 based on the identification information of user 1, thereby determining the past input information D51 from user 1. The identification information of user 1, the source of the input, is determined, for example, by performing login processing when the control system 5000e is first used.

[0475] The consultation website 541, like the voice recognition unit 513v, receives inquiries related to the object of inquiry of the control system 5000e. The consultation website 541 receives inquiries not through voice, but through data such as text or images. The text or images received by the consultation website 541 are stored as input information D51 in the history database 543.

[0476] Engineering design tool 542 is a tool used to create programs. The target of the query for control system 5000e is sometimes a system such as a FA system. Engineering design tool 542 is provided in this case as a tool for creating programs that run on the target system. Programs created using engineering design tool 542 are stored in the history database 543.

[0477] For example, the control decision unit 520 determines, based on past input information D51 from user 1 (the sending source), whether user 1 is already using the query object or is currently discussing its use. The control decision unit 520 specifies the preconditions accordingly. For example, if user 1 is already using the query object, the control decision unit 520 specifies the preconditions by sending back specific information. On the other hand, if user 1 is currently discussing its use, the control decision unit 520 specifies the preconditions by highlighting the appeal of the query object.

[0478] The control decision unit 520 can also specify the preconditions by using a program previously created by user 1 as an example to answer questions related to the program when the newly input information D51 is related to the program.

[0479] The control generation unit 512 sets the preconditions specified by the control decision unit 520 for the learning model unit 500. As a result, the learning model unit 500 generates response information D52 corresponding to the input information D51 input through the speech recognition unit 513v, according to the preconditions specified by the control decision unit 512.

[0480] Other aspects can be the same as other control systems involved in this embodiment.

[0481] As described above, according to this modified example, the control decision unit 520 specifies the control object information and / or the output preference setting based on the past input information D51 from the user 1, which is the sending source. Therefore, response information D52 that easily matches the requirements of the sending source can be generated. Thus, high-performance response operations to the user can be further achieved.

[0482] Variations 5-6

[0483] Next, the sixth variation of the control system 5000 will be explained. Figure 35 This is a structural diagram showing a modified example of the control system 5000 according to this embodiment, namely, control system 5000f. Furthermore, elements identical to those in control systems 5000 to 5000e are labeled with the same reference numerals and their descriptions are omitted.

[0484] exist Figure 35 The control system 5000f shown has a defining unit 544, which is different from the control system 5000.

[0485] The clarification unit 544 clarifies the words indicated by the instructions contained in the input information D51, which has been transformed into text form by the speech recognition unit 513v, and then inputs them into the learning model unit 500.

[0486] At this point, if the clarification unit 544 is unsure which of several words the instruction in the input information D51 represents, it sends a query D57 back to user 1 for confirmation of the word indicated by the instruction. After clarifying the input information D51 in such a way that the word indicated by the instruction represents the word specified by user 1, the clarification unit 544 inputs it to the learning model unit 500.

[0487] The input information D51 sometimes contains indicators such as "this," "that," "this," and "that." If such indicators are directly input into the learning model unit 500, an appropriate response information D52 may not be obtained. Therefore, the explicitation unit 544 explicitly identifies which word in the input information D51 is indicated by the indicator before inputting it into the learning model unit 500. Existing parsing techniques can be used to determine the word indicated by the indicator.

[0488] At this point, the declarification unit 544 may be unable to determine which word the indicator represents solely by parsing the input information D51. For example, existing parsing techniques may be used to obtain the word indicated by the indicator and the probability of representing that word. In this case, if the probability is lower than a first threshold, the declarification unit 544 determines that it is impossible to determine which word the indicator represents through parsing, and therefore it is unclear.

[0489] If the clarification unit 544 cannot determine which word the instruction represents through parsing, it sends query D57 back to user 1 for user 1 to confirm the word indicated by the instruction. For example, the clarification unit 544 displays input information D51 and an unclear instruction, creates text to confirm which word the unclear instruction refers to as query D57, and outputs it as query D57v via the speech synthesis unit 514v. The clarification unit 544 determines the word indicated by the unclear instruction based on the subsequently input input information D51. Furthermore, after clarifying the input information D51 containing the unclear instruction using the determined word, it is input to the learning model unit 500.

[0490] The explicitation unit 544 can also, when it is impossible to determine which word the indicator represents through parsing, designate multiple words as object words and generate multiple input information D51 after expliciting the input information D51 in a way that uses the indicator to represent the object words, and input each input information D51 to the learning model unit. If the word indicated by the indicator and the probability representing that word are obtained through existing parsing techniques, the explicitation unit 544 can designate multiple words with probabilities higher than the second threshold as object words.

[0491] In this case, the learning model unit 500 generates response information D52 corresponding to each input information D51. Therefore, the speech recognition unit 513v outputs the word indicated by the instruction and the response information D52 when the instruction represents that word.

[0492] Here, the clarification of the instruction of the input information D51 received by the speech recognition unit 513v is explained. As explained by variation 5-3, sometimes the input information D51 is generated based on the image obtained by the image analysis unit 513i. In this case, similarly, the clarification unit 544 clarifies the word indicated by the instruction, including the input information D51 generated based on the image.

[0493] Other aspects can be the same as other control systems involved in this embodiment.

[0494] As described above, according to this modified example, the explicitation unit 544 explicitly defines the instructions contained in the input information D51 and inputs them to the learning model unit 500. This enables the generation of response information D52 that matches the intent of the sending source. Therefore, it is possible to further improve the performance of the response operation to the user.

[0495] Variations 5-7

[0496] Next, the seventh variation of the control system 5000 will be explained. Figure 36 This is a structural diagram showing a modified example of the control system 5000 according to this embodiment, namely, control system 5000g. Furthermore, elements identical to those in control systems 5000 to 5000f are labeled with the same reference numerals and their descriptions are omitted.

[0497] exist Figure 36 The control system 5000g shown has a guidance questioning unit 545, which is different from the control system 5000.

[0498] The guiding question unit 545 outputs a guiding question, which narrows the search scope of the learning model unit 500 for the reference information database 12, as an inquiry D57. Specifically, the guiding question unit 545 creates the text of the guiding question as an inquiry D57, and outputs it as an inquiry D57v via the speech synthesis unit 514v. Furthermore, if the learning model unit is given input information D51, it generates response information D52 accordingly based on the information obtained from searching the reference information database 12.

[0499] For example, the inquiry may involve multiple device models. In this case, the guidance questioning unit 545 outputs a guidance question to identify the target device model from among the multiple models. Specifically, the guidance questioning unit 545 outputs a guidance question to confirm the device model name. If multiple manuals exist for the identified device model, the guidance questioning unit 545 outputs a guidance question to identify the manual relevant to the inquiry among the multiple manuals. Furthermore, the guidance questioning unit 545 may also output a guidance question to identify the item relevant to the inquiry in the table of contents of the identified manual. Similarly, in the case of an inquiry related to a webpage rather than a manual, the guidance questioning unit 545 outputs a guidance question to identify which of the multiple webpages is relevant.

[0500] If the search range is smaller than the baseline range of the reference information database 12, the guidance questioning unit 545 inputs range information representing the narrowed range to the learning model unit 500. Then, the learning model unit 500 uses the range indicated by the range information as the search range and generates response information D52 accordingly, based on the information obtained from searching the reference information database 12.

[0501] Furthermore, the guidance question unit 545 can also input the answer to the guidance question as scope information into the learning model unit 500. In this case, after determining the search scope based on the scope information, the learning model unit 500 generates response information D52 corresponding to the information obtained by searching the reference information database 12 for the determined search scope.

[0502] The wider the search scope of the reference information database 12, the more likely it is to extract data that is weakly related to the desired content. Response information D52 is generated based on the extracted data. Therefore, if data that is weakly related to the desired content is extracted, the likelihood of not being able to generate appropriate response information D52 increases. The more the guiding questions are used to narrow the search scope of the reference information database 12, the higher the likelihood of generating appropriate response information D52.

[0503] Other aspects can be the same as other control systems involved in this embodiment.

[0504] As described above, according to this modified example, the prompting unit 545 outputs a prompting question that narrows the search scope of the reference information database 12. This allows the generation of response information D52 that matches the intent of the sending source. Therefore, it is possible to further improve the performance of the response operation to the user.

[0505] Furthermore, while examples of system structures corresponding to the operation of interest have been illustrated in the above embodiments, the control system according to the present invention is not limited to these examples. For instance, the control system according to the present invention can also be a suitable combination of one or more of the above embodiments.

[0506] As an example, the control system involved in this invention can also combine the structure of Embodiment 1 and the structure of Embodiment 4, and input information representing the solution obtained from sensor data using the function of Embodiment 4 into the control system of Embodiment 1 and transform it into a program to directly control the target device 2.

[0507] Furthermore, the various embodiments and modifications are not limited to the examples described above, and can be appropriately modified within the scope of the disclosure.

[0508] In addition, the control system and control method involved in this invention include the control system and control method described in the following notes.

[0509] (Note 1)

[0510] A control system for assisting people or objects in performing tasks using equipment. The control system is characterized by having: An input interface receives input of first information, which represents the environment in which the operation is performed, i.e., the conditions or requirements of the work environment; A model processing unit, configured to access a specified learning model; and The output interface, based on the output from the learning model, outputs second information to assist the task. The model processing unit inputs model input data based on the first information into the learning model, and receives model output data corresponding to the model input data from the learning model. The model output data includes the information used in the task. The output interface outputs the second information based on the model output data.

[0511] (Note 2)

[0512] According to the control system described in Appendix 1, in which, The first piece of information includes information indicating a request for control or operation content from the device. The model input data is data that presents the control content or operation content represented by the first information in a form that matches the input of the learning model. The model output data includes information used in the control or operation of the device corresponding to the control or operation content represented by the model input data. The second information includes information that describes the information contained in the model output data used in the control or operation of the device in a prescribed form that can be determined by the output target of the output interface.

[0513] (Note 3)

[0514] According to the control system described in Appendix 2, in which, The output target of the output interface is the device or an interface that requests control from the device. The second information is output to the device or to an interface that requests control from the device, thereby controlling the device.

[0515] (Note 4)

[0516] According to the control system described in Appendix 2, in which, It also includes an executable code generation unit, which generates and outputs executable code that the device can execute. The output target of the output interface is the executable code generation unit. The second piece of information is output to the executable code generation unit, and as a result, the device is controlled by the generated executable code.

[0517] (Note 5)

[0518] According to the control system described in Appendix 2, in which, The output target of the output interface is the terminal operated by the user. The second piece of information is output to the terminal, which in turn controls the device.

[0519] (Note 6)

[0520] According to the control system described in Appendix 1, in which, The first piece of information includes information representing the conditions in the working environment. The model input data is data that presents the state of the working environment, represented by the first information, in a form that matches the input of the learning model. The model output data includes information related to the analysis results of the working environment status represented by the model input data and / or methods for improving that status. The second information includes information that describes the analysis results of the situation in the operating environment and / or the method for improving the situation in a prescribed form that can be determined by the output target of the output interface.

[0521] (Note 7)

[0522] According to the control system described in Appendix 1, in which, The model processing unit is configured to access both the first learning model and the second learning model. The model processing unit inputs first model input data based on the first information into the first learning model, and receives first model output data corresponding to the first model input data from the first learning model. The model processing unit inputs second model input data based on the first model output data into the second learning model, and receives second model output data corresponding to the second model input data from the second learning model. The output interface outputs the second information based on the output data of the second model.

[0523] (Postscript 8)

[0524] According to the control system described in Appendix 7, among which, The first piece of information includes information indicating a request for control or operation content from the device. The first model input data is data that represents the control content or operation content indicated by the first information in a form that matches the input of the first learning model. The output data of the first model includes information that presents the control content or operation content represented by the input data of the first model in a more general or more specific way. The second model input data is data that represents the control content or operation content through the first model output data, presented in a form that matches the input of the second learning model. The second model output data includes information used in the control or operation of the device corresponding to the control or operation content represented by the second model input data. The second information includes information describing the information used in the control or operation of the device contained in the second model output data in a prescribed form that can be determined by the output target of the output interface.

[0525] (Note 9)

[0526] According to the control system described in Appendix 7, among which, The first piece of information includes information representing the conditions in the working environment. The first model input data is data that represents the state of the working environment as indicated by the first information, presented in a form that matches the input of the first learning model. The output data of the first model includes the analysis results of the working environment status represented by the input data of the model. The second model input data is data that represents the analysis results of the working environment status as indicated by the first model output data, presented in a form that matches the input of the second learning model. The output data of the second model includes information on methods for improving the conditions in the work environment, corresponding to the analysis results of the work environment represented by the input data of the second model. The second information includes information that describes, in a prescribed form that can be determined by the output target of the output interface, information contained in the second model output data that is at least related to the method of improving the condition in the working environment.

[0527] (Postscript 10)

[0528] According to the control system described in Appendix 1, in which, The operation is a response made by a person or thing using equipment. The first piece of information includes information representing a request for a response in the working environment, i.e., information requesting a response. The model input data is data that presents the request-response content represented by the first information in a form that matches the input of the learning model. The model output data includes information used in the response corresponding to the request-response content represented by the model input data. The second information includes information that describes the information used in the response contained in the model output data in a prescribed form that can be discerned by the output target of the output interface.

[0529] (Postscript 11)

[0530] According to the control system described in Appendix 3, among which, The output target of the output interface is a screen operation interface that requests control from the device via the operation screen. The model output data includes information about the operation screen recorded in a prescribed form that can be determined by the output target of the output interface. The operation screen is used to actually perform operations on the device that correspond to the control content or operation content represented by the model input data.

[0531] (Postscript 12)

[0532] According to the control system described in any one of Appendix 1 to Appendix 11, wherein, The input interface receives input from multiple users representing the first information indicating the needs of the operating environment. The model processing unit inputs the model input data, which includes the first information input from the plurality of users, into the learning model, and receives the model output data corresponding to the model input data from the learning model.

[0533] (Postscript 13)

[0534] According to the control system described in any one of Appendix 1 to Appendix 12, wherein, The learning model is a language learning model that takes natural language as input and obtains output, an image learning model that takes an image as input and obtains output, and a multi-model model that takes natural language and an image as input and obtains output.

[0535] (Postscript 14)

[0536] According to the control system described in any one of Appendix 1 to Appendix 13, wherein, The model processing unit is configured to access both the first learning model and the second learning model. One of the first learning model and the second learning model is a local learning model in which the database of the reference target is limited to internal information. The first learning model and the other of the second learning model are global learning models in which the database of the reference target is not limited to internal information.

[0537] (Postscript 15)

[0538] According to the control system described in any one of Appendix 1 to Appendix 13, wherein, The model processing unit is configured to access both the first learning model and the second learning model. One of the first learning model and the second learning model is a learning model capable of referencing information specifically defined in the work environment. The first learning model and the other of the second learning model are learning models that cannot be referenced to information specifically defined in the work environment.

[0539] (Postscript 16)

[0540] According to the control system described in any one of Appendix 1 to Appendix 15, wherein, It has an output confirmation unit that performs simulation of the control and state of the device based on model output data output from the learning model.

[0541] (Postscript 17)

[0542] According to the control system described in any one of Appendix 1 to Appendix 16, wherein, Based on the information collected from the output target of the output interface, the learning model is subjected to additional learning or the correctness of the output information is determined, and the flow control of the output information is performed.

[0543] (Postscript 18)

[0544] According to the control system described in any one of Appendix 1 to Appendix 17, wherein, It has an input processing unit that queries the input source when the first information contains unclear or uncertain information.

[0545] (Postscript 19)

[0546] According to the control system described in any one of Appendix 1 to Appendix 18, wherein, The query includes information on the correction, addition, and cancellation of the input and output data displayed for the learning model.

[0547] (Postscript 20)

[0548] According to the control system described in any one of Appendix 1 to Appendix 19, wherein, The first piece of information is time-series data that shows the environment in which the operation is performed, i.e., the conditions or requirements in the work environment, together with information about the time.

[0549] (Postscript 21)

[0550] According to the control system described in any one of Appendix 1 to Appendix 19, wherein, It includes: a model information storage unit that stores model information as an execution environment for the learning model; and a model control unit that receives the model input data and outputs the model output data based on the model input data and the information stored in the model information storage unit.

[0551] (Postscript 22)

[0552] A control method for assisting people or objects in performing operations using equipment. The characteristic of this control method is that, The input interface receives input of first information, which represents the environment in which the operation is performed, i.e., the conditions or requirements within the work environment. The model processing unit, configured to access a specified learning model, inputs model input data based on the first information into the learning model, and receives model output data from the learning model that corresponds to the model input data and contains information used in the task. The output interface outputs second information based on the output of the learning model to assist the task.

[0553] Industrial applicability

[0554] The control system of this invention can be readily used as part of a work assistance system to assist in tasks performed by people or objects. Furthermore, the control system of this invention can be readily used as a control system for controlling equipment when it is used to perform a certain control or task. Here, the control system can also be readily used as a control system for controlling FA (Automatic Facilitation) equipment, a control system in a home or building, or a control system for controlling information processing devices such as server devices that process information on a network.

[0555] Explanation of the label

[0556] 1000, 1000a, 1000b, 1000c, 2000, 3000, 3000a, 3000b, 3000c, 3000d, 3000e, 4000, 4000a, 5000, 5000a, 5000b, 5000d Control Systems

[0557] 100, 200, 300, 300a, 300b, 400, 400a, 400b, 500, 500a, 500b Learning Model Department

[0558] 10, 20 Information processing devices

[0559] 11. Model Information Storage Department

[0560] 12. Reference Information Storage Department

[0561] 101 Model Control Department

[0562] 102 Input Section

[0563] 103 Output Section

[0564] 105 Pre-processing Department

[0565] 106 Post-processing Department

[0566] 107 Model Generation Department

[0567] 104, 104a Control Department

[0568] 201 Input Processing Department

[0569] 202 Output Confirmation Department

[0570] 203 Revision Confirmation Department

[0571] 1 user

[0572] 1a Input Source

[0573] 2. Object device

[0574] 2a Output target

[0575] 3. User Interface

[0576] 4 Controller

[0577] Analysis Sections 41-1 and 41-2

[0578] 42 Switching Unit

[0579] 43 Output switching switch

[0580] 5 sensors

[0581] 6. Model Interface

[0582] 7. Monitor

[0583] 8. Maintenance personnel

[0584] Equipment Information Storage Department (110, 210, 310, 410)

[0585] 120 Execution Code Generation Department

[0586] 230 Status Acquisition Department

[0587] 311 Input Interface

[0588] 312 Output Interface

[0589] 313 Environmental Information Storage Department

[0590] 511 Database Retrieval Department

[0591] 512 Control Generation Department

[0592] 513v Speech Recognition Department

[0593] 513i Image Resolution Unit

[0594] 514v Image Compositing Unit

[0595] 514p Program Generation Department

[0596] 515 True / False Judgment Section

[0597] 516 Emotion Assessment Department

[0598] 517 Evaluation Results

[0599] 518 Registration and Judgment Department

[0600] 519 Additional Study Department

[0601] 531 Call Confirmation Department

[0602] 532 Output Selection Section

[0603] 533 Output Switching Unit

[0604] 541 Consulting Website

[0605] 542 Engineering Design Tools

[0606] 543 Resume Database

[0607] 544 Clarification Department

[0608] 545 Guided Questioning Department

[0609] D101 Model Input Data

[0610] D102 Model Information

[0611] D103 Model Output Data

[0612] D104 Model Reference Information

[0613] Input information for D11, D21, D31, D41, D51, D51v, and D51i.

[0614] D12 Control Description

[0615] D22, D34 control commands

[0616] Analysis results of D42a and D42b

[0617] Response information for D52, D52v, D52i, D52p, D52a, D52b, and D52c

[0618] D32, D32a, D32b Operation Instructions

[0619] D320 Operation Information

[0620] Equipment information for D13, D23, D33, and D43

[0621] D33a Environmental Information

[0622] D14 Execution Code

[0623] D44a, D44b Result Information

[0624] Status information for D15, D25, D35, and D45

[0625] Feedback information for D16, D26, D36, and D46

[0626] Inquiries regarding D17, D27, D37, D47, D57, and D57v.

[0627] Supplementary information for D18, D28, D38, and D48

[0628] D59 Evaluation Information

Claims

1. A control system having: The input interface receives input information from the sending source and inputs it into the learning model unit; A control decision unit, which specifies the preconditions for the learning model unit to generate response information in accordance with the input information previously input from the sending source; and The output interface receives response information generated by the learning model unit according to the preconditions specified by the control decision unit and the input information input through the input interface, and outputs it to the sending source.

2. The control system according to claim 1, wherein, The preconditions indicate at least one of the following: information that becomes the focus when generating the response information and information indicating the tendency of the response information.

3. The control system according to claim 1 or 2, wherein, The control decision unit specifies the preconditions in accordance with the emotions inferred from the input information.

4. The control system according to any one of claims 1 to 3, wherein, The control decision unit specifies the preconditions accordingly, based on at least one of the attributes, regions, and languages ​​inferred from the input information.

5. The control system according to any one of claims 1 to 4, wherein, The control system also includes an image generation unit that generates and outputs a response image based on the response information.

6. The control system according to claim 5, wherein, The input information includes an image. The image generation unit generates the response image by processing the input image contained in the input information based on the response information.

7. The control system according to any one of claims 1 to 6, wherein, The control system also includes a program generation unit that generates and outputs a program based on the response information.

8. The control system according to any one of claims 1 to 7, wherein, The input information is in text format. The control system also includes an image parsing unit that, when an input image is input from the sending source, transforms the input image into text and includes it in the input information.

9. The control system according to any one of claims 1 to 8, wherein, The control system further includes a call confirmation unit, which, based on at least one of the input information and the response information, switches between outputting the response information and calling a second response function different from that of the learning model unit to perform response processing.

10. The control system according to claim 9, wherein, If the call confirmation unit determines that the accuracy of the response information is lower than the benchmark value, it will call the second response function.

11. The control system according to any one of claims 1 to 10, wherein, The control system also has a correctness determination unit, which determines the correctness of the content of the response information. If the content of the response information is determined to be incorrect, the learning model unit outputs the response again.

12. The control system according to any one of claims 1 to 11, wherein, The control system further includes an additional learning unit, which enables the learning model unit to perform additional learning based on the input information, the response information, and the evaluation of the response information by the sending source.

13. The control system according to claim 12, wherein, The evaluation of the sending source is input by the sending source after the response information is output, or is determined based on the emotion inferred from the input information input by the sending source after the response information is output.

14. The control system according to any one of claims 1 to 13, wherein, The control system further includes a definition unit that, when it is unclear which of a plurality of words the instruction in the input information represents, enables the sending source to confirm the word indicated by the instruction, and then defines the input information in such a way that the word indicated by the instruction represents the word specified by the sending source before inputting it into the learning model unit.

15. The control system according to any one of claims 1 to 13, wherein, The control system further includes a definition unit, which, when it is unclear which of a plurality of words the instruction in the input information represents, sets the plurality of words as the word of the object, and then defines the input information by using the instruction to represent the word of the object before inputting it to the learning model unit.

16. The control system according to any one of claims 1 to 15, wherein, If the learning model is given the input information, it generates the response information accordingly, based on the information obtained by retrieving from the reference information database. The control system also includes a guiding questioning unit that outputs guiding questions to the sending source to narrow down the search scope of the reference information database.

17. A control method, wherein, The computer receives input information from the sending source and inputs it into the learning model unit. The computer specifies the preconditions for the learning model to generate response information, corresponding to the input information previously input from the sending source. The computer receives response information generated by the learning model unit in accordance with the specified preconditions and the input information, and outputs it to the sending source.

18. A control program that causes a computer to function as a control system, the control system performing the following processes: The input interface processes the input information received from the sending source and inputs it into the learning model unit. The control decision process specifies the preconditions for the learning model to generate response information, corresponding to the input information previously input from the sending source. as well as The output interface processing receives response information generated by the learning model unit according to the preconditions specified by the control decision processing and the input information input through the input interface processing, and outputs it to the sending source.

Images