Control system, control method, and control program

Abstract

The control system (3000) has an input interface (311) that accepts input of input information (D31) from a user (1) as a command to a target device (2). The input information (D31) is information indicating a command composed of a plurality of operations to the target device (2). The learning model unit (300) inputs the input information (D31) to a learning model and outputs operation instructions (D32) indicating sequential control for realizing a series of controls in the target device (2) for implementing the plurality of operations.

Description

Technical Field

[0001] This disclosure relates to control systems, control methods, and control procedures. Background Technology

[0002] In recent years, the application of AI (artificial intelligence) has been continuously developing. In particular, generative AI, which is capable of generating its own content, is becoming more widespread, and it is anticipated that the application scope of AI will expand. The application of AI is not limited to tasks within the home, but also considers tasks in various facilities such as buildings, factories, stations, schools, hospitals, or commercial facilities, as well as in various outdoor locations / scenes such as roads, field facilities, and the air or sea.

[0003] For example, Patent Document 1 describes a machining program generation device that uses a large-scale language model to generate a program for controlling machinery.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2021-060806 Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] To assist in tasks performed by people or machines, beyond the generation of equipment control programs, it is considered that a learning model may undertake part or all of the tasks involved in the task. In addition, "tasks performed by people or machines" includes not only physical spatial tasks performed by people or machines, but also data spatial tasks such as information processing performed by processors such as CPUs (central processing units).

[0009] Examples of tasks performed by objects include the following.

[0010] Operations performed by various equipment such as robots, machinery, devices, and sensors.

[0011] Operations carried out by various mobile vehicles such as cars, trams, buses, aircraft, and ships.

[0012] This operation may include, for example, operations referred to as control, processing, manipulation, instruction, calculation, input, output, display, communication, testing, manufacturing, conversion, generation, measurement, irradiation, emission, inhalation, heat dissipation, heating, cooling, recording, reading out, shaping, driving, moving, transporting, flying, surveying, monitoring, measuring, and extraction.

[0013] Furthermore, as examples of tasks performed by humans, the following examples can be cited.

[0014] • Tasks performed by humans involving other humans or other living beings as the other party

[0015] Human operations on various equipment

[0016] This task may include tasks such as conversation, audiovisual, confirmation, operation, monitoring, instruction, coordination, and explanation.

[0017] Furthermore, the above example is only one instance, and the work that serves as an auxiliary object of this disclosure is not limited to this.

[0018] When using certain learning models to enable information processing devices to perform some or all of the tasks contained in a human or object's operation, the appropriateness of the model's output sometimes becomes a problem. Furthermore, the appropriateness of the model's input, which sometimes affects the model's output, becomes a problem.

[0019] Furthermore, depending on the specific equipment being controlled, appropriate control may be impossible without understanding the current situation. In such cases, situation identification can become problematic. Furthermore, it's necessary to consider situations where identification is required to include not only the current situation but also past situations, creating a continuous flow of control. For example, when determining subsequent controls based on past control actions, ensuring the continuity of control can sometimes become an issue due to the precision of situation identification.

[0020] Furthermore, in situations where immediate control of the equipment is required, the response time from giving instructions to the learning model to obtaining the result sometimes becomes a problem.

[0021] Furthermore, the maintainability of the model becomes an issue when there are changes or additions to the equipment, as the model sometimes needs to be relearned.

[0022] Thus, various problems still exist in utilizing learning models. Depending on the size of the problem, even if one manages to use a learning model to achieve higher efficiency or performance in a task, it may actually reduce the efficiency or performance of the task.

[0023] These problems regarding the use of learning models become more pronounced, especially as the tasks used as aids become more complex and sophisticated.

[0024] Therefore, the purpose of this disclosure is to further improve the efficiency or performance of tasks performed by people or objects by utilizing learning models.

[0025] Methods for solving problems

[0026] The control system disclosed herein includes: an input interface that accepts input of first information, the first information representing a user command to an object device, the command consisting of multiple operations on the object device; and a learning model unit that inputs the first information into a learning model and outputs second information representing sequential control, the sequential control being a series of controls in the object device for implementing the multiple operations.

[0027] Invention Effects

[0028] This disclosure utilizes a learning model to output information representing the sequential control within the target device for complex commands requiring multiple operations. Therefore, according to this disclosure, it is possible to further improve the efficiency or performance of user operations. Attached Figure Description

[0029] Figure 1 This is a structural diagram showing an example of the control system of Embodiment 1.

[0030] Figure 2 This is an explanatory diagram showing an example of the structure of the learning model section.

[0031] Figure 3 This is an explanatory diagram showing another structural example of the learning model section.

[0032] Figure 4 This is an explanatory diagram showing another structural example of the learning model section.

[0033] Figure 5 This is a structural diagram showing an example of an information processing device that serves as the operating environment of a control unit containing a learning model unit.

[0034] Figure 6 This is an explanatory diagram illustrating an example of model learning in the model generation section.

[0035] Figure 7 This is a flowchart illustrating an example of the operation of the control system in Embodiment 1.

[0036] Figure 8 This is a structural diagram showing another example of the control system of Embodiment 1.

[0037] Figure 9 This is a structural diagram showing another example of the control system of Embodiment 1.

[0038] Figure 10 This is a structural diagram showing another example of the control system of Embodiment 1.

[0039] Figure 11 This is a structural diagram illustrating an example of the control system of Embodiment 2.

[0040] Figure 12This is a structural diagram showing another example of the control system of Embodiment 2.

[0041] Figure 13 This is a flowchart illustrating an example of the operation of the control system in Embodiment 2.

[0042] Figure 14 This is a structural diagram illustrating an example of the control system of Embodiment 3.

[0043] Figure 15 This is a flowchart illustrating an example of the operation of the control system in Embodiment 3.

[0044] Figure 16 This is a structural diagram showing another example of the control system of Embodiment 3.

[0045] Figure 17 This is a flowchart illustrating a modified example of embodiment 3.

[0046] Figure 18 This is a structural diagram showing another example of the control system of Embodiment 3.

[0047] Figure 19 This is a structural diagram showing another example of the control system of Embodiment 3.

[0048] Figure 20 This is a structural diagram showing another example of the control system of Embodiment 3.

[0049] Figure 21 This is a structural diagram showing another example of the control system of Embodiment 3.

[0050] Figure 22 This is a flowchart illustrating a modified example of embodiment 3.

[0051] Figure 23 This is a structural diagram illustrating an example of the control system of Embodiment 4.

[0052] Figure 24 This is a flowchart illustrating an example of the operation of the control system in Embodiment 4.

[0053] Figure 25 This is a structural diagram showing another example of the control system of Embodiment 4.

[0054] Figure 26 This is a flowchart illustrating a modified example of embodiment 4.

[0055] Figure 27 This is a structural diagram illustrating an example of the control system of Embodiment 5.

[0056] Figure 28 This is a flowchart illustrating an example of the operation of the control system in Embodiment 5.

[0057] Figure 29 This is a structural diagram showing another example of the control system of Embodiment 5.

[0058] Figure 30 This is a structural diagram showing another example of the control system of Embodiment 5.

[0059] Figure 31 This is a structural diagram showing another example of the control system of Embodiment 5.

[0060] Figure 32 This is a structural diagram showing another example of the control system of Embodiment 5.

[0061] Figure 33 This is a flowchart illustrating a modified example of embodiment 3.

[0062] Figure 34 This is a flowchart illustrating a modified example of embodiment 3.

[0063] Figure 35 This is a flowchart illustrating another example of the operation of a variation of embodiment 3.

[0064] Figure 36 This is a structural diagram showing another example of the control system of Embodiment 3.

[0065] Figure 37 This is a structural diagram showing another example of the control system of Embodiment 3.

[0066] Figure 38 This is a flowchart illustrating a modified example of embodiment 3.

[0067] Figure 39 This is a structural diagram showing another example of the control system of Embodiment 3.

[0068] Figure 40 This is a flowchart illustrating a modified example of embodiment 3. Detailed Implementation

[0069] Hereinafter, to illustrate this disclosure in more detail, the embodiments for carrying out this disclosure will be described with reference to the accompanying drawings. Hereinafter, the same reference numerals will be used to denote the same elements, and descriptions will be omitted.

[0070] Implementation method 1.

[0071] In this embodiment, an example of using a learning model to assist in tasks related to code generation for an object device will be described.

[0072] Figure 1 This is a structural diagram showing an example of the control system 1000 according to Embodiment 1. Figure 1 The control system 1000 shown is a control system for controlling equipment using a learning model, and includes a learning model unit 100, an equipment information storage unit 110 (denoted as equipment information DB in the figure) and an execution code generation unit 120.

[0073] In addition, Figure 1 The diagram shows user 1 and object device 2, but they can also be included as control system 1000. In this case, "user 1" can also be replaced with "user terminal 1". The same applies to other embodiments.

[0074] When the learning model unit 100 receives input information D11, it outputs control description D12. When the learning model unit 100 receives input information D11, it outputs control description D12 based on model information D102, which will be described later.

[0075] In this embodiment, the learning model unit 100 is configured to output a control description D12 corresponding to the input information D11 when input information D11 is input. Alternatively, the learning model unit 100 may be a model and its operating environment configured to generate and output the control description D12 based on the input information D11, device information D13, and / or other referable information in the learning model unit 100 (such as model reference information D104 described later) when input information D11 is input.

[0076] In this embodiment, the input information D11 includes information representing the control content requested for the target device 2. The input information D11 may, for example, be text, images, voice, or a combination thereof representing the control content for the target device 2. The input information D11 may also be text, images, voice, or a combination thereof representing multiple control contents for the target device 2. Furthermore, the input information D11 may also include information representing control content performed continuously in time; in this case, it may be time-series data containing a defined data structure including text, images, voice, or a combination thereof representing such control content. The method of representing the control content is based on the input form of the model used by the learning model unit 100, but is not limited to this if the learning model unit 100 includes error handling, correction processing, or conversion processing in its preceding stages.

[0077] As an example of representing the control content in input information D11, a method can be given where, based on determining the control to be performed on the target device 2, the values ​​of the parameters used for that control and the state after control are specified. In this case, input information D11 may, for example, include information determining the control and information indicating the values ​​of the parameters used for that control or the state after control. The values ​​of the parameters used for control may, for example, include values ​​related to the type of control (ON / OFF, etc.), orientation, quantity, and time. Examples of control content include "Enable function X" for a PLC (programmable logic controller), "Move the front end to location A" for a robotic arm, and "Lower the set temperature by 1 degree" for an air conditioner. Furthermore, as an example of representing the control content in input information D11, a method can be given using various information such as docstrings that describe functions, specifications, or specifications, design documents, operating commands, control codes, and source code applicable to other models or other devices.

[0078] Furthermore, the method of representing control content in input information D11 is not limited to the explicit representation methods described above. For example, when control is implemented through a certain operation, the corresponding control content can be shown by displaying the operation content. Additionally, for example, the method of implicitly representing it can be achieved through the user 1's words and actions associated with a specific control, the result of the target device 2's actions, or the same control command for other models. In other words, input information D11 includes not only information directly representing control content for the target device 2, but also information indirectly represented using operation content corresponding to that control content, the user 1's words and actions, or images of the target device 2. For example, as content representing control content related to the temperature control of an air conditioner, the user 1's word "hot," or actions such as wiping sweat, rolling up sleeves, or fanning themselves can be used. In this case, input information D11 can use information such as text, voice, or images representing the user 1's speech, or images representing the user 1's actions (moving images). As other examples, information representing control content related to the arm control of a robotic device can include the robot's posture after control, information on the destination location of movement of a specified part, imitation actions of robot actions performed by a person or other object (a simulator that performs the robot's simulated actions, including objects on the screen), or information on instructions to the robot (action instructions based on gestures such as finger pointing).

[0079] Here, the form of input information D11 is not particularly limited. For example, this information can also be text, images, voice, data described in a specified design language, control descriptions (including source code and information described in a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), or execution code. Furthermore, this information is appropriately combined. Additionally, in this disclosure, unless otherwise specifically distinguished, the term "text" may include, in addition to representing natural language in text, data described in a specified design language, control descriptions (including source code and information described in a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), and execution code—data that is mechanically discernible and cannot be determined by humans—as well as mechanically discernible data such as data described in a specified design language, control descriptions (including source code and information described in a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), and execution code.

[0080] The control description D12 contains information related to the control described in a prescribed form that can be determined by the subsequent execution code generation unit 120. The control description D12 may be, for example, source code described in a prescribed programming language. Alternatively, the control description D12 may also be a group of commands described in a form (platform language) processed by a prescribed programming platform. Here, the prescribed programming platform may include no-code programming platforms and low-code programming platforms.

[0081] The device information storage unit 110 stores device information D13, which is information related to the target device 2. Device information D13 may include, for example, information representing the function, performance, structure, dimensions, operation, and / or control method of the target device 2. Furthermore, device information D13 may also include, for example, information related to the program used to control the target device 2. Device information D13 may also be information obtained by digitizing the manual or operating instructions of the target device 2. This digitization includes text digitization, image digitization, speech-based digitization, and combinations thereof. Device information D13 is used, for example, as supplementary information when the learning model unit 100 outputs control description D12.

[0082] Furthermore, the device information D13 may also include information indicating the state of the object device 2. This information may include not only the current state of the object device 2 but also information indicating past states. For example, the device information D13 may also include time-series data with a defined data structure representing the state of the object device 2. The information indicating the state of the object device 2 may be information output from the object device 2 or information input by user 1 or other devices. This information may include various types of information output from the object device 2 (e.g., error messages, log information, notification information, etc.). Hereinafter, in this embodiment, the information indicating the state of the object device 2 will sometimes be specifically referred to as state information D15.

[0083] When the control description D12 is input, the execution code generation unit 120 generates and outputs executable code D14, which is code that the target device 2 can execute, based on the control description D12. The executable code D14 may, for example, be a group of codes described in machine language. The executable code D14 may, for example, contain information used by the target device 2 when it is actually controlled. The executable code D14 may, for example, contain any information related to control described in a form that the target device 2 can discern. The execution code generation unit 120 may, for example, be a compiler that converts the control description D12 into executable code D14.

[0084] The executable code D14 output from the executable code generation unit 120 is input to the target device 2. As a result, the target device 2 operates according to the executable code D14 output from the executable code generation unit 120. The input of the executable code D14 to the target device 2 can be direct from the executable code generation unit 120, or indirectly via a communication network, other devices (servers, various conversion devices, etc.), or by human intervention.

[0085] There are no particular limitations on the object device 2. In addition, the object device 2 is assumed to be a device that can actually execute the execution code D14, but it is not limited to this if there is an interface between the object device 2 and the device, such as a writing device, that allows the object device 2 to read the execution code.

[0086] The target device 2 can be, for example, a PLC, a processing machine, a robot, radar, a sensor, a camera, a projector, or a communication device. Furthermore, the target device 2 can also be, for example, an air conditioner, a refrigerator, a television, lighting, or a washing machine. Additionally, the target device 2 can also be, for example, an elevator, a moving body, a handling device, other machinery, or a control device for controlling such machinery. Furthermore, the target device 2 can also be equipment operating in power generation / transformation / storage plants, water treatment plants, etc., or a control device for controlling other equipment. Furthermore, the learning model unit 100 can also generate executable code D14. For example, if the control description D12 is an interpreted language and the target device 2 is a device capable of receiving and directly executing the control description D12, the control description D12 can be considered as executable code D14. In this case, the executable code generation unit 120 can be omitted. Alternatively, the control description D12 can also be converted into executable code D14 more suitable for the processing of the target device 2 through compilation or optimization.

[0087] Figure 2 This is an explanatory diagram showing an example of the structure of the learning model unit 100. (See diagram for details.) Figure 2 As shown, the learning model unit 100 may also include a model control unit 101 that operates on the information processing device 10 and a model information storage unit 11 (denoted as model information DB in the figure) that stores model information D102. Here, the model information storage unit 11 may also be composed of multiple databases connected via a network.

[0088] Model information D102 contains information about the model. For example, model information D102 may also include information representing the correlation between the model input data D101 and the model output data D103. Furthermore, model information D102 may also include information representing candidates for the model output data D103. Additionally, model information D102 may also include information representing candidates for the model output data D103 and information representing the relationships between these candidates. Moreover, model information D102 may also include information that defines the behavior of the learning model, such as constraints, weighted variables, and evaluation functions—that is, model parameters.

[0089] The model can also be a machine learning model that has undergone supervised learning, reinforcement learning, or unsupervised learning. This model can be, for example, a model obtained by performing learning according to deep learning, genetic programs, functional logic programs, and other well-known algorithms / methods. Furthermore, the model can also be, for example, a model referred to as an NN (Neural Network), CNN (Convolutional Neural Network), RNN (Recurrent Neural Network), VAE (Variational Autoencoder), GAN (Generative Adversarial Networks), Diffusion model, Transformer model, LLM (Large Language Model), VLM (Visual Language Model), BERT (Bidirectional Encoder Representations from Transformers), GPT (Generative Pre-trained Transformer), or CLIP (Contrastive Language Image Pre-training). Furthermore, the model can also be a rule-based model that obtains output results by referring to a pre-determined table or based on pre-determined conditions. Additionally, the aforementioned models are not exclusive; for example, LLM, VLM, BERT, and GPT are included in the Transformer model. Moreover, for example, the Transformer model is included in the NN model. Furthermore, learning algorithms and models can also be obtained by combining multiple types. Models also include those obtained by combining and learning from multiple different types of data, known as multimodal models.

[0090] When the model control unit 101 receives model input data D101, it outputs model output data D103 corresponding to the model input data D101 based on the model input data D101 and model information D102. When the model control unit 101 receives model input data D101, for example, it uses the model represented by the model information D102 and outputs model output data D103 corresponding to the model input data D101.

[0091] The model control unit 101 is implemented, for example, by a CPU or similar device included in the information processing apparatus 10 that operates according to a program. Hereinafter, the learning model unit 100 is sometimes referred to as an artificial intelligence unit. Here, an artificial intelligence unit refers to an AI with intelligent functions such as reasoning and judgment, and its operating environment. Therefore, the model control unit 101 may also include an AI with intelligent functions such as reasoning and judgment, and its operating environment. The model control unit 101 may, for example, be an AI with a learning model as described above, and its operating environment. The model control unit 101 may also be an element (module) of the control unit 104 included in the information processing apparatus 10.

[0092] In addition, such as Figure 3 As shown, the learning model unit 100 may also include a reference information storage unit 12 (denoted as reference information DB12 in the figure) for storing model reference information D104. Here, the reference information storage unit 12 may also consist of multiple databases connected via a network. The same applies to other storage units (e.g., device information storage units, etc.) described later.

[0093] Model reference information D104 is information referenced by the model control unit 101 for outputting model output data. Model reference information D104 may include the history of past model input data and / or the history of past model output data. Furthermore, model reference information D104 may include information that maps the feature quantities contained in past inputs to the feature quantities contained in the outputs made for those inputs. Additionally, model reference information D104 may include information evaluating the results output for past inputs.

[0094] Furthermore, the model reference information D104 may also include information associated with the representations or concepts contained in the model input data D101. For example, the model reference information D104 may also include information that maps a specific representation or concept that may be contained in the model input data D101 to other representations or concepts associated with that representation or concept. Here, among the other representations or concepts associated with a certain representation or concept, there are representations or concepts that further specify that representation or concept, and other representations or concepts conceived based on that representation or concept. For example, the model reference information D104 may also include information that maps a specific representation or concept that may be contained in the model input data D101 to other representations or concepts associated with that representation or concept. As an example, the model reference information D104 may also include information that maps a specific representation or concept that may be contained in the model input data D101 to information associated with that representation or concept. Model reference information D104 may also include, for example, information that maps retrieval keywords to values ​​extracted from possible representations or concepts in the model input data D101. Model reference information D104 may also include information for so-called grounding. Furthermore, model reference information D104 may also include a so-called knowledge graph describing real-world entities and the relationships between them. In a knowledge graph, various types of information are systematically linked and represented using a graph structure.

[0095] Furthermore, the model reference information D104 may also contain information for so-called attention. For example, model reference information D104 may also contain information indicating the correlation between performances or concepts that may be contained in the model input data D101 and other performances or concepts. Additionally, model reference information D104 may contain a feature map that uses keyword information as a feature quantity, extracted from performances or concepts that may be contained in the model output data D103 associated with performances or concepts that may be contained in the model input data D101. Furthermore, model reference information D104 may also contain information that maps a query extracted from performances or concepts that may be contained in the model input data D101 to the keyword information used for retrieval corresponding to that query.

[0096] exist Figure 3 In the example shown, when the model control unit 101 receives the model input data D101, it outputs the model output data D103 based on the model input data D101, the model information D102, and the model reference information D104.

[0097] Alternatively, the learning model unit 100 may include a search engine for retrieving the model reference information D104, or an interface to that search engine, instead of the reference information storage unit 12. In this case, the search scope of the search engine can be an external network or a specific network. Here, as one of the external or specific networks, a database (e.g., device information DB, etc.) provided by the control system of this disclosure can be used.

[0098] When referred to as a "learning model," it sometimes refers to a computer algorithm that outputs something based on learned information from input information, or the learned information itself. However, when referred to as a "learning model" in an operational context, it usually refers to the actual program that enables such a computer algorithm to perform actions and its operational environment. In this disclosure, the latter is used. To distinguish it from models that represent simple algorithms or groups of learned information, a model that actually performs actions based on information stored in model information D102 is called a "learning model." In the control system of this disclosure, a learning model unit (particularly model control unit 101) is provided, which corresponds to such a learning model. Therefore, in the following description of the control system, when referred to as a "learning model," it refers to the learning model unit or, particularly, the model control unit 101 therein.

[0099] Figure 4 This is an explanatory diagram showing another structural example of the learning model unit 100. (See diagram for example.) Figure 4 As shown, the learning model unit 100 may also include an input unit 102, an output unit 103, and a control unit 104.

[0100] Input unit 102 accepts model input data D101. Input unit 102 may also accept model input data D101 input by user 1, etc. Input unit 102 may also accept model input data D101 constituting time series data. In this case, input unit 102 may accept model input data D101 constituting time series data sequentially, or it may accept model input data D101 buffered to some extent. In addition, input unit 102 may accept model input data D101 input from multiple input sources. In this case, input unit 102 may accept model input data D101 with information attached to the input source (e.g., user identifier, user attribute information, etc.), or it may accept the model input data D101 after input unit 102 has identified the input source and attached the input source information to the model input data D101, or it may accept the model input data without doing anything. Input unit 102 may be implemented, for example, by various input devices (e.g., pointing device, keyboard, voice input device, image input device, data reading device, data input device corresponding to various communication interfaces, etc.) provided by information processing device 10. Alternatively, the input unit 102 can be implemented using an external device of the information processing device 10. In this case, the information processing device 10 only needs to include an interface with the input unit 102.

[0101] Output unit 103 outputs the object generated by control unit 104. Here, the object includes model output data D103 or data generated based on the model output data D103. Furthermore, output unit 103 can also output the object to multiple output destinations if the object generated by control unit 104 contains information for multiple output destinations. In this case, output unit 103 can output the same data to multiple output destinations or output different data to each output destination. Output unit 103 can be implemented, for example, through various output devices included in information processing device 10 (e.g., display device, voice output device, image output device, data writing device, data output device corresponding to various communication interfaces, etc.). Alternatively, output unit 103 can also be implemented through an external device of information processing device 10. In this case, information processing device 10 only needs to include an interface with output unit 103.

[0102] The control unit 104 operates on the information processing device 10 and includes a preprocessing unit 105 and a postprocessing unit 106 in addition to the model control unit 101 described above.

[0103] The preprocessing unit 105 performs processing to improve the accuracy of the objects generated by the control unit 104. For example, the preprocessing unit 105 may also add, modify, or delete features, or transform (including processing) the model input data D101.

[0104] For example, when the input unit 102 receives model input data D101, the preprocessing unit 105 can modify (including adding and deleting) features or transform (including processing) the model input data D101. Modification of features or transformation (including processing) of data includes not only changes in data format but also changes in the representation or concept expressed by the data. The modified data from the preprocessing unit 105 is then input as model input data D101 to the subsequent model control unit 101. The processing performed by the preprocessing unit 105 includes what is called "editing" the prompts for the model control unit 101.

[0105] The preprocessing unit 105 may, for example, decompose the model input data D101 into specified unit data. Furthermore, the preprocessing unit 105 may, for example, perform a process of combining multiple model input data D101. Moreover, the preprocessing unit 105 may perform feature changes or data transformations after decomposing the model input data D101 into specified unit data, or it may perform feature changes or data transformations after combining multiple model input data D101.

[0106] The post-processing unit 106 corrects the object if, for example, there is a problem with the object generated by the control unit 104 (especially the model control unit 101). The post-processing unit 106 may also use the aforementioned indicator map to determine if there is a problem with the object. For example, it may compare the similarity between the relationship represented by the indicator map, the relationship between the performance or concept contained in the model input data and the performance or concept contained in the model output data, and / or the relationship between the performance or concept contained in the model output data, and determine that there is a problem with the object if the relationship represented by the indicator map deviates from the predetermined distance.

[0107] Furthermore, the components other than the model control unit 101 mentioned above are not essential and can be selected for installation as appropriate.

[0108] In addition, the model information D102 and other information used in learning the model can be pre-prepared information or information obtained as needed via a communication network.

[0109] Figure 5 This is a structural diagram showing another example of an information processing device 10 that serves as an operating environment including a learning model unit 100, a control unit 104, etc. Figure 5 The information processing device 10 shown may also include a control unit 104a, which includes a learning model unit 100 (especially a model control unit 101), an input processing unit 201, an output confirmation unit 202, and a correction confirmation unit 203.

[0110] The input processing unit 201 receives input information D11 from input source 1a, such as user 1. Furthermore, the input processing unit 201 outputs the received input information D11 as model input data D101 to the learning model unit 100.

[0111] At this time, the input processing unit 201 may, for example, output the data after modifying the elements or transforming the data of the input information D11 as model input data D101. The input processing unit 201 may also, for example, remove noise from the input information D11. Furthermore, if the input information D11 contains qualitative information, the input processing unit 201 may convert that information into quantitative information. Furthermore, if the input information D11 contains quantitative information, the input processing unit 201 may, for example, correct the quantity based on the device that requested the input information D11 and its operating environment. The input processing unit 201 may perform so-called anchoring processing, that is, changing the representation or concept shown by the input information D11 into a more specific representation or concept.

[0112] Furthermore, the input processing unit 201 may also return a query to the input source if the input information D11 contains ambiguous or uncertain information. As a query, the input processing unit 201 may output, for example, a message confirming the input content, a message suggesting a correction scheme for the input information D11, or a message requesting the re-entry of the input information D11 with a different state or expression. Furthermore, the correction scheme for the input information D11 may also be generated by the correction confirmation unit 203, which will be described later. Hereinafter, information that modifies, adds, or cancels the content representing the input and output data of the learning model after input and output is sometimes referred to as supplementary information D18. A correction scheme is an example of supplementary information D18.

[0113] The output confirmation unit 202 performs a simulation of the control and state of the target device 2 based on the model output data D103 output from the learning model unit 100. The output confirmation unit 202 can also perform the simulation by converting the model output data D103 into control information that conforms to the specifications of a simulator (not shown) capable of simulating the control and state of the target device 2. The output confirmation unit 202 may also have simulator functionality. During the simulation, the output confirmation unit 202 may also utilize information obtained from the output destination 2a of the model output data D103. Here, the output destination 2a includes the output destination of information generated based on the model output data D103. The information obtained from the output destination 2a may, for example, include the state information D15 and / or feedback information D16, described later.

[0114] The output confirmation unit 202 can also confirm, for example, the state of the object device 2, the state of the system containing the object device 2, and / or the state of the workpieces possessed by the object device 2. Furthermore, before performing action confirmation, the output confirmation unit 202 can generate and display human-understandable intermediate products based on the model output data D103 or information generated based on the model output data D103. Examples of intermediate products include the source code for the control program and an operation image of the controller of the object device 2 for issuing operation commands to the object device 2. Additionally, the output confirmation unit 202 can also display the simulation results along with the reliability indicators of the learning model.

[0115] Here are examples of reliability metrics for learning models. For instance, an evaluation network could be included, which, during pre-learning, accumulates the results of evaluating each input to the learning model. This network learns from both the inputs and the evaluation results. When using the learning model, the inputs to the learning model are also fed into the aforementioned evaluation network, and the output is used as the reliability metric.

[0116] In addition, for example, a learner may be provided that clusters the output of the learning model during pre-learning. When using the learning model, the output of the learning model is also input into the learner, and the clustering result is used as a reliability indicator.

[0117] Alternatively, for example, an evaluation network may be provided, which, during prior learning, accumulates the results obtained by evaluating the results whenever there is an input to the learning model, and learns the feature quantities of the input with high evaluation results. When using the learning model, the input of the learning model is also input into the aforementioned evaluation network, and the similarity between the feature quantities of its output result and the feature quantities of the learning result is used as a reliability index.

[0118] Alternatively, for example, it may have a learner that, during pre-learning, accumulates the results obtained by evaluating the input to the learning model whenever there is an input, clusters the inputs of the learning model with high evaluation results, and when using the learning model, also inputs of the learning model are input to the aforementioned learner, and the clustering result is used as a reliability index.

[0119] The correction and verification unit 203 uses the simulation results implemented by the output verification unit 202 to determine the validity of the model output data D103 and / or model input data D101. For example, the correction and verification unit 203 may also compare the state of the object device 2 shown in the simulation results with the state of the object device 2 determined by the input information D11, model output data D103, and / or model input data D101 to determine whether correct control has been performed, thereby determining the validity of the model output data D103 and / or model input data D101. The correction and verification unit 203 may also determine that correct control has been performed if the state of the object device 2 shown in the simulation results matches the state of the object device 2 determined by the model output data D103 and / or model input data D101. The state of the object device 2 compared here is not limited to one.

[0120] In addition, the correction and confirmation unit 203 can, for example, determine the legitimacy of the model output data D103 and / or the model input data D101 by confirming whether the state or control trajectory of the object device 2 shown in the simulation results is consistent with the control shown in the input information D11, or whether it does not contain pre-prohibited content.

[0121] In addition, the correction and confirmation unit 203 can also prompt the input source 1a of the input information D11 with the simulation results, so that it can answer whether the desired control has been performed, thereby judging the legitimacy of the model output data D103 and / or the model input data D101.

[0122] The correction and confirmation unit 203 can also correct the model input data D101 if it determines that the model output data D103 and / or the model input data D101 are incorrect. In addition, the correction and confirmation unit 203 can also generate supplementary information D18 for the input information D11 instead of correcting the model input data D101, and output it to the input source 1a.

[0123] The control system 1000 can also be, for example, equipped with Figures 1-5 The structure shown in any of the diagrams serves as the operating environment for the learning model unit 100. Similarly to the learning model unit 100, in this case, part or all of this structure may be an internal structure of the control system 1000, or it may be an external structure.

[0124] Furthermore, the aforementioned learning model unit 100 and its surrounding structure are merely illustrative examples. Not all components are necessary structures; installation can be selected appropriately based on the desired function.

[0125] Figure 6 This is an illustrative diagram showing an example of model learning. For example... Figure 6As shown, the model information D102 can also be generated by the model generation unit 107 using the model learning data D105 for machine learning.

[0126] The model generation unit 107 is a processing unit that generates or updates model information D102 based on the input model learning data D105 according to a predetermined algorithm. The model generation unit 107 is implemented, for example, by a CPU or similar device included in the information processing apparatus 20 that operates according to a program. Here, the algorithm followed by the model generation unit 107 can be a machine learning algorithm corresponding to the learning model, such as supervised learning, reinforcement learning, or unsupervised learning, or it can be deep learning, genetic programming, functional logic programming, or other known algorithms.

[0127] Furthermore, the model generation unit 107 can also generate or update model information D102 based on the model reference information D104 for the input model learning data D105. Additionally, the model generation unit 107 can also generate and update model information D102 based on the model output data D103 from the model control unit 101 for the input model learning data D105.

[0128] The model learning data D105 is not particularly limited. For example, when using supervised learning as the learning algorithm, the model learning data D105 may include candidate input model data D101 and corresponding candidate output model data D103. Furthermore, the model learning data D105 may also include the actual input model input data D101 and / or the actual output model output data D103. By appropriately using the actual input model data D101 and / or output model data D103, feedback control can be performed. Additionally, the model learning data D105 may also include information obtained from the equipment or processing unit of the system that actually performs the learning model's actions.

[0129] The model information D102 generated or updated by the model generation unit 107 is provided to the model control unit 101 through the model information storage unit 11. Alternatively, the model generation unit 107 may also directly output the model information D102 to the model control unit 101.

[0130] The model generation unit 107 may, for example, generate model information D102 using the input model learning data D105 before the model control unit 101 uses the model information D102, and store it in the model information storage unit 11.

[0131] The update of model information D102 by model generation unit 107 can also be a process known as FineTune.

[0132] In addition, the model generation unit 107 may be included in the control system 1000 or in a system other than the control system 1000.

[0133] Furthermore, despite Figure 1 The learning model unit 100, device information storage unit 110, and device information D13 are shown separately, but the device information storage unit 110 and device information D13 may also be part of the learning model unit 100. That is, the learning model unit 100 may include the device information storage unit 110 and device information D13. For example, the learning model unit 100 may include the device information storage unit 110 as one of the reference information storage units 12 described later. Furthermore, by using the device information D13 in the model learning stage of learning the model used by the learning model unit 100, it can be configured to pre-assemble the device information D13 into the model. In this case, the device information storage unit 110 may be omitted.

[0134] Furthermore, the learning model unit 100 may be part or all of the internal structure of the control system 1000, or it may be an external structure of the control system 1000. If it is an external structure of the control system 1000, the control system 1000 only needs to replace part or all of the learning model unit 100 with an interface that allows information exchange with an external system possessing that part or all of it. For example, the control system 1000 may also use the model information storage unit 11, referred to as the core of the learning model, as an external structure. Furthermore, for example, the control system 1000 may also use the model information storage unit 11, referred to as the core of the learning model, and the model control unit 101, which implements the algorithm for the model, as external structures.

[0135] Hereinafter, in the control system 1000, in order to distinguish between the model control unit 101 that undertakes the algorithm for learning the model and the part that performs the processing of issuing requests to and receiving responses from such a model control unit 101, the part that performs the latter processing is sometimes referred to as the "model processing unit". More specifically, the model processing unit is equivalent to the part other than the model control unit 101 in the information processing device 10, control unit 104, or control unit 104a described above. In addition, if the model control unit 101 exists in an internal environment, the model processing unit may also be implemented by an OS (Operating System) that calls the learning model application, a prompt word application (and a control unit that serves as its operating environment) that operates on the information processing device 10. In addition, if the model control unit 101 exists in an external environment, the model processing unit may also be implemented by a browser, a client application (and a control unit that serves as its operating environment) that operates on the information processing device 10.

[0136] Furthermore, the structure of the aforementioned learning model and the information processing device serving as its operational environment, as well as the relationship between the learning model and the control system equipped with the learning model, are the same in other embodiments.

[0137] In this embodiment, the input information D11 corresponds to the model input data D101. Furthermore, the control description D12 corresponds to the model output data D103. The learning model unit 100 (especially the model control unit 101) may, for example, be configured to, upon receiving the input information D11, output the control description D12 corresponding to the input information D11 based on the model information D102 and, if necessary, based on the model reference information D104.

[0138] Furthermore, in this case, the model generation unit 107, corresponding to the learning model unit 100, can, for example, use the model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D11 that can be input to the model control unit 101. Additionally, the model generation unit 107 can, for example, use the model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D11 that can be input to the model control unit 101 and candidates for corresponding control descriptions D12.

[0139] In this embodiment, the learning model unit 100 may be, for example, a language learning model such as LLM that takes natural language as input and obtains output results, and its operating environment. Furthermore, the learning model unit 100 may be, for example, an image learning model such as VLM that takes images as input and obtains output results, and its operating environment. Additionally, the learning model unit 100 may be, for example, a multimodal model that takes natural language and images as input and obtains output results, and its operating environment. In this case, the input information D11 may also be input via text data, image data, a combination of text data and image data, or a data form that can be converted into them (speech data, dynamic images as a combination of speech data and image data, etc.). Furthermore, the learning model used in this embodiment is not limited to the models described above.

[0140] In this embodiment, the input information D11 received by the control system 1000 can be referred to as information related to the working environment, specifically the requirements (in this case, the control content requested for the object device 2) in the environment where the operation is performed. Therefore, the input information D11 received by the control system 1000 can be referred to as an example of first information representing the requirements in the working environment. Furthermore, the control description D12 and the execution code D14 can be referred to as information corresponding to such input information D11 for the operation (the operation involved in the control of the object device 2). Hereinafter, the control description D12 output from the working environment of the learning model, which is input based on the model input data of the input information D11, to the specified output destination will sometimes be referred to as second information.

[0141] Here, in the relationship between input information D11 and model input data, in the case of model input data based on input information D11, it may include the input information D11 itself, information after converting the input information D11 into a form that conforms to the input of the learning model, and information supplementing the input information D11. Similarly, in the relationship between model output data and the second information, in the case of the second information based on model output data, it may include the model output data itself, data after converting the model output data into a form that conforms to the input of the output destination, and data supplementing the model output data. The relationship between input / output information and model input / output data in other embodiments is similar.

[0142] Next, the operation of the control system 1000 of this embodiment will be explained. Figure 7 This is a flowchart illustrating an example of the operation of the control system 1000.

[0143] exist Figure 7 In the example shown, firstly, the control system 1000 accepts the input information D11 (step S110). For example, the input unit 102 or the input processing unit 201 described above can also accept the input information D11. The accepted input information D11 is input to the learning model unit 100 as model input data D101.

[0144] In step S110, the control system 1000 can also accept multiple input information D11. In addition, the control system 1000 can also accept input information D11 that better meets the needs of user 1 by engaging in dialogue with user 1, that is, repeatedly inputting and outputting information associated with input information D11 with user 1.

[0145] Next, the control system 1000 performs a process for generating a control description D12 using the learning model unit 100 (step S111). In step S111, the learning model unit 100 generates and outputs a control description D12 corresponding to the input information D11. For example, the learning model unit 100 (more specifically, the model control unit 101) outputs a control description D12 corresponding to the input information D11 based on model information D102 and the input information D11, and, if necessary, based on model reference information D104 including device information D13. The learning model unit 100 may, for example, use a learning model capable of generating text data to generate a control description D12 of text data based on the input information D11.

[0146] In step S111, the learning model unit 100 (more specifically, the preprocessing unit 105 or the input processing unit 201) may, before the processing of the model control unit 101, add, modify, or delete elements, or transform (including process) data in the input information D11 to improve the accuracy of the control description D12. Furthermore, in step S111, the learning model unit 100 (more specifically, the postprocessing unit 106) may, after the processing of the model control unit 101, determine whether there are problems with the control description D12, and if so, perform corrective processing on the control description D12.

[0147] The control description D12 output from the learning model unit 100 is input to the execution code generation unit 120. When the control description D12 is input, the execution code generation unit 120 generates execution code D14 based on the input control description D12 (step S112).

[0148] Next, the execution code D14 generated by the execution code generation unit 120 is input to the target device 2 (step S113). As already explained, the input of the execution code D14 to the target device 2 can be directly input from the control system 1000 (more specifically, the execution code generation unit 120), or indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or by hand.

[0149] Therefore, the object device 2 performs an action according to the input execution code D14.

[0150] The control system 1000 may also acquire status information D15 (step S114) when the state of the target device 2 changes due to the result of outputting execution code D14 indicating that the target device 2 is under control. The acquired status information D15 is stored in the device information storage unit 110, for example, as part of the device information D13. The control system 1000 may also use the acquired status information D15 to update the device information D13 stored in the device information storage unit 110. Furthermore, the control system 1000 may output the acquired status information D15 as information indicating the control result to the user 1, the learning model unit 100, or other devices not shown. In addition, if the control system 1000 does not use the status information D15, the processing in step S114 may be omitted.

[0151] The control system 1000 may also repeat the process of steps S110 to S114 multiple times (for example, until the desired control of the target device 2 is completed).

[0152] In addition, the control system 1000 can also output the control description D12 to the user 1's operation terminal, etc., and perform subsequent processing (such as code generation in the code generation unit 120) based on the user 1's confirmation of the content.

[0153] The state information D15 input to the learning model unit 100 is used, for example, for additional learning by the learning model unit 100. The learning model unit 100 may also update the model information D102 and / or the model reference information D104 based on the input state information D15.

[0154] As described above, according to this embodiment, it is not necessary for user 1 to create control description D12, and execution code D14 can be generated based on input information D11 input from user 1, thus enabling efficient operation of the controlled device 2.

[0155] Furthermore, in this embodiment, the input information D11 can also be text, image, voice, or a combination thereof that explicitly or implicitly represents the control content for the target device 2, thereby further suppressing the inconvenience of inputting the input information D11 and achieving high efficiency in controlling the target device 2.

[0156] Furthermore, according to this embodiment, a control description D12 can be generated using a learning model based on the input information D11. Therefore, even if the user 1 does not know the detailed specifications of the object device 2, the specifications of the control description D12, or other information used to control the object device 2, a control description D12 corresponding to the input information D11 can still be generated, thus enabling high-performance operation of the object device 2. Here, high-performance operation of the object device 2 also includes high precision control of the object device 2.

[0157] Furthermore, in this embodiment, the status information D15 obtained after controlling the object device 2 based on the input information D11 can be used for the generation of the next control description D12, thus further improving the performance of the operation of the object device 2.

[0158] Furthermore, in the example above, only one object device 2 is shown; however, there may be multiple object devices 2 that are controlled by the control system 1000. In such a case, for example, the input information D11 may contain information that can identify the object device 2, and the object device 2 may be identified based on the input information D11 by the input side (input unit 102, preprocessing unit 105, input processing unit 201) of the learning model unit 100. Alternatively, the learning model unit 100 may output control content that identifies the object device 2 as a result of learning.

[0159] Variation 1-1.

[0160] Next, a variation of the control system 1000 will be described. Figure 8 This is a structural diagram showing an example of a control system 1000a, which is a variation of the control system 1000 of this embodiment. Furthermore, elements identical to those in the control system 1000 are labeled with the same reference numerals and their descriptions are omitted.

[0161] exist Figure 8 In the control system 1000a shown, the output from the learning model unit 100 is input to the subsequent execution code generation unit 120 after being confirmed by the user 1.

[0162] In this embodiment, user 1 can confirm the control description D12 output from the learning model unit 100 and input input information D11 based on the confirmation result. Alternatively, user 1 can confirm not only the control description D12 output from the learning model unit 100, but also the feedback information D16 from the execution code generation unit 120 and / or the object device 2, and input input information D11 based on these confirmation results. In this case, user 1 can input not only new content in the input information D11, but also corrections, additions, or cancellations of previously input content in the input information D11. In this case, the input information D11 can include instructions for the learning model unit 100. For example, user 1 can input instructions for eliminating defects contained in the input information D11 or defects contained in the output control description D12, along with the feedback information D16, as input information D11. Here, the instructions for eliminating defects also include input for finding the cause of the defect and the solution to the defect.

[0163] Feedback information D16 may also include a response from the processing unit to the subsequent processing unit when the learning model unit 100 requests control. Furthermore, feedback information D16 may also include information obtained from the processing unit after the learning model unit 100 requests control from the subsequent processing unit. For example, feedback information D16 may also include a response from the execution code generation unit 120 to the request when a control description D12 is input to the execution code generation unit 120 and the generation of the control description D12 is requested. Furthermore, feedback information D16 may also include a response from the target device 2 to the request when execution code D14 is input to the target device 2 and the execution of the code is requested. Feedback information D16 may include status information D15. Feedback information D16 may be output directly to user 1, or it may be output to user 1 via the execution code generation unit 120 or an output device (not shown) provided by the control system 1000.

[0164] Furthermore, the feedback information D16 can include information used to determine whether the control requested by the processing unit is correctly executed in the subsequent processing unit of the learning model unit 100. This information is not limited to information directly obtained from the processing unit. For example, it can also be information obtained from other people, devices, networks, or AI (none shown). The feedback information D16 can, for example, include parsing information such as execution time or control trajectory information, used to determine whether the execution code D14 can correctly execute the target control. User 1 can also, for example, instruct the learning model unit 100, based on such information contained in the feedback information D16, to control the timing of the process in the control description D12 or adjust the lead time, etc.

[0165] Furthermore, for example, user 1 can also interact with the learning model unit 100 using feedback information D16, and each time determine the validity of the output control description D12 (whether there is a problem). User 1 can also output the control description D12 to the execution code generation unit 120 if it determines that the control description D12 has no problem.

[0166] The feedback information D16 can be obtained, for example, in step S114 described above.

[0167] In addition, Figure 8 The example shown is of user 1 inputting control description D12 into execution code generation unit 120, but inputting control description D12 into execution code generation unit 120 can also be performed by learning model unit 100 that receives instructions from user 1.

[0168] In this example, the control description D12 may also contain a description corresponding to a low code or no code.

[0169] In this example, the exchange of information between user 1 and learning model unit 100 can be carried out, for example, through a terminal owned by user 1, or through a user interface (e.g., input unit 102) owned by the information processing device 10 that enables the learning model unit 100 to perform actions.

[0170] Alternatively, the update of input information D11 in this example can be performed not by user 1 but by the control system 1000 side (e.g., correction confirmation unit 203, etc.).

[0171] Furthermore, feedback information D16 can also be input to the learning model unit 100. The feedback information D16 input to the learning model unit 100 is used, for example, for additional learning by the learning model unit 100. The learning model unit 100 can update model information D102 and / or model reference information D104 based on the input feedback information D16.

[0172] In other respects, it can be the same as other control systems in this embodiment.

[0173] As described above, in this modified example, user 1 can simultaneously confirm the control description D12 output from the learning model unit 100 and interact with the learning model unit 100 through additional instructions, vulnerability negotiation, etc., while correcting the input information D11. Therefore, high precision of the output control description D12 can be achieved. As a result, efficient and high-performance operation of the controlled device 2 can be achieved.

[0174] Variations 1-2.

[0175] Next, a second variation of the control system 1000 will be described. Figure 9 This is a structural diagram showing an example of control system 1000b, which is a variation of control system 1000. Furthermore, elements identical to those in control system 1000 and control system 1000a are labeled with the same reference numerals and their descriptions are omitted.

[0176] exist Figure 9In the control system 1000b shown, the learning model unit 100 differs in that it returns query D17 to the user 1. Examples of query D17 include re-querying for ambiguous or uncertain input information D11, querying for a solution, and requesting re-entry after a change in state or behavior. As a re-query for ambiguous or uncertain input information D11, the learning model unit 100 may also output query D17 requesting more specific information along with a reference prompt to the user 1. Furthermore, as a query for a solution, the learning model unit 100 may output query D17 providing candidate solutions along with a reference position prompt to the user 1. Additionally, as a query for a solution, the learning model unit 100 may output information about the most likely solution and query D17 checking its correctness along with a reference position prompt to the user 1. Furthermore, the learning model unit 100 may temporarily generate an intermediate control description as a human-readable intermediate control description, and output the generated intermediate control description along with query D17 checking its correctness to the user 1.

[0177] The output of D17 can also be queried after step S110 described above.

[0178] The learning model unit 100 can also update the input information D11 or determine the interpretation (meaning assignment) of the input information D11 when it receives a response from user 1 to the query D17.

[0179] The processing of the learning model unit 100 described in this example may also be installed as part of the input unit 102 or preprocessing unit 105 of the learning model unit 100, or as part of the input processing unit 201 (not shown) of the information processing device 10.

[0180] As described above, in this modified example, for the input information D11, a query D17 is output to user 1, and the update or interpretation of the input information D11 is determined based on the user's response, thus eliminating the uncertainty of the input information D11. As a result, the output control description D12 can be made more precise, thereby achieving higher efficiency and performance in the operation of the controlled device 2.

[0181] Variations 1-3.

[0182] Next, a third variation of the control system 1000 will be described. Figure 10 This is a structural diagram showing an example of a control system 1000c, which is a variation of the control system 1000. Furthermore, elements identical to those in control systems 1000, 1000a, and 1000b are labeled with the same reference numerals and their descriptions are omitted.

[0183] Figure 10The control system 1000c shown also includes a status acquisition unit 130. The status acquisition unit 130 acquires feedback information D16 indicating the processing result or status information D15 indicating the state of the device after processing from the processing destination of the control description D12 output by the learning model unit 100 and the execution code D14 generated based on the control description D12. Here, the feedback information D16 or the status information D15 may include information such as execution time or control trajectory information, which is used to determine whether the execution code D14 correctly executed the target control.

[0184] The status acquisition unit 130 may, for example, input the acquired information to the learning model unit 100. Furthermore, the status acquisition unit 130 may, for example, update the device information D13 based on the acquired information. Additionally, the status acquisition unit 130 may, for example, generate supplementary information (including addition, correction, and cancellation) to the input information D11 based on the acquired information, and input this supplementary information D18 to the learning model unit 100. Furthermore, the status acquisition unit 130 may, for example, generate supplementary information (including addition, correction, and cancellation) to the control description D12 based on the acquired information, and input this supplementary information D18 to the learning model unit 100.

[0185] The status acquisition unit 130 may, for example, generate control instructions for new content, or instructions for adding, modifying, or canceling content shown in the input information D11, as supplementary information D18, and input them to the learning model unit 100. Furthermore, the status acquisition unit 130 may, for example, input instructions for eliminating defects contained in the input information D11 or defects contained in the output control description D12, together with the acquired information, as supplementary information D18 to the learning model unit 100.

[0186] The status acquisition unit 130 may, for example, determine whether the acquired information is information indicating normal processing or normal status of the processing destination. If not, it inputs the supplementary information D18, which indicates the correction, addition, or cancellation of the content shown in the input information D11, together with the acquired information, to the learning model unit 100.

[0187] The learning model unit 100 may, for example, update the model information D102 and / or the model reference information D104 based on the input information (state information D15, feedback information D16, supplementary information D18, etc.).

[0188] The generation of supplementary information D18 can also be performed, for example, in step S115 described above. Furthermore, the output destination of supplementary information D18 can also include parts other than the learning model unit 100. The control system 1000 can, for example, output the supplementary information D18 generated by the status acquisition unit 130 to user 1 or other devices not shown.

[0189] Furthermore, the status acquisition unit 130 can also acquire the running results of the simulator (not shown) of the object device 2 or the running results in the debug mode of the object device 2 without actually running the object device 2. The debug mode of the object device 2 refers to a mode in which execution code is executed on the control board of the object device 2 but no actual device control is performed, and only the internal state is updated; it is also called the idle operation mode. By utilizing the debug mode, it is possible to safely attempt to execute code D14 on the object device 2 in a state close to actual control.

[0190] Not limited to this variation, as a method for determining the legitimacy of the control description D12 output from the learning model unit 100 and the executable code D14 generated based on the control description D12 without actually causing the object device 2 to operate, the executable code generation unit 120 may also be connected to switch between the object device 2 and the simulator as the output destination of the executable code D14. The simulator includes a simulator that makes the icon of the object device 2 move in the augmented reality space. Furthermore, the executable code generation unit 120 may also provide information indicating whether to execute in normal mode or debug mode when outputting the executable code D14 to the object device 2.

[0191] The processing of the status acquisition unit 130 in this example can also be installed as part of the input unit 102, preprocessing unit 105 and postprocessing unit 106 of the learning model unit 100, or the input processing unit 201, output confirmation unit 202 and correction confirmation unit 203 (all not shown) of the information processing device 10.

[0192] In other respects, it can be the same as other control systems in this embodiment.

[0193] As described above, in this modified example, for the input information D11, the state acquisition unit 130 obtains feedback information D16 indicating the processing result or state information D15 indicating the state of the processed device from the target device 2, which is the output destination of the model output data D103 and / or information generated based on the model output data D103, or the execution code generation unit 120. Based on the obtained information, it appropriately issues supplementary information D18 to the learning model unit 100. This enables high precision in the control description D12, thereby achieving high efficiency and high performance in the operation of the controlled target device 2.

[0194] Furthermore, in this modified example, for example, human-machine collaboration (state acquisition unit 130) can improve the accuracy of input to the learning model unit 100, thus also helping to reduce the workload of user 1.

[0195] Implementation method 2.

[0196] Next, this second embodiment will be described. In this embodiment, an example of using a learning model to assist in operations related to the control of the target device will be described.

[0197] The following considers the control of various control devices within a factory, such as PLCs, machining centers, robots, sensors, material handling equipment, and other machinery. While skilled operators may be familiar with a wide variety of control devices and their complex control methods, sometimes changes in configuration may necessitate operation by less experienced operators. Furthermore, when introducing new control devices (including version upgrades), all operators must be informed of the corresponding control methods; insufficient knowledge can lead to errors.

[0198] In such cases, even if the specific control methods, such as control commands, control signals, control codes, or commands to the controller corresponding to the control equipment, are unknown, reliable control can still lead to increased efficiency and performance in operations, making it the preferred option.

[0199] Furthermore, the application scenarios for controlling the equipment are not limited to the factory, and the application scenarios of this implementation method are also not limited to the factory.

[0200] Figure 11 This is a structural diagram showing an example of a control system 2000 according to Embodiment 2. Figure 11 The control system 2000 shown is a control system for controlling equipment using a learning model, and includes a learning model unit 200 and an equipment information storage unit 210 (referred to as equipment information DB in the figure).

[0201] When input information D21 is input, the learning model unit 200 outputs control command D22. For example, when input information D21 is input, the learning model unit 200 outputs control command D22 based on model information D102. The structure of the learning model unit 200 can be basically the same as that of the learning model unit 100 in Embodiment 1.

[0202] In this embodiment, the learning model unit 200 is a model and its operating environment configured to output a control command D22 corresponding to the input information D21 when input information D21 is input. Alternatively, the learning model unit 200 may also be configured to generate and output a control command D22 based on the input information D21, device information D23, and other referable information in the learning model unit 200 when input information D21 is input.

[0203] In this embodiment, the input information D21 includes information representing control content for the target device 2. The input information D21 may, for example, be text, images, speech, or a combination thereof representing the control content for the target device 2. The input information D21 may also be text, images, speech, or a combination thereof representing multiple control contents for the target device 2. Furthermore, the input information D21 may also include information representing control content that occurs continuously in time; in this case, it may be time-series data with a defined data structure including text, images, speech, or a combination thereof representing such control content. The method of representing the control content is based on the input format of the model used by the learning model unit 200, but is not limited to this if the learning model unit 200 includes error processing, correction processing, or conversion processing in its preceding stages.

[0204] The method of representing the control content in the input information D21 can be the same as in Embodiment 1. For example, based on determining the control to be performed on the target device 2, the values ​​of the parameters used to perform the control and the state after the control can be specified. In this case, the input information D21 may include, for example, information determining the control and information indicating the values ​​of the parameters used to perform the control or the state after the control. Furthermore, the input information D21 may include not only information directly representing the control content for the target device 2, but also information indirectly represented by operation content corresponding to the control content, the user 1's words and actions, the image of the target device 2, or the same control commands from other models.

[0205] Control instruction D22 contains information related to the control of object device 2, expressed in a prescribed form that can be determined by object device 2 or an interface requesting control of object device 2. Control instruction D22 may also contain information indicating a control request for object device 2. Control instruction D22 may be, for example, a control command, control signal, or control code for object device 2. Furthermore, control instruction D22 may also be, for example, a command described in a form processed by a prescribed controller corresponding to object device 2.

[0206] The device information storage unit 210 stores device information D23, which is information related to the target device 2. The processing of the device information storage unit 210 and device information D23 is basically the same as that of the device information storage unit 110 and device information D13 in Embodiment 1. Furthermore, in this embodiment, the device information D23 may, for example, include information for controlling the target device 2. The device information D23 is used, for example, as supplementary information when the learning model unit 200 outputs control command D22. Hereinafter, in this embodiment, information indicating the state of the target device 2 will sometimes be specifically referred to as state information D25.

[0207] In this embodiment, the learning model unit 200 may be, for example, a language learning model such as LLM that takes natural language as input and obtains output results, and its operating environment. Furthermore, the learning model unit 200 may be, for example, an image learning model such as VLM that takes images as input and obtains output results, and its operating environment. Additionally, the learning model unit 200 may be, for example, a multimodal model that takes natural language and images as input and obtains output results, and its operating environment. In this case, the input information D21 may also be input via text data, image data, a combination of text data and image data, or a data form that can be converted into them (speech data, animated images as a combination of speech data and image data, etc.). Furthermore, the learning model used in this embodiment is not limited to the models described above.

[0208] In this embodiment, for the sake of simplicity, sometimes the symbols corresponding to the constituent elements provided in the learning model unit 100 are directly used to describe the constituent elements provided in the learning model unit 200, but this is only applicable to those provided in the learning model unit 200. The same applies in other embodiments.

[0209] In this embodiment, the input information D21 corresponds to the model input data D101. Furthermore, the control command D22 corresponds to the model output data D103. The learning model unit 200 (especially the model control unit 101) may, for example, be configured to, upon receiving the input information D21, output the control command D22 corresponding to the input information D21 based on the model information D102 and, if necessary, based on the model reference information D104.

[0210] Furthermore, in this case, the model generation unit 107, corresponding to the learning model unit 200, can, for example, use the model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D21 that can be input to the model control unit 101. Additionally, the model generation unit 107 can, for example, use the model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D21 that can be input to the model control unit 101 and candidates for corresponding control commands D22.

[0211] Code D26 is feedback information indicating the control result of object device 2. In this embodiment, status information D25 and / or feedback information D26 may also be obtained from the output destination of model output data D103 and / or information generated based thereon. For example, the control system 2000 may output the obtained status information D25 and / or feedback information D26 as information indicating the control result to user 1, learning model unit 200, or other devices not shown. Furthermore, the control system 2000 may also generate supplementary information D28 about the input and output data of learning model unit 200 based on the obtained status information D25 and / or feedback information D26, and issue it to user 1, learning model unit 200, or other devices not shown. In addition, the control system 2000 may also be configured to return query D27 to user 1 if the input information D21 contains ambiguous or uncertain information. The processing of query D27 is the same as that of query D17 in embodiment 1.

[0212] Figure 12 This is a structural diagram illustrating another example of a control system 2000. (See diagram below.) Figure 12 As shown, the control system 2000 may also include a status acquisition unit 230 for acquiring status information D25 and / or feedback information D26 and issuing supplementary information D28. The status acquisition unit 230 is the same as the status acquisition unit 130 in Embodiment 1.

[0213] In this embodiment, the target device 2 is not particularly limited. Furthermore, while it is assumed that the target device 2 is a device capable of receiving and actually controlling the control command D22, it is not limited to this if a conversion device such as a controller or converter is included between the target device 2 and the control device. In this case, the conversion device can control the target device 2 by receiving the control command D22.

[0214] In this embodiment, the input information D21 received by the control system 2000 can be referred to as information related to the working environment, specifically the requirements (in this case, the control content requested for the object device 2) in the environment where the operation of the object device 2 is performed. Therefore, the input information D21 received by the control system 2000 can be referred to as an example of first information representing the requirements in the working environment. Furthermore, the control instruction D22 can be referred to as information corresponding to such input information D21 for the operation (the operation involved in the control of the object device 2). Hereinafter, the control instruction D22 output from the working environment of the learning model, which is input based on the model input data of the input information D21, to the specified output destination will sometimes be referred to as second information.

[0215] Next, the operation of the control system 2000 of this embodiment will be explained. Figure 13 This is a flowchart illustrating an example of the operation of the control system 2000.

[0216] exist Figure 13 In the example shown, firstly, the control system 2000 accepts the input information D21 (step S210). For example, the input unit 102 or the input processing unit 201 described above can also accept the input information D21. The accepted input information D21 is input to the learning model unit 200 as model input data D101.

[0217] Next, the control system 2000 performs the generation process of control command D22 using the learning model unit 200 (step S211). In step S211, the learning model unit 200 (more specifically, the model control unit 101) outputs control command D22 corresponding to the input information D21 based on the model information D102 and the input information D21, and if necessary based on the model reference information D104 including the device information D23.

[0218] In step S211, the learning model unit 200 may, for example, use a learning model capable of generating binary data, and generate control instructions D22 for binary data based on the input information D21. Alternatively, the learning model unit 200 may, for example, use a learning model capable of generating text data, and generate control instructions D22 for text data based on the input information D21. Furthermore, the learning model unit 200 may, for example, use a learning model capable of generating image data, and generate control instructions D22 for image data based on the input information D21. Additionally, the learning model unit 200 may, for example, use a learning model capable of generating speech data, and generate control instructions D22 for speech data based on the input information D21.

[0219] In step S211, the above processing may also be performed by the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 200.

[0220] The control command D22 output from the learning model unit 200 is input to the target device 2, for example (step S212). The input of the control command D22 to the target device 2 can be directly input from the control system 2000 (more specifically, the learning model unit 200 or the information processing device 10 that serves as its operating environment), or indirectly input via a communication network or other devices (servers, various conversion devices, etc.).

[0221] Therefore, the target device 2 performs actions according to the input control command D22.

[0222] The control system 2000 can also obtain status information D25 and feedback information D26 (step S213) when the result of outputting control command D22 is that the target device 2 is controlled, and the state of the target device 2 changes, or when there is feedback from the target device 2. In addition, the processing of step S213 is not necessary and can be omitted appropriately.

[0223] The control system 2000 may also repeat the process of steps S210 to S213 multiple times (for example, until the desired control of the target device 2 is completed).

[0224] As described above, according to this embodiment, even if user 1 does not know the specific control method for object device 2, control command D22 can be generated based on input information D21 input from user 1, and object device 2 can be controlled based on the generated control command D22. Therefore, the operation related to the control of object device 2 can be made more efficient and sophisticated.

[0225] Furthermore, according to this embodiment, even based on ambiguous information, the device can be controlled to an appropriate state.

[0226] Implementation method 3.

[0227] Next, this third embodiment will be described. In this embodiment, an example of using a learning model to assist in operations related to the target device will be described.

[0228] The following considers operating various devices such as air conditioners, refrigerators, televisions, lighting, washing machines, projectors, various sensors, and communication equipment within a home or building. In recent years, even consumer-oriented devices have offered increasingly sophisticated functions and more complex controls. Efforts have been made to simplify complex controls through user interfaces, remote controls, and other controllers; however, even with these improvements, it is difficult to remember all the operations, and while desired functions are provided, they are sometimes not easily accessible.

[0229] Furthermore, even though they are the same function, different models often have different function names, subtle differences in functions, and different control methods. When importing different models through replacement or purchase, you have to start from scratch to remember their differences, which is quite tedious.

[0230] In addition, depending on the device, there are devices that can automatically control the user to an appropriate state by pre-memorizing past operation history and understanding the action environment. However, when the appropriate state varies from person to person, it is sometimes difficult to control accurately in scenarios where many people are gathered or even when there is only one person, the appropriate state may vary depending on physical condition.

[0231] In such cases, even if the specific operating method is unknown and the operator is not familiar with setting the appropriate state, if the operation to achieve the desired state can be performed simply, it will lead to higher efficiency and performance of the operation, and is therefore preferred.

[0232] Furthermore, the scenarios for operating the device are not limited to homes or buildings, and the application scenarios of this embodiment are not limited to homes or buildings.

[0233] Figure 14 This is a structural diagram showing an example of the control system 3000 of Embodiment 3. Figure 14 The control system 3000 shown is a control system for operating equipment using a learning model. It includes a learning model unit 300, an equipment information storage unit 310 (referred to as equipment information DB in the figure), an input interface 311 (referred to as input IF in the figure), and an output interface 312 (referred to as output IF in the figure).

[0234] When input information D31 is input, the learning model unit 300 outputs operation instruction D32. For example, when input information D31 is input, the learning model unit 300 outputs operation instruction D32 based on model information D102. The structure of the learning model unit 300 can be basically the same as that of the learning model unit 100 in Embodiment 1.

[0235] In this embodiment, the learning model unit 300 is configured to output an operation instruction D32 corresponding to the input information D31 when input information D31 is input. Alternatively, the learning model unit 300 may also be configured to generate and output an operation instruction D32 based on the input information D31, device information D33, and other referable information in the learning model unit 300 when input information D31 is input.

[0236] In this embodiment, the input information D31 includes information representing the operation content requested for the target device 2. The input information D31 may, for example, be text, images, speech, or a combination thereof representing the operation content for the target device 2. The input information D31 may also be text, images, speech, or a combination thereof representing multiple operation contents for the target device 2. Furthermore, the input information D31 may also include information representing operation contents performed sequentially in time; in this case, it may be time-series data containing text, images, speech, or a combination thereof representing such operation contents. The method of representing the operation content is based on the input form of the model used by the learning model unit 300, but is not limited to this if the learning model unit 300 includes error handling, correction processing, or conversion processing in its preceding stages.

[0237] As an example of how the operation content in the input information D31 is represented, the values ​​of the parameters used to perform the operation and the state after the operation can also be specified based on determining the operation to be performed on the target device 2. In this case, the input information D31 may, for example, include information determining the operation and information indicating the values ​​of the parameters used to perform the operation or the state after the operation. The values ​​of the parameters used to perform the operation may, for example, include values ​​related to the type of operation (ON / OFF, etc.), orientation, quantity, and time. Furthermore, the input information D31 may include not only information directly representing the operation content for the target device 2, but also information indirectly represented using control content corresponding to the operation content, the user 1's words and actions, images of the target device 2, or the same operation instructions from other models.

[0238] Operation instruction D32 contains information related to the operation of object device 2, expressed in a prescribed form that can be discerned by object device 2 or an interface (including a person) requesting control of object device 2. Operation instruction D32 may also contain information indicating an operation request or control request for object device 2. Operation instruction D32 may be, for example, an operation command, operation signal, operation code, control command, control signal, or control code for object device 2. Furthermore, operation instruction D32 may also be, for example, a command described in a form processed by a prescribed controller corresponding to object device 2. Operation instruction D32 can be said to be the addition of the concept of operation-related information to the aforementioned control instruction D22. Furthermore, if the interface is a person, i.e., if a person requests control of object device 2, operation instruction D32 may be information indicating the operation method of object device 2, shown in a discernible form.

[0239] The device information storage unit 310 stores device information D33, which is information related to the target device 2. The processing of the device information storage unit 310 and device information D33 is basically the same as that of the device information storage unit 110 and device information D13 in Embodiment 1. Furthermore, in this embodiment, the device information D33 may also include, for example, information for operating the target device 2. The device information D33 may include, for example, information indicating the steps of the operation actually performed on the target device 2 for the operation content. In addition, the device information D33 may include, for example, commands, signals, codes, etc., issued to the target device 2. The device information D33 is used, for example, as supplementary information when the learning model unit 300 outputs the operation instruction D32. Hereinafter, in this embodiment, information indicating the state of the target device 2 will sometimes be specifically referred to as state information D35.

[0240] For example, device information D33 may include mechanical information such as the manual for device 2 and control code information such as the communication specifications for the remote control used to operate device 2. The manual may also include images.

[0241] Input interface 311 is an interface that accepts input information D31 from user 1 and inputs it into the learning model unit 300. Input interface 311 may also be an interface that converts the input information D31 from user 1 into data that conforms to the input of the learning model unit 300 and outputs it. Alternatively, input interface 311 may be provided as an example of the input unit 102 described above.

[0242] Output interface 312 is an interface that receives operation commands D32 from the learning model unit 300 and outputs them to a specified output destination. Alternatively, output interface 312 may be provided as an example of the output unit 103 described above. Output interface 312 may also be an interface that converts the operation commands D32 output from the learning model unit 300 into data conforming to a specified output destination and outputs it. In this embodiment, the output destination of output interface 312 may include the object device 2, the controller 4 (not shown), a specified display 7 (not shown), the user 1's operating terminal (not shown), and non-physical spatial projection displays such as holograms or projections.

[0243] In this embodiment, the learning model unit 300 may be, for example, a language learning model such as LLM that takes natural language as input and obtains output results, and its operating environment. Furthermore, the learning model unit 300 may be, for example, an image learning model such as VLM that takes images as input and obtains output results, and its operating environment. Additionally, the learning model unit 300 may be, for example, a multimodal model that takes natural language and images as input and obtains output results, and its operating environment. In this case, the input information D31 may also be input via text data, image data, a combination of text data and image data, or data forms that can be converted into them (speech data, animated images as a combination of speech data and image data, etc.). Furthermore, the learning model used in this embodiment is not limited to the models described above.

[0244] In this embodiment, the input information D31 corresponds to the model input data D101. Furthermore, the operation instruction D32 corresponds to the model output data D103. The learning model unit 300 (especially the model control unit 101) may, for example, be configured to, upon receiving the input information D31, output the operation instruction D32 corresponding to the input information D31 based on the model information D102 and, if necessary, based on the model reference information D104.

[0245] Furthermore, in this case, the model generation unit 107, corresponding to the learning model unit 300, can, for example, use the following model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D31 that can be input to the model control unit 101. Furthermore, the model generation unit 107 can, for example, use the following model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D31 that can be input to the model control unit 101 and candidates for corresponding operation instructions D32.

[0246] Although figures are omitted, in this embodiment, status information D35 and / or feedback information D36 can be obtained from the model output data D103 of the learning model unit 300 and / or the output destination of information generated based thereon. The control system 3000 may, for example, output the obtained status information D35 and / or feedback information D36 as information indicating a response result to user 1, the learning model unit 300, or other devices not shown. Furthermore, the control system 3000 may be configured to return an inquiry D37 to user 1 if the input information D31 contains ambiguous or uncertain information. Additionally, the control system 3000 may generate supplementary information D38 regarding the input and output data of the learning model unit 300 based on the obtained status information D35 and / or feedback information D36, and issue it to user 1, the learning model unit 300, or other devices not shown. The processing of status information D35, feedback information D36, inquiry D37, and supplementary information D38 can be substantially the same as in Embodiment 1.

[0247] Furthermore, the control system 3000 may also include a status acquisition unit 330 (not shown), which acquires status information D35 and / or feedback information D36, and issues supplementary information D38 as needed. The status acquisition unit 330 is the same as the status acquisition unit 130 in Embodiment 1.

[0248] In this embodiment, the target device 2 is not particularly limited. Furthermore, while it is assumed that the target device 2 is a device capable of receiving the operation command D32 and performing control corresponding to the operation content indicated by the operation command D32, this limitation does not apply if a conversion device or operator, such as a controller 4 or a converter, is included between the target device 2 and the conversion device. In this case, the conversion device or operator can simply accept the operation command D32 to operate the target device 2.

[0249] In this embodiment, the input information D31 received by the control system 3000 can be referred to as information related to the operating environment, specifically the requirements (in this case, the operation content requested for the object device 2) within the environment where the operation is performed. Alternatively, the input information D31 can also be a command related to the requirements (in this case, the operation content requested for the object device) within the operating environment where the operation is performed. Therefore, the input information D31 received by the control system 3000 can be referred to as an example of first information representing a requirement or command in the operating environment. Furthermore, the operation instruction D32 can be referred to as information corresponding to such input information D31 for the operation (the operation involved in the operation of the object device 2). Hereinafter, the operation instruction D32 output from the operating environment of the learning model, which is input with model input data based on the input information D31, to a predetermined output destination will sometimes be referred to as second information.

[0250] Next, the operation of the control system 3000 in this embodiment will be explained. Figure 15 This is a flowchart illustrating an example of the operation of the control system 3000.

[0251] exist Figure 15 In the example shown, firstly, the input interface 311 of the control system 3000 receives input information D31 (step S310). The received input information D31 is input to the learning model unit 300 as model input data D101.

[0252] Input information D31 is a command from user 1 to object device 2. Furthermore, the command can also consist of multiple operations performed on object device 2. That is, the command can be a simple operation such as "increase the temperature by 1°C," or it can be a complex command requiring multiple operations on object device 2.

[0253] Next, the control system 3000 performs the generation process of the operation instruction D32 using the learning model unit 300 (step S311).

[0254] In step S311, the learning model unit 300 inputs input information D31 (first information) into the learning model and outputs operation instruction D32 (second information). This operation instruction D32 represents a series of controls, i.e., sequential controls, in the target device 2. Sequential control refers to a series of controls in the target device 2 used to implement multiple operations. These multiple operations are required to implement the command input as input information D31. For example, the learning model unit 300 can output information showing sequential control through the control code of the target device 2 to the target device 2 via the output interface 312 as operation instruction D32 (second information).

[0255] Specifically, as described below.

[0256] In step S311, the learning model unit 300 (more specifically, the model control unit 101) generates and outputs an operation instruction D32 corresponding to the input information D31 based on the model information D102 and the input information D31, and as needed based on the model reference information D104 containing the device information D33.

[0257] In step S311, the learning model unit 300 may, for example, use a learning model capable of generating binary data, and generate an operation instruction D32 for binary data based on the input information D31. Alternatively, the learning model unit 300 may, for example, use a learning model capable of generating text data, and generate an operation instruction D32 for text data based on the input information D31. Furthermore, the learning model unit 300 may, for example, use a learning model capable of generating image data, and generate an operation instruction D32 for image data based on the input information D31. Additionally, the learning model unit 300 may, for example, use a learning model capable of generating speech data, and generate an operation instruction D32 for speech data based on the input information D31.

[0258] In step S311, the above processing may also be performed by the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 300.

[0259] The operation command D32 output from the learning model unit 300 is output to a designated output destination, for example, via the output interface 312. Here, the designated output destination may also be the object device 2, the controller 4, the designated display 7 (not shown), the user 1's operation terminal (not shown), or a spatial projection display such as a hologram or projection, which is not a physical form. When the operation command D32 is input to the designated output destination, the object device 2 is operated according to the input operation command D32 (step S312).

[0260] For example, output interface 312 can also output operation command D32 to the target device 2. In this case, the target device 2, which receives operation command D32 (e.g., operation command, operation signal, operation code, control command, control signal, or control code), can also perform actual control according to operation command D32. Furthermore, output interface 312 can also output operation command D32 to the controller 4 corresponding to the target device 2. In this case, the controller 4, which receives operation command D32 (e.g., commands, operation commands, operation signals, operation codes, etc., which are indirect control information for the target device 2), can also operate the target device 2 according to operation command D32. The controller 4 can also output direct control information such as control codes to the target device 2 based on the control information represented by the received operation command D32, thereby operating the target device 2. Here, controller 4 can be, for example, an operation panel provided with the target device 2, or a remote control corresponding to the target device 2 directly operated by the user. Controller 4 includes controllers inherent to the target device 2 and general-purpose controllers. Furthermore, output interface 312 can also output operation command D32 to the user 1's operating terminal or a specified display. In this case, the user 1's operating terminal or display that received the operation instruction D32 (e.g., information indicating the operation method) displays the operation instruction D32. Then, the user 1 can also operate the object device 2 or controller 4 by referring to the displayed operation instruction D32.

[0261] The input of the operation command D32 to the output destination can be directly input from the control system 3000 (more specifically, the learning model unit 300 or the information processing device 10 that serves as its operating environment), or indirectly input via a communication network or other devices (servers, various conversion devices, etc.).

[0262] Therefore, the object device 2 performs an action according to the operation instruction D32.

[0263] The result of the control system 3000 outputting operation command D32 is that the target device 2 is operated, and the state of the target device 2 changes. If there is feedback from the target device 2 whose state has changed, the control system 3000 can also obtain state information D35 and feedback information D36 (step S313). Furthermore, step S313 is not mandatory and can be omitted appropriately.

[0264] The control system 3000 may also repeat the processing of steps S310 to S313 multiple times (for example, until the desired operation is completed on the target device 2).

[0265] As described above, according to this embodiment, even if user 1 does not know the specific operation method for object device 2, operation instruction D32 can be generated based on the input information D31 input from user 1, and object device 2 can be operated based on the generated operation instruction D32. Therefore, the efficiency and sophistication of operations related to the operation of object device 2 can be achieved.

[0266] Furthermore, according to this embodiment, the device can be operated in an appropriate state even based on ambiguous information. Moreover, according to this embodiment, the device can be operated in an appropriate state regardless of the device itself, even without having learned how to operate it.

[0267] Furthermore, even simple operations for the user can sometimes involve complex commands requiring multiple actions. Examples of such commands include natural language input such as "The morning is naturally cool with low humidity; it starts to get cooler in the afternoon." Additionally, image-based commands such as "An image showing the desired changes in room temperature and humidity" can be used. Even with complex commands requiring multiple actions, the learning model can output information representing sequential control—a series of controls within the target device used to execute the command. Therefore, according to this embodiment, the efficiency or performance of the user's work can be further improved.

[0268] Variation 3-1.

[0269] Next, a variation of the control system 3000 will be described. Figure 16 This is a structural diagram showing an example of a control system 3000a, which is a variation of the control system 3000 of this embodiment. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0270] Figure 16 The control system 3000a shown also includes an input determination unit 31. The input determination unit 31 is a unit that, upon receiving input information D31, parses the input information D31 and switches the control destination for the input information D31. In this modified example, the input determination unit 31 switches the control destination for the input information D31 between the learning model unit 300 and the output interface 312 or the target device 2 via the output interface.

[0271] The input judgment unit 31 uses the command rules to determine whether the command represented by the input information D31 (first information) conforms to the command contained in the command rules.

[0272] A command rule is information that maps commands to a device to control codes sent to that device to execute those commands. For example, a command rule maps a command to device 2 to the control code of device 2. Furthermore, the command rule specifies information input as input information D31 in the command to device 2. Moreover, the information corresponding to the command to device 2, i.e., the control code of device 2, can be output as an operation instruction D32 in the command rule. For example, the information corresponding to the command to device 2 can also be a control signal, parameter, etc.

[0273] If the command represented by input information D31 conforms to the commands included in the command rules, the input judgment unit 31 outputs the control code corresponding to the command represented by input information D31 to the target device 2 via the output interface 312. Furthermore, if the command represented by input information D31 does not conform to the commands included in the command rules, the input judgment unit 31 outputs input information D31 to the learning model unit 300.

[0274] Alternatively, the input judgment unit 31 can also use machine learning to determine whether the command for the target device 2 corresponds to the control code of the target device 2.

[0275] Furthermore, when the input determination unit 31 determines that the command represented by the input information D31 corresponds to the control code of the target device 2, it outputs the control code corresponding to the command represented by the input information D31 to the target device 2 via the output interface 312. Conversely, when the input determination unit 31 determines that the command represented by the input information D31 does not correspond to the control code of the target device 2, it outputs the input information D31 to the learning model unit 300.

[0276] The following provides further explanation.

[0277] The input determination unit 31 can, for example, switch the control destination for the input information D31 based on whether the input information D31 conforms to the command rules for the operation of the target device 2. If the input information D31 conforms to the command rules for the operation of the target device 2, the input determination unit 31 can directly input the input information D31 to the output interface 312. Conversely, if the input information D31 does not conform to the command rules for the operation of the target device 2, the input determination unit 31 can input the input information D31 to the learning model unit 300.

[0278] For example, a rule-based model can be used to determine whether the operation conforms to the command rules. Here, the input judgment unit 31 can also be a relatively lightweight learning model compared to the learning model unit 300. That is, if the model used in the learning model unit 300 is set as the higher-level AI, the model used in the input judgment unit 31 can also be a lighter-weight lower-level AI with lower processing power. In this way, in the control system 3000a, processing can also be performed collaboratively by the lower-level AI and the higher-level AI.

[0279] Alternatively, in the output interface 312, multiple outputs that are considered to be operation instructions D32 can be sorted according to their appropriateness, arranged from outputs with high appropriateness to outputs with low appropriateness, or the outputs with high appropriateness can be displayed with larger text or area than the outputs with low appropriateness.

[0280] In order to distinguish the information input into the output interface 312, the operation instruction D32 output from the learning model unit 300 is sometimes referred to as operation instruction D32a, and the input information D31 output to the output interface 312 is referred to as operation instruction D32b.

[0281] In this example, output interface 312 can simply be an interface that accepts operation command D32a or operation command D32b and outputs to the specified output destination.

[0282] Next, the operation of the control system 3000a in this modified example will be explained. Figure 17 This is a flowchart illustrating an example of the operation of the control system 3000a.

[0283] exist Figure 17 In the example shown, firstly, the input interface 311 of the control system 3000a receives the input information D31 (step S310). The received input information D31 is then input to the input judgment unit 31.

[0284] Next, the input determination unit 31 determines whether the input information D31 conforms to the command rules for the operation of the target device 2 (step S321). Here, if it is determined that the input information D31 conforms to the command rules for the operation of the target device 2 ("Yes" in step S321), the input information D31 is input to the output interface 312 (proceeding to step S322). On the other hand, if it is determined that the input information D31 does not conform to the command rules for the operation of the target device 2 ("No" in step S321), the input information D31 is input to the learning model unit 300 (proceeding to step S311).

[0285] Processing of steps S311 to S313 Figure 15 The example shown is the same.

[0286] In step S322, the output interface 312 outputs the control code corresponding to the input information D31 as an operation instruction D32b to the designated output destination in the command rules. As a result, the target device 2 performs an action according to the operation instruction D32b.

[0287] In other respects, it can be the same as other control systems in this embodiment.

[0288] As described above, according to this variation, when the input from user 1 conforms to the command rules for operating object device 2, object device 2 can be operated according to that input. On the other hand, when it does not conform, object device 2 can be operated using a learning model. Therefore, it is possible to further improve the efficiency and reduce the cost of operations related to the operation of object device 2.

[0289] Variation 3-2.

[0290] Next, other variations of the control system 3000 will be described. In this variation, a learning model is used to generate coordinated operating instructions that include multiple inputs.

[0291] Figure 18 This is a structural diagram showing an example of a control system 3000b, which is a variation of the control system 3000 of this embodiment. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0292] exist Figure 18 In the control system 3000b shown, the input interface 311 receives input information D31 from multiple users 1.

[0293] For example, input interface 311 accepts multiple commands from different users 1-1, 1-2, ..., 1-n (n is a natural number greater than 2) as multiple input information D31 (the first information).

[0294] Then, the learning model unit 300 coordinates multiple commands represented by multiple input information D31. As a result of the coordination, the learning model unit 300 outputs information representing a sequential control of the multiple commands as operation instruction D32 (second information).

[0295] Input interface 311 receives input information D31 from multiple users 1 and inputs it into the learning model unit 300. At this time, input interface 311 can accept input information D31 with information of the user 1 who is the input source, or it can accept input information D31 based on the user 1 who is the input source and the information of the input source added by input interface 311, or it can accept input information D31 without doing anything.

[0296] The learning model unit 300 can be configured to output a model and its operating environment of the operation instruction D32 corresponding to the input information D31 group when the input interface 311 receives the input information D31 group. The operation instruction D32 corresponding to the input information D31 group is information that indicates a sequential control of multiple commands represented by multiple input information D31 groups. The learning model unit 300 can also be configured to generate and output a model and its operating environment of the operation instruction D32 based on the input information D31 group, device information D33, and other referable information in the learning model unit 300 when the input information D31 group is input.

[0297] Next, the operation of the control system 3000b in this modified example will be explained. Figure 33 This is a flowchart illustrating an example of the operation of the control system 3000b.

[0298] Input interface 311 receives multiple input information D31 from multiple users 1 and inputs it into learning model unit 300 (step S330).

[0299] The learning model unit 300 coordinates multiple commands represented by multiple input information D31 and outputs information representing a sequential control of the multiple commands as an operation instruction D32 (step S331).

[0300] Processing of steps S312 to S313 Figure 15 The example shown is the same.

[0301] For example, the learning model unit 300 may also use a language learning model such as LLM, which takes natural language as input and outputs the result, to extract suitable solutions in the language space (more specifically, in the feature vector space containing information of the language space), thereby generating and outputting operation instructions D32 that are compromise solutions for different operation contents represented by the input information D31. At this time, the learning model unit 300 may also refer to the history of the input information D31 of each user 1 as an input source and / or the history of the operation instructions D32 of each user 1 as an input source.

[0302] In other respects, it can be the same as other control systems in this embodiment.

[0303] As described above, according to this modified example, even when information related to different operation content is input from multiple users, the learning model unit 300 can generate more appropriate operation instructions D32 that coordinate these contents, thus enabling further high functionality of operations related to the operation of the object device 2.

[0304] According to this variation, for example, it is possible to coordinate the needs of multiple users with different preferences for thermal environment and output control codes and parameters that are the best or better suitable equipment for multiple needs. In the learning model unit, when opposite words appear, such an output is averaged, so that commands that utilize both opposite words can be issued.

[0305] Thus, according to the control system of this variant, the coordinated result is the ability to output more complex sequential control. Specifically, it is possible to output complex sequential control such as "blowing air only on a person who is hot (e.g., on the right), while directing the airflow downwards on the left side so as not to blow on the person, and the room temperature rises by 0.5°C instead."

[0306] Variation 3-3.

[0307] Next, other variations of the control system 3000 will be described. In this variation, a learning model is used to generate the user interface for the operation screen.

[0308] Figure 19 This is a structural diagram showing an example of a control system 3000c, which is a variation of the control system 3000 of this embodiment. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0309] Figure 19 The control system 3000c shown also has an operation screen user interface 3 (referred to as operation screen UI in the figure). In addition, the learning model unit 300 generates an operation screen that actually performs the operation corresponding to the input information D31 on the target device 2, as operation instruction D32.

[0310] The operation screen generated by the learning model unit 300 can be, for example, a screen API with the following functions: accepting user input along with a description of the operation content, and outputting control commands D34, such as control codes, corresponding to the accepted operation input. API is short for Application Programming Interface. Here, the output of control codes, etc., corresponding to the operation input also includes a method of sequentially outputting multiple control commands D34 based on one operation input. Furthermore, this operation screen can also be a screen API with operation descriptions corresponding to two or more different operation contents, operation input acceptance, and control command output. The learning model unit 300 can also, for example, extract operation information representing two or more different operation contents as operation commands D32 corresponding to input information D31, and generate a screen API corresponding to each operation information, which has operation input acceptance and control command output.

[0311] In addition, the operation screen generated by the learning model unit 300 can also be a screen after changing the display method of the existing operation screen in the following ways: emphasizing the display of the operation part corresponding to the corresponding operation content, restricting the display of operation functions, or changing the position and shape (shape, size, color, etc.) of the UI components on the display screen.

[0312] The operation screen user interface 3 is an interface that displays an operation screen for the target device 2 and accepts user input related to operation on the operation screen. The operation screen user interface 3 can also be implemented, for example, through a touch panel display, a controller equipped with operation buttons and a display section. Furthermore, the operation screen user interface 3 can also be implemented through a display device such as a monitor that cooperates with an operation input device such as a mouse.

[0313] The operation screen user interface 3 can also be provided on the device operation device of the target device 2. The operation screen user interface 3 can also be an example of the device operation device of the target device 2. Furthermore, the device operation device can also have an input interface 311. That is, the device operation device can also be a remote control for inputting commands from user 1 to the target device 2.

[0314] In addition, the device operating device may also have the following function: by displaying the sequential controls agreed upon by user 1 on the display screen, it can assist user 1 in inputting commands to the target device 2.

[0315] Furthermore, in this modified example, the output interface 312 outputs the operation instruction D32 (operation screen) output from the learning model unit 300 to the operation screen user interface 3.

[0316] Furthermore, in this modified example, the learning model unit 300 may also have the function of confirming the operation expected by the user 1 through a dialogue. In such a case, the learning model unit 300 may, for example, after displaying the operation screen as operation command D32, and upon receiving information indicating a request to retrieve operation command D32 again, retrieve operation command D32 again based on changes such as changes to a portion of the input information, a portion of the model parameters, or the reference destination of the reference information.

[0317] Next, the operation of the control system 3000c in this modified example will be explained. Figure 34 This is a flowchart illustrating an example of the operation of the control system 3000c.

[0318] Processing in step S310 and Figure 15 The steps are the same as in step S310.

[0319] In addition, as mentioned above, the device operating device can also assist user 1 in inputting commands to the target device 2 by displaying the sequential controls previously agreed upon by user 1 on the display screen.

[0320] In step S341, the learning model unit 300 generates an operation screen representing sequential control as operation instruction D32 (second information) representing sequential control. The operation screen representing sequential control is an operation screen that displays sequential control. For example, for input information D31 "The morning is a naturally cool state with low humidity, and it starts to cool down in the afternoon," the following sequential control is output as operation instruction D32. Examples of sequential control could be "Automatic dehumidification operation from 8:00 AM to 12:00 AM, and automatic operation from 0:00 PM to 5:00 PM at a temperature -2°C lower than the current temperature." The learning model unit 300 generates an operation screen to prompt the user 1 with such sequential control.

[0321] In step S342, the learning model unit 300 displays an operation screen on the operation screen user interface 3 (device operation device). The learning model unit 300 allows the user 1 to select whether to agree to sequential control via the operation screen.

[0322] In step S343, it is determined whether user 1 has agreed to sequential control in the operation screen user interface 3 (device operation device). For example, an "agree" button may be displayed on the operation screen, and if the "agree" button is pressed, it is determined that the sequential control has been agreed to.

[0323] If sequential control is approved, proceed to step S344.

[0324] If sequential control is not agreed upon, proceed to step S310.

[0325] In step S344, the operation screen user interface 3 (device operation device) sends control codes representing the sequential control of the target device 2 by the user 1 to the target device 2. Thus, the target device 2 is operated.

[0326] The processing of step S313 and Figure 15 The steps are the same as S313.

[0327] Additionally, in step S310, the device operating mechanism can also obtain sequential control information after user 1's consent as feedback information D36. When user 1 inputs a command, displaying the feedback information D36 on the display screen assists user 1 in inputting commands to the target device 2.

[0328] Next, another example of the operation of the control system 3000c in this modified example will be described. Figure 35 This is a flowchart illustrating another example of the operation of the control system 3000c.

[0329] Processing in step S310 and Figure 15 The steps are the same as in step S310.

[0330] In step S351, the learning model unit 300 generates an operation instruction D32 (operation screen) representing a candidate list of sequential control as the second information.

[0331] In step S52, the learning model unit 300 displays an operation screen on the operation screen user interface 3 (device operation device). The learning model unit 300 allows the user 1 to select sequentially from a candidate list via the operation screen.

[0332] In step S353, it is determined whether user 1 has selected sequential control from the candidate list in the operation screen user interface 3 (device operation device). For example, selection buttons may be displayed in each candidate in the candidate list, and if a selection button is pressed, it is determined that the corresponding sequential control has been selected. In addition, if the "Next" button is pressed without any selection, it can also be determined that user 1 does not agree to all sequential control in the candidate list.

[0333] If sequential control is selected, the process proceeds to step S354.

[0334] If not selected from the sequential control, the process returns to step S310.

[0335] In step S354, the operation screen user interface 3 (device operation device) sends control codes representing the sequential control of the target devices 2 selected by user 1 to the target devices 2. Thus, the target devices 2 are operated.

[0336] The processing of step S313 and Figure 15 The steps are the same as S313.

[0337] As described above, in this modified example, the learning model unit 300 can be used to generate operation screens that have undergone processing (such as screen API construction or display mode changes) so that the desired operation can be performed simply or easily. Therefore, the efficiency of operations related to the operation of the target device 2 can be further improved. In addition, according to this modified example, the user 1 can perform actual operations while checking the description of the operation instructions generated by the learning model unit 300, so that the operation can be performed without errors.

[0338] Furthermore, according to this modified example, user 1 can confirm whether the sequential control operation instructions generated by the learning model unit 300 are indeed feasible. Therefore, the state of the target device 2 can be made closer to the state desired by user 1.

[0339] For example, if various functions are configured on the remote control's screen, the screen becomes very cluttered. According to this variation, the remote control's operation screen can be simplified to the functions desired by the user.

[0340] Furthermore, according to this variation, when user 1 inputs the desired function via voice or text, the corresponding control menu or a candidate control menu can be displayed on the operation screen. Thus, user 1 can select the appropriate operation.

[0341] In addition, in this modified example, the highest probability candidate can be displayed in the largest position among the several candidates that appear on the operation screen, and then displayed in smaller positions in sequence, etc., to make it more suitable for user 1.

[0342] As a sequential control, it is preferable to display the control after combining multiple operations. Sequential control is, for example, the control after combining multiple operations such as "starting the machine and operating it at cooling / room temperature of 28°C after 2 hours, and then disconnecting the power after 5 hours" or "setting the fan to strong during the initial cooling operation, but switching to weak during dehumidification after 1 hour".

[0343] Furthermore, in this modified example, the operation screen output corresponding to the input information can also be supplemented as candidate operation commands based on past operation history, operation frequency, etc. Additionally, if the operation content is stored on the device operation device (remote controller), the supplementary operation commands can be displayed without querying the learning model unit about past operation history.

[0344] Variations 3-4.

[0345] Next, other variations of the control system 3000 will be described. In this variation, the learning model also uses environmental information to generate operating commands.

[0346] Figure 20 This is a structural diagram showing an example of a control system 3000d, which is a variation of the control system 3000 of this embodiment. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0347] Figure 20 The control system 3000d shown also has an environmental information storage unit 313 (referred to as environmental information DB in the figure).

[0348] The learning model unit 300 outputs an operation instruction D32 (second information) representing the sequential control corresponding to a specific environment, based on the environment information D33a representing the environment of the object device 2.

[0349] Environmental information D33a includes at least one of the following states as the environment: the state of object device 2, the state of the surroundings of object device 2, and the state of user 1 containing vital sign data of user 1. Environmental information D33a may also include all of the state of object device 2, the state of the surroundings of object device 2, and the state of user 1 containing vital sign data of user 1.

[0350] The following provides further explanation.

[0351] The environmental information storage unit 313 stores environmental information D33a, which is information about the environment at the operating destination of the object device 2. The environmental information D33a may also include information related to the space where the object device 2 operates. In this modified example, information related to objects or people existing in the space where the object device 2 operates, as well as user 1, the operator of the object device 2, are also included as part of the environment. Therefore, the environmental information D33a may also include information related to that object, person, or user 1.

[0352] Environmental information D33a may include, for example, information related to a person such as attributes, temperature, location, posture, and heart rate. Furthermore, environmental information D33a may also include information related to the space such as its location, temperature, humidity, and brightness. Moreover, environmental information D33a can retain information indicating its progression even when such space- or person-related information changes. Here, information indicating progression is also referred to as time-series data or historical information. Environmental information D33a may also be incorporated into the model reference information D104 of the learning model.

[0353] Environmental information D33a can also be obtained, for example, through sensors not shown.

[0354] The learning model unit 300 is configured, for example, to generate and output a model and its operating environment of operation instruction D32 based on input information D31, device information D33, environment information D33a and other referable information in the learning model unit 300 when input information D31 is input.

[0355] As described above, according to this variant, the learning model can also use environmental information D33a related to the space in which the object device 2 operates to generate operation instructions D32, thus further enabling highly functionalized operations related to the operation of the object device 2.

[0356] Variations 3-5.

[0357] Next, other variations of the control system 3000 will be described. In this variation, two learning models are combined to generate operating instructions. Figure 21This is a structural diagram showing an example of a control system 3000e, which is a variation of the control system 3000 of this embodiment. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0358] Figure 21 The control system 3000e shown includes a learning model unit 300a as a first learning model unit 300 and a learning model unit 300b as a second learning model unit 300, in place of... Figure 20 The learning model unit 300 is shown.

[0359] When input information D31 is input, the learning model unit 300a outputs operation information D320. The learning model unit 300a may also be configured to generate and output a model and its action environment that, when input information D31 is input, generates and outputs operation information D320 based at least on input information D31 and environment information D33a.

[0360] Here, the operation information D320 includes information related to the operation of the target device 2, and this information is represented in a prescribed form that can be determined by the subsequent learning model unit 330b. Here, the operation information D320 may also be information supplemented (including addition, correction, or cancellation) to the operation content shown by the input information D31 based on the environment information D33a. The operation information D320 may also be information showing changes in the operation content or its manifestation as represented by the input information D31, depending on the state of the space in which the target device 2 is driven. The learning model unit 300a may be a model primarily anchored to the input information D31.

[0361] For example, even if the expected operation is the same, it is believed that language performance will differ depending on the environment driven by the object device 2, and the way phenomena are recognized will also differ. For example, the operation represented by input information D31 may differ depending on dialect, wording habits, company language, family language, perceptual differences such as hot / cold, etc.

[0362] The learning model unit 300a, for example, functions by absorbing differences in language performance and / or differences in phenomenon recognition, and refining them into more generalized or specific content. The learning model unit 300a can also be a local learning model that obtains output results based on local information, such as when the destination's database is limited.

[0363] The operation information D320 generated by the learning model unit 300a is input into the learning model unit 300b.

[0364] The learning model unit 300b is basically the same as the learning model unit 300 described above. However, instead of the input information D31, the operation information D320 generated by the learning model unit 300a is input.

[0365] When operation information D320 is input, the learning model unit 300b outputs operation instruction D32. The learning model unit 300b may also be configured to generate and output operation instruction D32 based on operation information D320, device information D33, and other referable information within the learning model unit 300b when operation information D320 is input, along with its operational environment. The learning model unit 300b may also be a global learning model capable of freely accessing global information such as external networks to obtain output results.

[0366] Figure 22 This is a flowchart illustrating the action example of this variation. Figure 22 In the example shown, when the input interface 311 of the control system 3000e receives the input information D31 in step S310, the input information D31 is input to the learning model unit 300a.

[0367] Next, the control system 3000e performs a process for generating operation information D320 using the learning model unit 300a (step S331). In step S331, the learning model unit 300a (more specifically, the model control unit 101) outputs operation information D320 corresponding to the input information D31 based on the model information D102 and the input information D31, and, if necessary, based on the model reference information D104 including environmental information D33a. The operation information D320 output from the learning model unit 300a is input to the learning model unit 300b.

[0368] Next, the control system 3000e performs the process of generating the operation instruction D32 using the learning model unit 300b (step S332). In step S332, the learning model unit 300b (more specifically, the model control unit 101) generates and outputs the operation instruction D32 corresponding to the operation information D320 based on the model information D102 and the input operation information D320, and, if necessary, based on the model reference information D104 including the device information D33.

[0369] Subsequent processing can be the same as other control systems in this embodiment.

[0370] As described above, the learning model unit 300 includes a first learning model unit 300a and a second learning model unit 300b. The first learning model unit 300a inputs input information D31 (first information) into the first learning model and outputs operation information D320 representing sequential control corresponding to a specific environment based on environment information D33a. The second learning model unit 300a inputs the operation information D320 into the second learning model and outputs the control code of the target device 2 used to implement the operation information D320 as second information to the target device 2 via the output interface 312.

[0371] As described above, according to this variant, for the input information D31 input from user 1, operation instructions D32 can be generated based on the absorption of differences in language expression and / or differences in the recognition of phenomena to make the content more general or specific, thereby further realizing the high functionality of operations related to the operation of the object device 2.

[0372] Furthermore, using the structure shown in variations 3-4, the learning model unit 300 can also generate operation instructions D32 that average out differences in language performance and / or differences in phenomenon recognition based on environmental information D33a, device information D33, and model reference information D104 containing past operation history. However, according to this variation, the task of the learning model can be clearly divided into the absorption of performance differences and the conversion into operation instructions, thus enabling the learning model to learn specifically for the above tasks, resulting in a compact design that minimizes the scale of learning.

[0373] Variations 3-6.

[0374] Next, other variations of the control system 3000 will be described. Figure 36 This is a structural diagram showing an example of a control system 3000f, which is a variation of the control system 3000 of this embodiment. Furthermore, elements identical to those in the control system 3000 are labeled with the same reference numerals and their descriptions are omitted.

[0375] In the control system 3000f, in addition to the structure of the control system 3000, there is also a general operation device 4a and a display information 350.

[0376] The general-purpose operating device 4a is a universal operating device capable of operating both the target device 2 and other devices different from the target device 2. The general-purpose operating device 4a can operate multiple devices of different types used by user 1. For example, like all air conditioners or all kitchen appliances used by user 1, the general-purpose operating device 4a can also be an operating device capable of operating each category of equipment. Furthermore, it can also be an operating device for each area, capable of operating all devices installed in the living room, such as air conditioners, air purifiers, lighting, circulators, televisions, and audio equipment used by user 1.

[0377] Display format information 350 stores information about the display format, i.e., the display format shown in the general operating device 4a.

[0378] The learning model unit 300 uses display format information 350 to display the display indicating sequential control as the second information on the display screen of the general operation device 4a. The learning model unit 300 can also obtain the information in Modification 3-3 via the display screen of the general operation device 4a. Figure 19The description states that user 1 agrees to or selects sequential control to control object device 2.

[0379] The general-purpose operating device 4a in this modification can also have the same functions as the device operating device described in modifications 3-3. For example, the input interface 311 and the general-purpose operating device 4a can be the same device. As the general-purpose operating device 4a, for example, if it has a microphone and a communication unit for querying the learning model unit 300 (LLM / VLM), it can be configured as a remote control device.

[0380] As described above, according to this variation, the device can be operated using a device familiar to user 1, which further enables the efficiency of operations related to the operation of the target device.

[0381] Variations 3-7.

[0382] Next, other variations of the control system 3000 will be described. In this variation, the learning model also uses user history information to generate operating instructions.

[0383] Figure 37 This is a structural diagram showing an example of a control system 3000g, which is another variation of the control system 3000d in variations 3-4 of this embodiment. Furthermore, regarding the control system 3000d of variations 3-4 (… Figure 20 Identical elements are labeled with the same number and their descriptions are omitted.

[0384] Figure 37 The control system 3000g shown also has a user history information storage unit 360 (referred to as user history information DB in the figure).

[0385] The learning model unit 300 outputs operation instructions D32 (second information) representing the sequential control corresponding to user 1, based on user history information D361 containing the history of user 1's commands to object device 2.

[0386] User history information D361 is, for example, information obtained by correlating the commands input by user 1, the current state of user 1 and object device 2, the output control codes of object device 2, and changes in the environment.

[0387] The learning model unit 300 also outputs operation instructions D32 (second information) that represent sequential control corresponding to the states of user 1 and object device 2 based on user history information D361. That is, the learning model unit 300 outputs operation instructions D32 (second information) that represent sequential control corresponding to the state and preferences of user 1 and the state of object device 2 based on environmental information D33a and user history information D361.

[0388] The learning model unit 300 stores the user history information D361 in the user history information storage unit 360, for example, as follows.

[0389] (1) When input information D31 is input, the current status of user 1 and object device 2 is obtained according to environmental information D33a. The input information D31 is stored in the user history information storage unit 360 in correspondence with the current status of user 1 and object device 2.

[0390] (2) The operation instruction D32 output from the learning model in response to the input information D31 is stored in the user history information storage unit 360 in correspondence with the above-mentioned input information D31.

[0391] (3) The status of user 1 and object device 2 after operation via operation instruction D32 is obtained and stored in user history information storage unit 360 in accordance with the above-mentioned input information D31, etc. The status of user 1 and object device 2 after object device 2 is operated can also be obtained by sensors of object device 2, etc.

[0392] In addition, (1) to (3) above are examples of generating user history information D361. User history information D361 can also be generated by other methods.

[0393] Next, the operation of the control system 3000g in this embodiment will be explained. Figure 38 This is a flowchart illustrating an example of the operation of the control system 3000g.

[0394] The processing of steps S310 and S312~S313 Figure 15 The example shown is the same.

[0395] In step S361, the learning model unit 300 outputs operation instructions D32 (second information) based on environmental information D33a and user history information D361, which represent sequential control corresponding to the state and preferences of user 1 and the state and surrounding state of object device 2.

[0396] For example, when input information D31 is input, the learning model unit 300 generates and outputs an operation instruction D32 representing sequential control based on the input information D31, device information D33, environmental information D33a, and user history information D361. Thus, the operation instruction D32 becomes an appropriate instruction corresponding to the state of the target device 2 and its surroundings, as well as the state and preferences of the user 1.

[0397] Furthermore, the learning model unit 300 can also generate and output operation instructions D32 based on other referable information. For example, it can also refer to external information related to climate and weather.

[0398] As described above, according to this variation, the learning model can also use environmental information D33a and user history information D361 to generate operation instructions D32. Therefore, according to this variation, appropriate operation instructions D32 corresponding to the state of the object device 2 and its surroundings, as well as the state and preferences of the user 1, can be output. Thus, it is possible to further achieve high precision and high functionality in the operations involved in the operation of the object device 2.

[0399] Furthermore, according to this variation, the learning model can use environmental information and user history information to learn the user's preferences for each environment. Therefore, according to this variation, a more comfortable situation can be provided for user 1.

[0400] Variations 3-8.

[0401] Next, other variations of the control system 3000 will be described. In this variation, two learning models are combined to generate operating instructions. Figure 39 The control system 3000h is the control system 3000e of variations 3-5 and 3-7. Figure 21 ) and control system 3000g ( Figure 37 Variations of ) . Additionally, elements identical to those in control systems 3000e and 3000g are labeled with the same numbers and their descriptions are omitted.

[0402] In this modified example, the first learning model unit 300a inputs the input information D31 into the first learning model, and outputs the operation information D320, which represents the sequential control corresponding to the state of user 1 and object device 2, based on the environmental information D33a and the user history information D361.

[0403] The first learning model part 300b is the same as the variant example 3-5.

[0404] Figure 40 This is a flowchart illustrating the action example of this variation.

[0405] The processing of steps S310, S312~S313 and Figure 22 The example shown is the same.

[0406] In step S371, the control system 3000h performs a process to generate operation information D320 using the first learning model unit 300a. The first learning model unit 300a inputs input information D31 to the first learning model, generates and outputs operation information D320 corresponding to the input information D31. The first learning model is a model that outputs operation information D320 corresponding to the input information D31 based on environmental information D33a, user history information D361, and, if necessary, other external information. That is, the first learning model unit 300a is a model and its operating environment configured to output operation information D320 corresponding to the state of the object device 2 and its surroundings, and the state and preferences of the user 1. The operation information D320 output from the first learning model unit 300a is input to the second learning model unit 300b.

[0407] In step S372, the control system 3000h performs a process to generate operation instructions D32 using the second learning model unit 300b. The second learning model unit 300b inputs operation information D320 into the second learning model, generates and outputs operation instructions D32 corresponding to the operation information D320. The second learning model is a model that outputs operation instructions D32 corresponding to operation information D320 based on device information D33. That is, the second learning model unit 300b is a model and its operating environment configured to output operation instructions D32 such as control codes for the target device 2 based on device information D33. The operation instructions D32 output from the second learning model unit 300b are output to the output interface 312.

[0408] As described above, according to this variation, a first learning model and a second learning model can be defined. The first learning model is a user-specific model that considers the state of user 1 and object device 2, as well as their surrounding environment. The second learning model is a model that considers general information about object device 2. Here, the second learning model is generated based on general information about object device 2, and therefore can be installed, for example, at a factory of object device 2. On the other hand, the user-specific first learning model is a model that is updated according to the user's environment.

[0409] User-specific models can be implemented using compact models tailored to the user, making them easy to customize. On the other hand, device-wide models are universal for all users, thus enabling large-scale deployment.

[0410] According to this variation, since the user-specific model and the device-wide model are structured differently, it eliminates the need for manufacturers to prepare models for each user environment. Furthermore, because it is easy to update only the user-specific model, it provides a control system that gives users a sense of trust and personalization.

[0411] In the above embodiments 3 and variations 3-1 to 3-8, multiple parts can be combined for implementation. Alternatively, one part of these embodiments 3 and variations 3-1 to 3-8 can be implemented. Furthermore, these embodiments 3 and variations 3-1 to 3-8 can be implemented in any combination, either as a whole or in part.

[0412] That is, it is possible to freely combine the embodiments 3 and the variations 3-1 to 3-8, or to modify any of the constituent elements in the embodiments 3 and the variations 3-1 to 3-8, or to omit any of the constituent elements in the embodiments 3 and the variations 3-1 to 3-8.

[0413] Implementation method 4.

[0414] Next, this embodiment 4 will be described. In this embodiment, an example of using a learning model to assist in monitoring a certain work status will be described.

[0415] For example, consider monitoring anomalies in a factory's factory automation (FA) system, which includes control equipment such as robots and PLCs. While existing monitoring algorithms, such as rule-based algorithms, can handle obvious setup errors in the workpiece being controlled, we also consider situations where minor setup errors trigger anomalies in subsequent processes. In such cases, even if analysis is performed based on anomaly detection, it may be difficult to accurately grasp the situation and find improvement methods.

[0416] In this embodiment, by supporting the monitoring of work environments where the following situations are anticipated to occur, the monitoring operations are made more efficient and higher-performance. These situations refer to minor adverse conditions that escalate into major anomalies, situations that do not conform to existing rules, and situations where the cause is difficult to determine.

[0417] Figure 23 This is a structural diagram showing an example of the control system 4000 of Embodiment 4. Figure 23 The control system 4000 shown is a control system for monitoring specific operating conditions using a learning model. It includes a sensor 5, a learning model unit 400a, a learning model unit 400b, an equipment information storage unit 410 (denoted as equipment information DB in the figure), a model interface 6 (denoted as model IF in the figure), and a display 7.

[0418] Sensor 5 acquires data indicating the status of the operation being monitored. Hereinafter, the data acquired by sensor 5 will be referred to as sensor data. Sensor data may, for example, be image data obtained by capturing images of the operation being monitored. Furthermore, sensor data may, for example, be audio data obtained by recording audio of the operation being monitored. Additionally, sensor data may, for example, be measurement data obtained by measuring the position and other states of the person or object performing the operation being monitored.

[0419] Sensor 5 acquires sensor data on a continuous basis, but it can also be triggered, for example, by a human or other monitoring system. The sensor data acquired by sensor 5 is input as input information D41 to the learning model unit 400a. Alternatively, the sensor data itself, as input information D41, can be provided by a human or other monitoring system. In this case, sensor 5 can be omitted.

[0420] When input information D41 is input, the learning model unit 400a outputs the parsing result D42a. For example, when input information D41 is input, the learning model unit 400a outputs the parsing result D42a based on model information D102. The structure of the learning model unit 400a can be basically the same as that of the learning model unit 100 in Embodiment 1.

[0421] In this embodiment, the learning model unit 400a is a model and its operating environment configured to output a parsing result D42a corresponding to the input information D41 when input information D41 is input. Alternatively, the learning model unit 400a may be a model and its operating environment configured to generate and output the parsing result D42a based on the input information D41, device information D43, and other referable information (such as model reference information D104) within the learning model unit 400a when input information D41 is input. Here, as the model reference information D104, the learning model unit 400a may also refer to and utilize information related to the operation of the monitored object. Information related to the operation of the monitored object may, for example, include information indicating the location, person, object, sequence, and conditions of the operation. As the model reference information D104, the learning model unit 400a may also use information obtained by digitizing manual data containing information such as the conditions, setup environment, and operating procedures of the equipment used for the operation.

[0422] In this embodiment, the input information D41 includes information indicating the status of the work being monitored. Here, the work being monitored includes one or more work performed by a person or equipment. The input information D41 may, for example, be a measurement value, image, voice, or a combination thereof indicating the status of the work being monitored. The input information D41 may also, for example, be a measurement value, image, voice, or a combination thereof indicating the status of multiple work being monitored. Furthermore, the input information D41 may also include information indicating the status of work performed continuously in time. In this case, it may be time-series data containing a prescribed data structure including measurement values, images, voice, or a combination thereof indicating such status. The method of representing the work status is based on the input form of the model used by the learning model unit 400a, but is not limited to this if the learning model unit 400a includes error processing, correction processing, or conversion processing in its preceding stages.

[0423] The parsing result D42a contains information indicating the parsing result obtained by analyzing the work situation shown in the input information D41. The information representing the parsing result can also be information indicating the objects (environment) and / or phenomena that exist under the work situation shown in the input information D41. The information representing the parsing result can also be information explaining the work situation shown in the input information D41. The parsing result D42a can also be, for example, text explaining the work situation shown in the input information D41. Furthermore, the parsing result D42a can also be text focusing on the parts of the work situation shown in the input information D41 that differ from the normal situation and explaining those parts. Additionally, the form of the parsing result D42a can be other than text. The parsing result D42a can be described in a prescribed form that can be discerned by the subsequent learning model unit 400b; the format is not particularly limited, and can be, for example, text, images, speech, or a combination thereof.

[0424] As an example of interpreting a task situation, one could use the attributes of an object existing in that task situation to represent the object, use prescribed grammatical forms such as 5W1H or 7W1H to represent the phenomena occurring in that task situation, or further generalize based on such a concrete representation. In addition, examples could include: breaking down the task performed in that task situation into multiple viewpoints for interpretation and representing it from each viewpoint; or, if the task performed in that task situation includes multiple sub-tasks or steps, breaking down the task into sub-task units or step units and explaining it from each sub-task or step. The parsing result D42a can be said to be obtained by further adding a prescribed form of representation to the task situation represented by the input information D11 in the processes of concretization, subdivision, and / or singularity extraction. Thus, in the parsing result D42a, the task situation is represented in an easily understandable and organized state.

[0425] When the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b. For example, when the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b based on the model information D102. The structure of the learning model unit 400b can be basically the same as that of the learning model unit 100 in Embodiment 1.

[0426] In this embodiment, the learning model unit 400b is a model and its operating environment configured to output a parsing result D42b corresponding to the parsing result D42a when the parsing result D42a is input. Alternatively, the learning model unit 400b may also be a model and its operating environment configured to generate and output the parsing result D42b based on the parsing result D42a, device information D43, and / or information referential in the learning model unit (such as model reference information D104) when the parsing result D42a is input.

[0427] The analysis result D42b contains information representing methods for improving the work status, which are derived from the analysis results of the work status by the learning model unit 400a. This information can represent a recovery method to restore an abnormal state to normal, or it can represent a solution to a problem that occurs in an environment (work environment) where work is being monitored, such as when people are distressed or equipment stops.

[0428] Information representing the improvement method may include, for example, text, images, or audio illustrating the method, or control instructions (e.g., commands, control signals, control codes, etc.) for the device (object device 2) to which the method is implemented, a procedure description, timing diagram, source code, executable code, or controller commands used to cause the controller to execute the method. Information representing the improvement method may also include text, images, audio, data described in a specified design language, control descriptions (including source code and information described in a specified programming platform language), information described in other platform languages, control instructions (including control commands, control signals, control codes, and controller commands), executable code, and combinations of two or more of these elements. Examples of specified design languages ​​include, but are not limited to, UML (Unified Modeling Language).

[0429] Hereinafter, the learning model unit 400a is sometimes referred to as the first learning model unit 400, and the analysis result D42a is sometimes referred to as the first analysis result D42. In addition, the learning model unit 400b is sometimes referred to as the second learning model unit 400, and the analysis result D42b is sometimes referred to as the second analysis result D42.

[0430] As already explained, the parsing result D42a contains information representing the parsing result of the job status shown in the input information D41. Therefore, the learning model unit 400b can be a model and its operating environment configured to output a parsing result D42b corresponding to the parsing result shown in parsing result D42a. Here, if the information representing the parsing result is text explaining the job status represented by the input information D41, the learning model unit 400b can also be a model and its operating environment configured to output a parsing result D42b corresponding to the text explaining the job status.

[0431] The processing of equipment information storage unit 410 and equipment information D43 is basically the same as that of equipment information storage unit 110 and equipment information D13 in Embodiment 1. Furthermore, in this embodiment, equipment information storage unit 410 stores equipment information D43, which is information related to the equipment that is the target equipment 2 and the operation that is the object of monitoring. Here, the equipment related to the operation broadly includes the equipment required for the aforementioned situation analysis and the derivation of the improvement method. More specifically, it includes not only the equipment used for the operation, but also equipment that affects the person or equipment performing the operation. More specifically, equipment that affects the person or equipment performing the operation can also be equipment that directly or indirectly causes changes to the person or equipment performing the operation. Examples include equipment directly used for the operation (including various machines such as processing machines and conveying machines, as well as tools such as workbenches and tools), equipment that controls equipment directly used for the operation (power supplies, relays, switches, controllers, etc.), and equipment that changes the working environment (lighting equipment, air conditioning equipment, vacuum cleaners, cleaners, etc.).

[0432] For example, device information D43 is used as additional information when the learning model unit 400a and / or the learning model unit 400b outputs model output data D103 (parsed result D42a and parsed result D42b). Hereinafter, in this embodiment, information indicating the state of the object device 2 is sometimes specifically referred to as state information D45.

[0433] In this embodiment, the learning model unit 400a may also be an image learning model such as VLM (Visual Learning Model) and its operating environment, which takes an image as input and outputs a result. Furthermore, the learning model unit 400a may, for example, be a multimodal model and its operating environment, which takes natural language and an image as input and outputs a result. Similarly, the learning model unit 400b may, for example, be a language learning model such as LLM (Language Learning Model) and its operating environment, which takes natural language as input and outputs a result. In this case, the input information D41 may be input via text data, image data, a combination of text data and image data, or a data form that can be converted into them (speech data, animated images as a combination of speech data and image data, etc.). Furthermore, the learning model used in this embodiment is not limited to the models described above.

[0434] Model interface 6 is an interface that, upon receiving model output data (parsed results D42a and D42b) from learning model units 400a and 400b, outputs the model output data to a specified output destination. Model interface 6 may also be an interface that converts the model output data output from learning model units 400a and 400b into data conforming to a specified output destination and outputs it. Model interface 6 may also be provided as an example of the output unit 103 described above. In this embodiment, the object device 2 and the display 7 are used as the output destinations of model interface 6.

[0435] Model interface 6 can, for example, output the result information D44a, representing the situation analysis result contained in analysis result D42a and the improvement method contained in analysis result D42b, to display 7, and output the result information D44b, representing the improvement method contained in analysis result D42b, to object device 2. In this case, model interface 6 can also extract a portion of data from analysis result D42a and / or analysis result D42b, convert it into a data format suitable for the output destination, and output it as result information D44a and result information D44b.

[0436] In addition, Figure 23 In the example shown, the object device 2 and the display 7 are shown as the output destinations of the model interface 6, but the output destinations of the model output data are not limited to the above example. For example, if the improvement method for the state shown by the model output data as the output object includes information indicating control for the object device 2, the model interface 6 may, for example, output the model output data or information indicating the method directly to the object device 2, which is the implementation destination of the method, or output the model output data or information indicating the method to a conversion device (not shown) that converts the information into information that the object device 2 can accept. This conversion device may, for example, be the control system 1000 of Implementation 1, which converts the input information into control descriptions or execution codes that the object device 2 can determine.

[0437] Furthermore, the model interface 6 itself can also function as a conversion device. For example, the model interface 6 can not only control the output of model output data, but also convert the improvement method output by the learning model unit 400b into code that can be executed by the interpreter, output the converted code, or control the device based on the code. In addition, the model interface 6 can also control the processing flow, such as immediately executing high-urgency processing in the improvement method. Furthermore, the model interface 6 can also transmit prompts input via the display 7, such as responses to suggestions on the methods displayed on the display 7, to the learning model unit 400b.

[0438] Furthermore, the model interface 6 may also have the functions of the output confirmation unit 202 and the correction confirmation unit 203 described above. For example, the model interface 6 may determine the urgency of the parsed situation. If the urgency is determined to be low, it may inquire with the monitor or use a simulator to confirm the appropriateness of the improvement method. If the improvement method is inappropriate, it may transmit this information to the learning model unit 400b and urge it to output the improvement method again. In this case, the model interface 6 may also issue supplementary information D48 to the model input data of the learning model unit.

[0439] In this embodiment, input information D41 corresponds to model input data D101 of learning model unit 400a. Parsing result D42a corresponds to model output data D103 of learning model unit 400a. Parsing result D42a corresponds to model input data D101 of learning model unit 400b. Parsing result D42b corresponds to model output data D103 of learning model unit 400b. Learning model unit 400a (especially model control unit 101) may, for example, be configured to, upon receiving input information D41, output parsing result D42a corresponding to input information D41 based on model information D102 and, if necessary, based on model reference information D104. Furthermore, learning model unit 400b (especially model control unit 101) may, for example, be configured to, upon receiving parsing result D42a, output parsing result D42b corresponding to parsing result D42a based on model information D102 and, if necessary, based on model reference information D104.

[0440] Furthermore, in this case, the model generation unit 107, corresponding to the learning model unit 400a, can, for example, use model learning data D105 containing candidates of input information D41 that can be input to the model control unit 101 to perform machine learning and generate or update model information D102. Alternatively, it can use model learning data D105 containing candidates of input information D41 that can be input to the model control unit 101 and candidates of corresponding parsing results D42a to perform machine learning and generate or update model information D102. Similarly, the model generation unit 107, corresponding to the learning model unit 400b, can, for example, use model learning data D105 containing candidates of parsing results D42a that can be input to the model control unit 101 to perform machine learning and generate or update model information D102. Alternatively, it can use model learning data D105 containing candidates of parsing results D42a that can be input to the model control unit 101 and candidates of corresponding parsing results D42b to perform machine learning and generate or update model information D102.

[0441] Although figures are omitted, in this embodiment, status information D45 and / or feedback information D46 can be obtained from the model output data D103 of learning model units 400a and 400b and / or the output destination of information generated based thereon. The control system 4000 may, for example, output the obtained status information D45 and / or feedback information D46 as information representing control results to the user, learning model units 400a, 400b, or other devices not shown. Furthermore, the control system 4000 may be configured to return a query D47 to the user if the input information D41 contains ambiguous or uncertain information. Additionally, the control system 4000 may generate supplementary information D48 regarding the input and output data of learning model units 400a and 400b based on the obtained status information D45 and / or feedback information D46, and issue it to the user, learning model units 400a, 400b, or other devices not shown. The processing of status information D45, feedback information D46, inquiry D47, and supplementary information D48 can be basically the same as in embodiment 1. Here, the output of information to the user can be performed, for example, via the display 7 or the input / output interface provided by the information processing device 10 (not shown).

[0442] Furthermore, the control system 4000 may also include a status acquisition unit 430 (not shown), which acquires status information D45 and / or feedback information D46, and issues supplementary information D48 as needed. The status acquisition unit 430 is the same as the status acquisition unit 130 in Embodiment 1.

[0443] In this embodiment, the target device 2 is not particularly limited. Furthermore, it is assumed that the target device 2 is a device that can actually be controlled by receiving the parsing result D42b, but it is not limited to this if the above-described conversion device is included between the target device 2 and the target device 2.

[0444] In this embodiment, the input information D41 received by the control system 4000 can be referred to as information related to the conditions in the work environment (here, the conditions in the environment where the monitoring operation is performed). Therefore, the input information D41 received by the control system 4000 can be referred to as an example of first information representing the conditions in the work environment. Furthermore, the parsing results D42a and D42b can be referred to as information used for the operation (monitoring operation) corresponding to such input information D41. Hereinafter, the parsing results D42a and / or D42b output from the action environment of the learning model that is input based on the model input data of the input information D41 to the specified output destination will sometimes be referred to as second information.

[0445] Next, the operation of the control system 4000 in this embodiment will be explained. Figure 24This is a flowchart illustrating an example of the operation of the control system 4000.

[0446] exist Figure 24 In the example shown, firstly, the control system 4000 accepts the input information D41 (step S410). For example, the input unit 102 or the input processing unit 201 described above can also accept the input information D41. The accepted input information D41 is input to the learning model unit 400a as model input data D101.

[0447] Next, the control system 4000 performs a process to generate the parsing result D42a using the learning model unit 400a (step S411). In step S411, the learning model unit 400a (more specifically, the model control unit 101) outputs the parsing result D42a corresponding to the input information D41 based on the model information D102 and the input information D41, and, if necessary, based on the model reference information D104 including the device information D43. For example, the learning model unit 400a may use a learning model capable of generating text data to generate the parsing result D42a of text data based on the input information D41.

[0448] In step S411, the above processing may also be performed by the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 400a.

[0449] The parsing result D42a output from the learning model unit 400a is input to the learning model unit 400b. Furthermore, the parsing result D42a output from the learning model unit 400a is input to both the learning model unit 400b and the model interface 6. The parsing result D42a output from the learning model unit 400a can also be input to the model interface 6 via the learning model unit 400b. In this case, the learning model unit 400b can output model output data D103 containing both the parsing result D42a and the parsing result D42b.

[0450] Next, the control system 4000 performs a process of generating the parsing result D42b using the learning model unit 400b (step S412). In step S412, the learning model unit 400b (more specifically, the model control unit 101) outputs the parsing result D42b corresponding to the parsing result D42a based on the model information D102 and the input parsing result D42a, and, if necessary, based on the model reference information D104 including the device information D43. For example, the learning model unit 400b may also use a learning model capable of generating text data to generate the parsing result D42b of binary data based on the input parsing result D42a. Furthermore, the learning model unit 400b may also use a learning model capable of generating both text data and binary data to generate the parsing result D42b of both text data and binary data based on the input parsing result D42a.

[0451] In step S412, the above processing may also be performed by the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 400b.

[0452] The parsing result D42b output from the learning model section 400b is, for example, input into the model interface 6.

[0453] Model interface 6 controls object device 2 and / or displays information on display 7 based on the analysis results of learning model unit 400a and learning model unit 400b (step S413). In step S413, for example, model interface 6 outputs information based on analysis results D42a and D42b to a predetermined output destination. For example, based on analysis results D42a and D42b, model interface 6 outputs result information D44a representing the situation analysis result and improvement method to display 7, and outputs result information D44b representing the improvement method based on analysis result D42b to object device 2.

[0454] Result information D44a can also be expressed, for example, through text and voice, indicating the situation occurring in the work environment and improvement methods. Furthermore, result information D44b can also be expressed, for example, through text or control signals, indicating improvement methods.

[0455] Therefore, based on the result information D44a, the display 7 displays the status analysis result represented by the analysis result D42a and the improvement method represented by the analysis result D42b, and the object device 2 implements the improvement method represented by the analysis result D42b based on the result information D44b. Information input to the display 7 and the object device 2 can be directly input from the control system 4000 (more specifically, the model interface 6), or indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or manually.

[0456] The control system 4000 can also acquire status information D45 and feedback information D46 (step S414) when the state of the target device 2 changes due to control or other reasons, or when feedback is received from the output destination. In addition, the processing in step S414 is not necessary and can be omitted appropriately.

[0457] The control system 4000 may also repeat the processing of steps S410 to S414 multiple times (e.g., until the desired state is achieved in the object's operating environment).

[0458] As described above, according to this embodiment, different learning models are used to acquire the status and improvement methods in two stages, thereby improving the accuracy of the final product. As a result, it is possible to achieve high efficiency in the work involved in monitoring the work status.

[0459] For example, in scenarios involving situational awareness, it is important to broadly detect abnormal states in the work environment, such as "something has happened." On the other hand, in scenarios involving the development of improvement methods, specific information such as "stop the machine, move the workpiece to location A, and restart the machine after returning it to state B" becomes important.

[0460] When the level of abstraction of the information to be extracted (i.e., the target information) varies, there is a concern that attempting to learn / extract all of it using a single learning model might lead to a decrease in the accuracy of the output. In particular, improving the method requires specific approaches based on knowledge and information about the operating environment. In such cases, dividing the learning model and assigning appropriate domain knowledge (environmental information) can more reliably improve output accuracy.

[0461] Furthermore, the illusion problem becomes significant when a single learning model is used to obtain solutions for different tasks, such as situation mastery and the acquisition of improvement methods. This is because the function of adjusting the solution of one task (situation mastery) to make the solution of another task (acquiring improvement methods) appear reasonable may inherently function in the model algorithm. According to this embodiment, it also works for such illusion problems. That is, by dividing the learning model corresponding to the two tasks of situation mastery and the acquisition of improvement methods, modalities entering each learning model can be suppressed. As a result, the magnitude of illusions can be suppressed, thus improving the accuracy of the final product.

[0462] Furthermore, in this embodiment, the output results of the learning model unit 400a and the learning model unit 400b, namely the analysis results D42a and D42b, can be verbally displayed on the display 7. Therefore, by having a person confirm its content, hallucinations can be suppressed, thereby more reliably implementing the method for improving the situation.

[0463] Furthermore, the control system 4000 of this embodiment can be applied not only to the monitoring of the control system of the equipment in the factory, but also, for example, to the monitoring of logistics objects in a logistics system.

[0464] Variation Example 4-1.

[0465] Next, a variation of the control system 4000 will be described. Figure 25 This is a structural diagram showing an example of a control system 4000a, which is a variation of the control system 4000 of this embodiment. Furthermore, elements identical to those in the control system 4000 are labeled with the same reference numerals and their descriptions are omitted.

[0466] exist Figure 25 The control system 4000a shown differs from the control system 4000 in that it has two analysis units that analyze and improve the situation in different ways, and the analysis unit used is switched appropriately according to the situation that occurs.

[0467] Figure 25 The control system 4000a shown includes a first analysis unit 41-1 which uses the learning model unit 400a and the learning model unit 400b described above to obtain the part of the situation analysis and improvement method, and also has a second analysis unit 41-2, a switching unit 42 and an output switching switch 43.

[0468] The second analysis unit 41-2 is not particularly limited; it can be any unit that analyzes the input information D41 using a different method than the first analysis unit 41-1 to obtain a status quo and an improvement method. For example, the second analysis unit 41-2 could also be a unit that analyzes the status quo and obtains an improvement method based on rules. For instance, when the input information D41 is input, the second analysis unit 41-2 could determine whether the input information D41 conforms to a predetermined abnormality pattern, and if it conforms to any abnormality pattern, obtain an improvement method corresponding to that abnormality pattern. The second analysis unit 41-2 outputs an analysis result D42c that includes at least the status improvement method.

[0469] The analysis result D42c may, for example, include information equivalent to the result information D44a and information equivalent to the result information D44b described above. In this modified example, the analysis result D42c at least includes the result information D44b representing the improved method obtained by the second analysis unit 41-2.

[0470] In this modified example, the second analysis unit 41-2 may also be implemented as an internal execution module, for example, by a PLC, information processing device, or the like installed in the operating environment.

[0471] The switching unit 42 is a unit that switches the control destination for input information D41 according to predetermined conditions. In this modified example, the switching unit 42 switches the control destination for input information D41 between the first parsing unit 41-1 and the second parsing unit 41-2. For example, the switching unit 42 may also switch the control destination for input information D41 based on whether input information D41 conforms to existing rules. In this case, the switching unit 42 may switch the output destination of input information D41 to the second parsing unit 41-2 if input information D41 conforms to existing rules, and switch the output destination of input information D41 to the first parsing unit 41-1 if it does not conform to existing rules, thereby switching the control destination.

[0472] The switching unit 42 can, for example, switch the control destination for input information D41 according to instructions from the monitor. Furthermore, the switching unit 42 can, for example, switch the control destination for input information D41 based on time, work content, or the presence or absence of a monitor. Additionally, the switching unit 42 can, for example, switch the control destination for input information D41 based on whether an anomaly has occurred in the work environment. Here, whether an anomaly has occurred in the work environment can be determined, for example, by whether an anomaly signal has been generated. For example, the switching unit 42 can also switch the control destination for input information D41 to the first analysis unit 41-1 in the event of an anomaly. Furthermore, the switching unit 42 can, for example, switch the control destination for input information D41 based on the degree or urgency of the anomaly occurring in the work environment.

[0473] In addition, the switching unit 42 may also control the output switching switch 43 in conjunction with the switching of the control destination for the input information D41. The output switching switch 43 switches the connection path (circuit or communication path, etc.) that connects the output of the first parsing unit 41-1 or the output of the second parsing unit 41-2 to the object device 2 and the display 7, which are the output destinations of the parsing results.

[0474] For example, when the control destination for input information D41 is switched to the first parsing unit 41-1, the switching unit 42 may control the output switching switch 43 to connect the output of the first parsing unit 41-1 to the target device 2 and the display 7, and disconnect the output of the second parsing unit 41-2 from the target device 2 and the display 7. Similarly, when the control destination for input information D41 is switched to the second parsing unit 41-2, the switching unit 42 may control the output switching switch 43 to connect the output of the second parsing unit 41-2 to the target device 2 and the display 7, and disconnect the output of the first parsing unit 41-1 from the target device 2 and the display 7.

[0475] Figure 26This is a flowchart illustrating the action example of this variation. Figure 26 In the example shown, when the control system 4000a receives input information D41 in step S410, the switching unit 42 switches the control destination for input information D41 according to predetermined conditions (step S421). Figure 26 In the example shown, the switching unit 42 determines whether the input information D41 conforms to existing rules. If it determines that the input information does not conform (No in step S421), it proceeds to the first parsing process (step S422). On the other hand, if it determines that the input information D41 conforms to existing rules (Yes in step S421), it proceeds to the second parsing process (step S423).

[0476] In the first analysis process of step S422, the learning model unit 400a and learning model unit 400b, which are the first analysis units 41-1, analyze the situation and obtain an improvement method. The learning model unit 400a and learning model unit 400b output an analysis result D42a containing the situation analysis result and an analysis result D42b containing the situation improvement method as the result of the first analysis process.

[0477] In the second analysis process of step S423, the second analysis unit 41-2 analyzes the situation and obtains the improvement method according to existing rules. For example, the second analysis unit 41-2 outputs an analysis result D42c, which includes at least the situation improvement method, as the result of the first analysis process.

[0478] When the result of the parsing process of the first parsing unit 41-1 or the second parsing unit 41-2 is output, the target device 2 is controlled and / or the display 7 is made to display information based on the result of any parsing process, according to the state of the output switching switch 43 (step S424).

[0479] In this example, when the first parsing unit 41-1 performs parsing processing, the connection path connecting the output of the first parsing unit 41-1 to the object device 2 and the display 7 is established. In this case, the model interface 6 may, for example, output result information D44a representing the state parsing result and the improvement method to the display 7 based on the parsing result D42a and the parsing result D42b, and output result information D44b representing the improvement method based on the parsing result D42b to the object device 2. On the other hand, when the second parsing unit 41-2 performs parsing processing, the connection path connecting the output of the second parsing unit 41-2 to the object device 2 and the display 7 is established. In this case, the result information D44a representing the state parsing result and the improvement method may also be output to the display 7 based on the parsing result D42c output from the second parsing unit 41-2, and / or the result information D44b representing the improvement method may be output to the object device 2.

[0480] On display 7, for example, the result information D44b can be displayed in a way that the operator can confirm. In this case, the operator can also refer to the result information D44b displayed on display 7, confirm the improvement method shown in result information D44b, and implement the work involved in the method. In addition, the operator can also confirm the improvement method shown in result information D44b and determine its appropriateness. At this time, if the improvement method shown in result information D44b is inappropriate, the operator can also prompt the learning model unit 400b to obtain other improvement methods (re-acquiring the model output data). For example, when receiving a message indicating a request to re-acquire the model output data, the learning model unit 400b can also re-acquire the model output data based on changing a part of the input information, a part of the model parameters, or the reference destination of the reference information.

[0481] Subsequent processing can be the same as other control systems in this embodiment.

[0482] As described above, in this variation, multiple analysis units are configured to analyze the situation and acquire improvement methods using different approaches, and these units are switched according to the situation. Therefore, more situation-appropriate control is possible. For example, for problems with clear causes, the second analysis unit, which has a high processing load, immediately analyzes the situation and suggests / executes improvement methods; for problems with unclear causes, the first analysis unit, which uses a learning model, analyzes the complex situation and suggests / executes better improvement methods.

[0483] Furthermore, the above example illustrates how the first analysis unit 41-1 uses two learning models to analyze the situation and obtain an improvement method, but the structure of the first analysis unit 41-1 is not limited to the above example. For example, if analyzing the situation is not required, the learning model unit 400a can be omitted. Furthermore, if obtaining an improvement method is not required, the learning model unit 400b can also be omitted. Alternatively, it is also possible to analyze the situation and obtain an improvement method using only one learning model unit.

[0484] For example, if the first analysis unit 41-1 includes a learning model unit 400b that acquires a method for improving the situation based on input information D41, the switching unit 42 can also switch the control target for input information D41 to the first analysis unit 41-1 in case of an anomaly. In this case, the learning model unit 400b of the first analysis unit 41-1 is configured to output information indicating the improvement method corresponding to the anomaly situation indicated by input information D41 when input information D41 is input. At this time, the learning model unit 400b can also refer to the device information storage unit 410 accessible by the control system and output information indicating the improvement method corresponding to the situation.

[0485] Implementation method 5.

[0486] Next, this fifth embodiment will be described. In this embodiment, the following example will be described: using a learning model to assist the response operation in a call center, product site, etc., to information sent from a user. Here, the information sent from the user may include inquiries or opinions related to a certain service, information, phenomenon, or object.

[0487] Figure 27 This is a structural diagram showing an example of the control system 5000 of Embodiment 5. Figure 27 The control system 5000 shown includes a learning model unit 500, a reference information storage unit 12, a database retrieval unit 511 (denoted as DB retrieval unit in the figure), a control generation unit 512, a speech recognition unit 513v, and a speech synthesis unit 514v. Here, the reference information storage unit 12, the database retrieval unit 511, and the control generation unit 512 may also be provided as part of the learning model unit 500.

[0488] When input information D51 is input, the learning model unit 500 outputs response information D52 representing the response content. For example, when input information D51 is input, the learning model unit 500 outputs response information D52 based on model information D102. The structure of the learning model unit 500 can be basically the same as that of the learning model unit 100 in Embodiment 1.

[0489] In this embodiment, the learning model unit 500 is a model and its operating environment configured to output response information D52 corresponding to the input information D51 when input information D51 is input. Alternatively, the learning model unit 500 may also be a model and its operating environment configured to generate and output response information D52 based on the input information D51 and other referable information in the learning model unit 500 when input information D51 is input.

[0490] In this embodiment, the input information D51 includes information representing content sent from user 1, etc. The input information D51 may also include information representing content requesting a response in the working environment. For example, the input information D51 may be text, images, voice, or a combination thereof representing a query or opinion related to a certain service, information, phenomenon, or object. The input information D51 may also be text, images, voice, or a combination thereof representing multiple queries or opinions related to a certain service, information, phenomenon, or object. Furthermore, the input information D51 may also include information representing content sent consecutively in time; in this case, it may be time-series data containing a prescribed data structure including text, images, voice, or a combination thereof representing such content. The method of representing the content is based on the input form of the model used by the learning model unit 500, but is not limited to this if the learning model unit 500 has error handling, correction processing, or conversion processing at its front end.

[0491] Response information D52 contains information indicating a response to the content sent in input information D51. Response information D52 may also be, for example, information indicating a response to an inquiry or opinion related to a service, information, phenomenon, or object expressed in the content sent in input information D51.

[0492] The reference information storage unit 12 stores model reference information D104 referenced by the model control unit 101 of the learning model unit 500 for outputting response information D52. Model reference information D104 may include, for example, information related to a service, information, event, or object that may be included in the input information D51. Here, the reference information storage unit 12 may also specifically store information related to a particular service, information, phenomenon, or object as model reference information D104. Model reference information D104 may also include, for example, information obtained by digitizing the response manual. Furthermore, model reference information D104 may also include, for example, the history of previously input input information D51 or the transmission content contained therein. In this case, the reference information storage unit 12 may also store, together with information about the user 1 who is the transmission source (e.g., user identifier, user attribute information, etc.), the historical information representing previously input input information D51 or the transmission content contained therein as model reference information D104. Hereinafter, in this embodiment, information representing the state of the user 1 who is the transmission source is sometimes specifically referred to as state information D55.

[0493] The database retrieval unit 511 is a retrieval engine that references the information storage unit 12 and other databases. Based on requests from the model control unit 101 of the learning model unit 500, the database retrieval unit 511 accesses and outputs retrieval results to databases accessible to it. At this time, the database retrieval unit 511 can also restrict access to the target database.

[0494] The control generation unit 512 is an interface used to set the preconditions for the learning model unit 500 (especially the model control unit 101) to generate model output data. The control generation unit 512 may be, for example, an interface used by the learning model unit 500 to identify information as a control object and / or set output preferences. Here, control object information refers to information representing the focus object of control in the model control unit 101. The model control unit 101 may also be configured to generate model output data D103 based on the control object information represented by the control generation unit 512, according to the model input data D101. The control generation unit 512 may also identify a portion of the model input data input by the user as control object information, identify information generated by the model control unit 101 as control object information, or identify information generated by the model control unit 101 and corrected by other control units as control object information. The control object information and / or output preference setting may be specified by the user, by an external processing unit, or by the control generation unit 512 according to a predetermined algorithm.

[0495] When the input from user 1 includes speech-form input information D51v, the speech recognition unit 513v recognizes the speech represented by the input information D51v, converts it into a data form that conforms to the learning model unit 500, and outputs it. The speech recognition unit 513v may also, for example, convert the speech-form input information D51v into text-form input information D51.

[0496] The speech synthesis unit 514v converts the content shown in the response information D52 into speech form and outputs it. For example, if the response information D52, as output from the learning model unit 500, contains data in a form other than speech, the speech synthesis unit 514v converts that portion of the response information D52 into speech form and outputs it. For example, if the response information D52 is a data structure including a specification of a data form, the speech synthesis unit 514v may also convert data elements specified in that specification into speech form and output it. The speech synthesis unit 514v may also, for example, convert the text-based response information D52 into speech-based response information D52v.

[0497] Furthermore, in the example above, an example of using speech-form data in the input and output with user 1 was shown, but the data form used for input and output with user 1 is not limited to speech form. In this case, instead of the speech recognition unit 513v and the speech synthesis unit 514v, a processing unit that converts the data form used for input from user 1 into the data form used for input to the learning model unit 500, and a processing unit that converts the data form used for output from the learning model unit 500 into the data form used for input to user 1 can be provided.

[0498] Furthermore, if the learning model unit 500 can accept the data format used for input from user 1, the speech recognition unit 513v can be omitted. Furthermore, if user 1 can accept the data format used for output from the learning model unit 500, the speech synthesis unit 514v can be omitted.

[0499] In this embodiment, the input information D51 corresponds to the model input data D101. Furthermore, the response information D52 corresponds to the model output data D103. The learning model unit 500 (particularly the model control unit 101) may, for example, be configured to, upon receiving the input information D51, output the response information D52 corresponding to the input information D51 based on the model information D102 and, if necessary, based on the model reference information D104.

[0500] Furthermore, in this case, the model generation unit 107, corresponding to the learning model unit 500, can, for example, use the model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D51 that can be input to the model control unit 101. Additionally, the model generation unit 107 can, for example, use the model learning data D105 to perform machine learning and generate or update model information D102, wherein the model learning data D105 includes candidates for input information D51 that can be input to the model control unit 101 and candidates for corresponding response information D52.

[0501] Although figures are omitted, in this embodiment, status information D55 and / or feedback information D56 can also be obtained from the model output data D103 of the learning model unit 500 and / or the output destination of the information generated based thereon. For example, the control system 5000 can output the obtained status information D55 and / or feedback information D56 as information indicating the response result to a designated supervisor, the learning model unit 500, or other devices not shown. Furthermore, the control system 5000 can also be configured to return an inquiry D57 to the user 1 if the input information D51 contains ambiguous or uncertain information. Additionally, the control system 5000 can also generate supplementary information D58 regarding the input and output data of the learning model unit 500 based on the obtained status information D55 and / or feedback information D56, and issue it to the user 1, the designated supervisor, the learning model unit 500, or other devices not shown. The processing of status information D55, feedback information D56, inquiry D57, and supplementary information D58 can be basically the same as in Embodiment 1.

[0502] Furthermore, the control system 5000 may also include a status acquisition unit 530 (not shown), which acquires status information D55 and / or feedback information D56, and issues supplementary information D58 as needed. The status acquisition unit 530 is the same as the status acquisition unit 130 in Embodiment 1.

[0503] In this embodiment, the input information D51 received by the control system 5000 can be referred to as information related to a request in the work environment (here, the content of a message requesting a response in an environment that responds to an inquiry). Therefore, the input information D51 received by the control system 5000 can be referred to as an example of first information representing a request in the work environment. Furthermore, the response information D52 can be said to be information used for this operation (response operation) corresponding to such input information D51. Hereinafter, the response information D52 output from the action environment of the learning model, which is input with model input data based on input information D51, to a predetermined output destination will sometimes be referred to as second information.

[0504] Next, the operation of the control system 5000 in this embodiment will be explained. Figure 28 This is a flowchart illustrating an example of the operation of the control system 5000.

[0505] exist Figure 28 In the example shown, firstly, the control system 5000 accepts the input information D51v (step S510). For example, the input unit 102 or the input processing unit 201 described above can also accept the input information D51v. The accepted input information D51v is then input to the speech recognition unit 513v.

[0506] When the speech recognition unit 513v receives the input information D51v, it recognizes the speech contained in the input information D51v and converts it into input information D51 in the form of data that conforms to the input of the learning model unit 500 (step S511). The converted input information D51 is input into the learning model unit 500 as model input data D101.

[0507] Alternatively, if the speech recognition unit 513v is omitted, the received input information D51v can be input into the learning model unit 500 as model input data D101.

[0508] Next, the control system 5000 performs a process of generating response information D52 using the learning model unit 500 (step S512). In step S512, the learning model unit 500 (more specifically, the model control unit 101) outputs response information D52 corresponding to the input information D51 based on the model information D102 and the input information D51, and if necessary based on the model reference information D104.

[0509] In step S512, the above processing may also be performed by the preprocessing unit 105 and / or postprocessing unit 106 of the learning model unit 500.

[0510] The response information D52 output from the learning model unit 500 is input to the speech synthesis unit 514v, for example (step S513). The input of the response information D52 to the speech synthesis unit 514v can be directly input from the control system 5000 (more specifically, the learning model unit 500 or the information processing device 10, which serves as its operating environment), or indirectly input via a communication network or other devices (servers, various conversion devices, etc.) or by hand.

[0511] Next, the speech synthesis unit 514v converts the input response information D52 into speech-based response information D52v and outputs it (step S514). The speech synthesis unit 514v may also generate response information D52v by synthesizing speech that speaks the response content shown in the data form of response information D52, other than speech. Response information D52v is then output to user 1, who is the source of the input information D51v (step S515).

[0512] Furthermore, even without the speech synthesis unit 514v, the response information D52 output from the learning model unit 500 can also be output to the user 1, which is the source of the input information D51v.

[0513] As described above, in this embodiment, even without an operator or a site that has pre-embedded the content to be replied to, the learning model unit 500 can dynamically generate the reply information D52 and reply to the user who is the sending source when the information is sent from user 1. Therefore, the reply operation can be made more efficient and performant.

[0514] Variation 5-1.

[0515] Next, a variation of the control system 5000 will be described. Figure 29 This is a structural diagram showing an example of a control system 5000a, which is a variation of the control system 5000 of this embodiment. Furthermore, elements identical to those in the control system 5000 are labeled with the same reference numerals and their descriptions are omitted.

[0516] exist Figure 29 The control system 5000a shown here has a correctness judgment unit 515, which is different from the control system 5000.

[0517] The error judgment unit 515 determines whether the content shown in the response information D52, which is the output from the learning model unit 500, is correct. For example, the error judgment unit 515 may output the response information D52 to the user 1 or update the content of the reference information storage unit 12 only if it determines that the content shown in the response information D52 is correct.

[0518] Furthermore, the error judgment unit 515 may, for example, cause the learning model unit 500 to obtain other response information D52 (reacquiring the model output data) if it determines that the content shown in the response information D52 is incorrect. The error judgment unit 515 may also be provided as an example of the post-processing unit 106 described above.

[0519] In other respects, it can be the same as other control systems in this embodiment.

[0520] As described above, according to this modified example, it is determined whether the content shown in the response information output from the learning model unit 500 is correct. Based on the result, the presence or absence of output to the user, the reacquisition of the response information, and the updating of the reference information are performed, thereby further realizing the high performance of the response operation.

[0521] Variation 5-2.

[0522] Next, a second variation of the control system 5000 will be described. Figure 30 This is a structural diagram showing an example of a modified control system 5000b, which is a variation of the control system 5000 of this embodiment. Furthermore, elements identical to those in control system 5000 and control system 5000a are labeled with the same reference numerals and their descriptions are omitted.

[0523] like Figure 30 As shown, the control system 5000b may also include an emotion determination unit 516.

[0524] The emotion determination unit 516 uses the input information D51 and other information to determine the emotion of the user 1, who is the sender. In addition, the emotion determination unit 516 can also determine the emotion of the user 1 after the response information D52 from the learning model unit 500 is output to the user 1.

[0525] The emotion of user 1 determined by the emotion determination unit 516 can be input into the learning model unit 500 as the state information D55 contained in the model reference information D104, or it can be stored in the reference information storage unit 12 as a historical record along with the input and output data of the model.

[0526] As a method for recording in the reference information storage unit 12, the control system 5000b may also include a registration determination unit 518, which determines whether to allow the reference information storage unit 12 to record based on the emotion determination result of the emotion determination unit 516 on the user 1's emotion.

[0527] For example, if the determined emotion of User 1 is positive, then as a good example, the registration and determination unit 518 instructs the reference information storage unit 12 to record the model's input and output data as historical information. In this case, if there is an emotion determination result for User 1 before the learning model unit 500 outputs response information D52, the registration and determination unit 518 can also instruct the reference information storage unit 12 to record the model's input and output data, including the emotion information before and after the response, as historical information.

[0528] Furthermore, for example, if user 1's emotion is determined to be negative, then as a negative example, the registration and determination unit 518 can instruct the reference information storage unit 12 to record the model's input and output data as historical information. In this case, if there is an emotion determination result for user 1 before the learning model unit 500 outputs response information D52, the registration and determination unit 518 can also instruct the reference information storage unit 12 to record the model's input and output data, including emotion information before and after the response, as historical information.

[0529] In addition, the control system 5000b may also include an additional learning unit 519, which, when updating the content of the reference information storage unit 12, reconstructs (additionally learns) the model reference information D104 stored in the reference information storage unit 12 and other information referenced by the model control unit 101 based on the updated information.

[0530] Furthermore, the control system 5000b can replace the emotion determination unit 516 or, in addition to the emotion determination unit 516, also have an evaluation acquisition unit 517.

[0531] The evaluation acquisition unit 517 requests the evaluation of the response information D52 from user 1, and obtains the evaluation information D59 as its answer. The evaluation information D59 can be used, for example, in the same way as the emotion of user 1 mentioned above, for updating the information referenced by the model, supplementary learning, etc.

[0532] In addition, the control system 5000b may also include a control decision unit 520.

[0533] The control decision unit 520, based on the speech recognition results, emotion determination results, and / or evaluation results of the response information D52 from the input information from user 1, as well as instructions from the operator (not shown), specifies the control object information and / or sets the output tendency for the control generation unit 512. Here, the speech recognition results from the input information from user 1 may include information such as user 1's attributes, emotions, region, language, past usage history, and usage frequency. Furthermore, the control decision unit 520 may also set the synthesized speech for the speech synthesis unit 514v based on the speech recognition results, emotion determination results, and / or evaluation results of the response information D52 from the input information from user 1, as well as instructions from the operator (not shown).

[0534] For example, as an example of setting the output preference, the control decision unit 520 can specify the difficulty level of the explanation in the response, the speaking style (tone, pitch), language, grammatical level, politeness level, speaker's stance, and discourse direction. It can also specify the gender, tone, and pitch of the synthesized speech. Furthermore, as an example of setting the synthesized speech, the control decision unit 520 can specify the gender, speaking style, language, grammatical level, and politeness level of the synthesized speech. The control decision unit 520 can also perform these settings based on pre-determined setting rules, for example.

[0535] in addition, Figure 30 The elements of the control system 5000b shown can be appropriately selected according to the desired function.

[0536] In other respects, it can be the same as other control systems in this embodiment.

[0537] As described above, according to this modified example, the control decision unit 520 specifies the controlled object information and / or the output preference setting based on information obtainable from the control system 5000b, thus enabling the generation of response information that easily meets the request of the sending source. Therefore, it is possible to further achieve high-performance response operations to users.

[0538] Variation 5-3.

[0539] Next, a third variation of the control system 5000 will be described. Figure 31 This is a structural diagram showing an example of a modified control system 5000c, which is a variation of the control system 5000 of this embodiment. Furthermore, elements identical to those in control systems 5000, 5000a, and 5000b are labeled with the same reference numerals and their descriptions are omitted.

[0540] like Figure 31 As shown, the control system 5000c may also include an image resolution unit 513i, an image generation unit 514i, and a program generation unit 514p.

[0541] When the image parsing unit 513i receives input information D51i in image form from user 1, it parses the image represented by the input information D51i, converts it into a data form that conforms to the learning model unit 500, and outputs it. For example, the image parsing unit 513i can also convert the image-form input information D51i into text-form input information D51.

[0542] Alternatively, for example, if the input from user 1 includes an image capturing the operation screen of a product held by user 1, the image parsing unit 513i parses the image, determines which product's operation screen it is, and what operation state it is in, and converts it into explanatory text for output. Furthermore, for example, if the input from user 1 includes an image capturing a shopping website that user 1 is browsing, the image parsing unit 513i parses the image, determines which website's operation screen it is, and what operation state it is in, and converts it into explanatory text for output.

[0543] The image generation unit 514i generates and outputs an image based on the response information D52. For example, if the response information D52, which is the output from the learning model unit 500, contains data in a format other than an image, the image generation unit 514i can generate and output an image representing the content shown by the response information D52. For example, if the response information D52 is a data structure including a specification of the data format, the image generation unit 514i can convert the data elements specified in the specification as image format into image format and output them. The image generation unit 514i can also generate image-format response information D52v based on text-format response information D52. For example, the image generation unit 514i can also perform a synthesis process that adds the content shown by the text-format response information D52 as an annotation to the image contained in the input information D51. Furthermore, the image generation unit 514i can also perform a process that emphasizes a portion of the image contained in the input information D51 based on the text-format response information D52. The image generation unit 514i can also use a learning model to generate an image based on the input information (response information D52 and, if necessary, input information D51).

[0544] The program generation unit 514p converts the content shown in the response information D52 into the data form of a prescribed program and outputs it. For example, if the response information D52, as output from the learning model unit 500, contains a data form other than the data form of a prescribed program, the program generation unit 514p converts that portion of the response information D52 into the data form of the prescribed program and outputs it. For example, if the response information D52 is a data structure that includes a specified data form, the program generation unit 514p may also convert data elements whose specified data form of a prescribed program is specified in that specification into the data form of the prescribed program and output it. The program generation unit 514p may also convert the text-based response information D52 into the data form of the response information D52p, which is the data form of a prescribed program. The program generation unit 514p may also use a learning model to generate a prescribed program based on the input information.

[0545] The image parsing process of the image parsing unit 513i is performed, for example, in step S511 described above. Furthermore, the image generation process of the image generation unit 514i and the program generation process of the program generation unit 514p are performed, for example, in step S514 described above.

[0546] In other respects, it can be the same as other control systems in this embodiment.

[0547] As described above, according to this variation, inquiries and responses can be made not only by voice but also by a combination of voice and images. Therefore, for example, inquiries regarding the operation screen can be answered more effectively. Furthermore, according to this variation, in addition to voice and images, the program can be provided as response information to the sending source, thus enabling more effective responses to inquiries regarding handling adverse situations.

[0548] Variation 5-4.

[0549] Next, the fourth variation of the control system 5000 will be described. Figure 32 This is a structural diagram showing an example of a modified control system 5000d, which is a variation of the control system 5000 of this embodiment. Furthermore, elements identical to those in control systems 5000 to 5000c are labeled with the same reference numerals and their descriptions are omitted.

[0550] In this variant, the following function is provided: based on the query content from user 1 and / or the output result from the learning model, the response is switched to the response of operator 8 or the response of another learning model.

[0551] like Figure 32 As shown, the control system 5000d can also include a call confirmation unit 531 and an output selection unit 532.

[0552] Here, the control system 5000d includes a learning model unit 500a as a first response function, and an operator 8 and a communication channel with the operator 8 as a second response function. Alternatively, as a third response function, the control system 5000d may also include another learning model unit 500b with an algorithm or data different from that of the learning model unit 500a. Furthermore, as a second response function, it is also possible to include another learning model unit 500b with an algorithm or data different from that of the learning model unit 500a. In this case, as a third response function, the operator 8 and a communication channel with the operator 8 may also be included. Moreover, the type and number of response functions are not particularly limited. For example, the response function for switching destinations may also be a response system that does not use a learning model.

[0553] In this example, the following situation will be used as an example: the learning model unit 500a, which is set as the first response function, is the learning model unit 500; the second response function is the operator 8 and the communication channel with the operator 8; and the third response function is another learning model unit 500b, whose algorithm or the data used is different from that of the learning model unit 500a.

[0554] Here, the learning model unit 500a can be a local learning model that obtains output results based on local information, such as when the database of the reference destination is limited, while the learning model unit 500b can be a global learning model that obtains output results based on global information, such as when it can freely access external networks.

[0555] The confirmation unit 531 switches the processing destination for response processing based on the query content from user 1 and / or the output results from the learning model.

[0556] The call confirmation unit 531 can also call operator 8 as the second response function based on the query content from user 1 and / or the output result from the learning model, for example, if it is determined that the output accuracy of the first response function cannot be expected. The call confirmation unit 531 can also call operator 8 using a communication channel with operator 8, and input the input information D51 to the operator 8's operating device. Furthermore, the call confirmation unit 531 can also call operator 8 using a communication channel with operator 8, and input the input information D51 to the operator 8's operating terminal (not shown).

[0557] Furthermore, the call confirmation unit 531 can also call the learning model unit 500b as a third response function if it further determines that the second response function cannot be called or the accuracy of the output cannot be expected. For example, the call confirmation unit 531 can call the learning model unit 500b by inputting the input information D51 into the learning model unit 500b through the interface with the learning model unit 500b.

[0558] Here, the accuracy of the output can be determined using, for example, the evaluation value or likelihood of the response function's own output, or the reliability evaluation mentioned above. Furthermore, in cases where the response function itself outputs a message that is unclear or indicates a delegated call to other functions, the presence or absence of such a message can also be used to make a judgment.

[0559] Based on the switching result of the response processing by the call confirmation unit 531, the output selection unit 532 selects the response information D52 to be output to the user 1. When the switching result of the call confirmation unit 531 is that the execution entity of the response processing is set to the first response function, the output selection unit 532 outputs response information D52a as output from the first response function to the user 1. Furthermore, when the switching result of the call confirmation unit 531 is that the execution entity of the response processing is set to the second response function, the output selection unit 532 outputs response information D52b as output from the second response function to the user 1. Furthermore, when the switching result of the call confirmation unit 531 is that the execution entity of the response processing is set to the third response function, the output selection unit 532 outputs response information D52c as output from the third response function to the user 1.

[0560] The output selection unit 532 can also output the output from the selected response function to the user 1 by controlling the output switching switch (not shown). The output switching switch switches the connection path (circuit or communication path, etc.) between the response function as the execution subject and the user 1 as the output target.

[0561] Here, the connection path between the response function and user 1 may include various conversion devices and specified interfaces, such as the aforementioned speech synthesis unit, image generation unit, and program generation unit, as needed.

[0562] For example, when operator 8 outputs text input as response information D52b via the operation terminal, the connection path between the response function and user 1 may include a speech synthesis unit that converts text into speech. Furthermore, as output of the second response function, the output selection unit 532 can also accept information that has been corrected from the response information D52a output by the first response function. In this case, operator 8's operation terminal includes a text display unit and a text input unit, and the control system 5000d can, for example, accept response information D52b obtained by correcting a portion of the response information D52b output from operator 8's operation terminal.

[0563] In other respects, it can be the same as other control systems in this embodiment.

[0564] As described above, according to this variation, in addition to using the learning model unit 500 described above to generate responses, responses can also be generated by the operator, or by using other learning models (e.g., including a serial structure model that connects multiple models, a multimodal model, or a model specifically learned for a given device or service) to generate responses, thereby further achieving high performance in responding to users.

[0565] In robot control, including instructions for robot operation and coordinated human-robot work, the complexity and sophistication depend on the relative relationship between humans and machines. By adopting the structures of the above-described embodiments and their variations, the complexity and sophistication of robot control can be improved or reduced. Therefore, it is possible to mitigate deviations in the relative operational capabilities of humans towards machines or further reflect differences in human preferences for operating machines.

[0566] Furthermore, while each of the above embodiments has been described with an example of a system structure corresponding to the operation of interest, the control system disclosed herein is not limited to the examples described above. For instance, the control system of this disclosure may also be a suitable combination of one or more of the embodiments described above.

[0567] As an example, the control system disclosed herein can also combine the structure of Embodiment 1 with the structure of Embodiment 4, and input the information representing the solution obtained from the sensor data using the function of Embodiment 4 into the control system of Embodiment 1 and convert it into a program to directly control the object device 2.

[0568] Furthermore, the various embodiments and variations are not limited to the examples described above, and can be appropriately modified within the scope of disclosure.

[0569] Furthermore, the control systems and control methods disclosed herein include those described in the following notes.

[0570] (Note 1)

[0571] A control system for assisting a task performed by a person or object using equipment, characterized by comprising: an input interface for accepting input of first information, the first information representing the environment in which the task is performed, i.e., the conditions or requirements in the work environment; a model processing unit configured to access a prescribed learning model; and an output interface for outputting second information for assisting the task based on output from the learning model, wherein the model processing unit inputs model input data based on the first information into the learning model, accepts model output data corresponding to the model input data from the learning model, the model output data containing information for the task, and the output interface outputs the second information based on the model output data.

[0572] (Note 2)

[0573] According to the control system described in Appendix 1, the first information includes information representing control content or operation content requested for the device; the model input data represents the control content or operation content represented by the first information in the form of input conforming to the learning model; the model output data includes information for controlling or operating the device corresponding to the control content or operation content represented by the model input data; and the second information includes information that describes the information for controlling or operating the device contained in the model output data in a prescribed form that can be discerned in the output destination of the output interface.

[0574] (Note 3)

[0575] According to the control system described in Appendix 2, the output destination of the output interface is the device or an interface requesting control of the device, and the result of outputting the second information to the device or the interface requesting control of the device is that the device is controlled.

[0576] (Note 4)

[0577] According to the control system described in Appendix 2, it further includes an execution code generation unit that generates and outputs execution code as code that the device can execute. The output destination of the output interface is the execution code generation unit. The result of outputting the second information to the execution code generation unit is that the device is controlled by the generated execution code.

[0578] (Note 5)

[0579] According to the control system described in Appendix 2, the output destination of the output interface is a user-operated terminal, and the result of outputting the second information to the terminal is that the device is controlled.

[0580] (Note 6)

[0581] According to the control system described in Appendix 1, the first information includes information representing the state of the working environment, the model input data is data representing the state of the working environment represented by the first information in the form of input conforming to the learning model, the model output data includes information related to the analysis results and / or improvement methods of the state of the working environment represented by the model input data, and the second information includes information obtained by describing the information related to the analysis results and / or improvement methods of the state of the working environment in a prescribed form that can be discerned in the output destination of the output interface.

[0582] (Note 7)

[0583] According to the control system described in Appendix 1, the model processing unit is configured to access a first learning model and a second learning model. The model processing unit inputs first model input data based on the first information into the first learning model and receives first model output data corresponding to the first model input data from the first learning model. The model processing unit inputs second model input data based on the first model output data into the second learning model and receives second model output data corresponding to the second model input data from the second learning model. The output interface outputs the second information based on the second model output data.

[0584] (Note 8)

[0585] According to the control system described in Appendix 7, the first information includes information representing control content or operation content requested for the device; the first model input data represents the control content or operation content represented by the first information in the form of input conforming to the first learning model; the first model output data includes information showing the control content or operation content represented by the first model input data in a more generalized or specific manner; the second model input data represents the control content or operation content represented by the first model output data in the form of input conforming to the second learning model; the second model output data includes information for controlling or operating the device corresponding to the control content or operation content represented by the second model input data; the second information includes information that describes the information for controlling or operating the device contained in the second model output data in a prescribed form that can be discerned in the output destination of the output interface.

[0586] (Note 9)

[0587] According to the control system described in Appendix 7, the first information includes information representing the state of the working environment; the first model input data represents the state of the working environment represented by the first information in the form of input conforming to the first learning model; the first model output data includes the analysis result of the state of the working environment represented by the model input data; the second model input data represents the analysis result of the state of the working environment represented by the first model output data in the form of input conforming to the second learning model; the second model output data includes information related to the improvement method of the state of the working environment corresponding to the analysis result of the working environment represented by the second model input data; the second information includes information that describes, in a prescribed form that can be discerned in the output destination of the output interface, the information contained in the second model output data that is at least related to the improvement method of the state of the working environment.

[0588] (Postscript 10)

[0589] According to the control system described in Appendix 1, wherein the operation is a response performed by a person or object using equipment, the first information includes information representing the content of a response request, which is the content requesting a response in the operation environment, the model input data is data representing the response request content represented by the first information in the form of an input conforming to the learning model, the model output data includes information for the response corresponding to the response request content represented by the model input data, and the second information includes information that describes the information for the response contained in the model output data in a prescribed form that can be discerned in the output destination of the output interface.

[0590] (Postscript 11)

[0591] According to the control system described in Appendix 3, the output destination of the output interface is a screen operation interface that requests control of the device via an operation screen, and the model output data includes information of the operation screen described in a prescribed form that can be discerned in the output destination of the output interface. The operation screen is used to actually perform operations on the device that correspond to the control content or operation content represented by the model input data.

[0592] (Postscript 12)

[0593] According to any one of Appendices 1 to 11, the control system wherein the input interface accepts input of first information representing the needs of the working environment from a plurality of users, the model processing unit inputs model input data containing the first information input from the plurality of users into the learning model, and accepts model output data corresponding to the model input data from the learning model.

[0594] (Postscript 13)

[0595] The control system according to any one of Appendices 1 to 12, wherein the learning model is a language learning model that takes natural language as input and obtains output, an image learning model that takes an image as input and obtains output, and a multimodal model that takes natural language and an image as input and obtains output.

[0596] (Postscript 14)

[0597] According to any one of Appendices 1 to 13, the control system wherein the model processing unit is configured to access a first learning model and a second learning model, one of which is a local learning model whose reference destination database is restricted to internal information, and the other of which is a global learning model whose reference destination database is not restricted to internal information.

[0598] (Postscript 15)

[0599] According to any one of Appendices 1 to 13, the control system wherein the model processing unit is configured to access a first learning model and a second learning model, one of which is a learning model that can refer to information specifically specified in the operating environment, and the other of which is a learning model that cannot refer to information specifically specified in the operating environment.

[0600] (Postscript 16)

[0601] The control system according to any one of Appendices 1 to 15 includes an output confirmation unit that performs a simulation of the control and state of the device based on model output data output from the learning model.

[0602] (Postscript 17)

[0603] According to any one of the appendices 1 to 16, the control system performs additional learning of the learning model, or determination of the correctness of the output information, or flow control of the output information based on information collected from the output destination of the output interface.

[0604] (Postscript 18)

[0605] The control system according to any one of Appendices 1 to 17 includes an input processing unit that queries the input source when the first information contains ambiguous or uncertain information.

[0606] (Postscript 19)

[0607] The control system according to any one of Appendices 1 to 18, wherein the query includes information on the modification, addition, or cancellation of content shown in response to the input and output data of the learning model.

[0608] (Postscript 20)

[0609] According to any one of the appendices 1 to 19, the control system wherein the first information is time-series data representing the environment in which the operation is performed, i.e., the conditions or requirements in the work environment, together with information about time.

[0610] (Postscript 21)

[0611] The control system according to any one of Appendices 1 to 19 comprises: a model information storage unit that stores model information as an execution environment for the learning model; and a model control unit that accepts the model input data and outputs the model output data based on the model input data and the information stored in the model information storage unit.

[0612] (Postscript 22)

[0613] A control method for assisting a person or object using a device in performing an operation, characterized in that an input interface accepts input of first information, which represents the environment in which the operation is performed, i.e., the conditions or requirements in the work environment; a model processing unit configured to access a predetermined learning model inputs model input data based on the first information into the learning model; the learning model accepts model output data corresponding to the model input data, which contains information for the operation; and an output interface outputs second information based on the output from the learning model, which is used to assist the operation.

[0614] Industrial availability

[0615] The control system disclosed herein can be suitably applied as part of a work assistance system that assists people or things in their work. Furthermore, the control system disclosed herein can be suitably applied as a control system that controls equipment when it is used to perform a certain control or operation. Here, the control system can also be suitably applied as a control system for controlling FA (Automatic Facilitation) equipment, a control system in a home or building, a control system for controlling information processing devices such as a server device that processes information on a network.

[0616] Label Explanation

[0617] 1000, 1000a, 1000b, 1000c, 2000, 3000, 3000a, 3000b, 3000c, 3000d, 3000e, 4000, 4000a, 5000, 5000a, 5000b, 5000d Control System; 100, 200, 300, 300a, 300b, 400, 400a, 400b, 500, 500a, 500b Learning Model Unit; 10, 20 Information Processing Device; 11 Model Information Storage Unit; 12 Reference Information Storage Unit; 101 Model Control Unit; 102 Input Unit; 103 Output Unit; 105 Preprocessing Unit; 106 Postprocessing Unit; 107 Model Generation Unit; 104, 104a Control Unit; 201 Input Processing Unit; 202 Output Confirmation Unit; 203 Correction Confirmation Unit; 1 User; 1a Input Source; 2 Object Device; 2a Output Destination; 3 User Interface; 4 Controller; 4a General Operation Device; 41-1, 41-2 Analysis Unit; 42 Switching Unit; 43 Output Switch; 5 Sensor; 6 Model Interface; 7 Display; 8 Operator; 110, 210, 310, 410 Device Information Storage Unit; 120 Execution Code Generation Unit; 230 Status Acquisition Unit; 311 Input Interface; 312 Output Interface; 313 Environmental Information Storage Unit; 350 Display Format Information; 360 User History Information Storage Unit; 511 Database Retrieval Unit; 512 Control Generation Unit; 513v Speech Recognition Unit; 513i Image Analysis Unit; 514v Image Synthesis Unit; 514p Program Generation Unit; 515 Correction Judgment Unit; 516 517 Emotion Determination Department; 518 Evaluation Acquisition Department; 519 Registration Determination Department; 531 Additional Learning Department; 532 Call Confirmation Department; 532 Output Selection Department; 533 Output Switching Department; D101 Model Input Data; D102 Model Information; D103 Model Output Data; D104 Model Reference Information; D11, D21, D31, D41, D51, D51v, D51i Input Information; D12 Control Description; D22, D34 Control Commands; D42a, D42b Analysis Results; D52, D52v, D52i, D52p, D52a, D52b, D52c Response Information; D32, D32a, D32b Operation Commands; D320 Operation Information; D13, D23, D33, D43 Equipment Information; D33a Environmental Information; D14 Execution code; D44a, D44b Result information; D15, D25, D35, D45 Status information; D16, D26, D36, D46 Feedback information; D17, D27, D37, D47, D57 Inquiry; D18, D28, D38, D48 Supplementary information; D59 Evaluation information; D361 User history information.

Claims

1. A control system comprising: An input interface that accepts input of first information, which represents a user command to the target device, the command consisting of multiple operations on the target device; and The learning model unit inputs the first information into the learning model and outputs second information representing sequential control, which is a series of controls in the object device for implementing the multiple operations.

2. The control system according to claim 1, wherein, The learning model unit outputs the information of sequential control represented by the control code of the object device as the second information to the object device.

3. The control system according to claim 1 or 2, wherein, The control system includes an input determination unit that uses command rules obtained by mapping commands for the target device to control codes of the target device to determine whether the command represented by the first information conforms to the commands included in the command rules. If the command represented by the first information conforms to the command contained in the command rule, the input judgment unit outputs the control code corresponding to the command represented by the first information to the object device; if the command represented by the first information does not conform to the command contained in the command rule, the input judgment unit outputs the first information to the learning model unit.

4. The control system according to claim 1 or 2, wherein, The control system includes an input judgment unit that uses machine learning to determine whether a command for the target device corresponds to the control code of the target device. When the input judgment unit determines that the command represented by the first information corresponds to the control code of the target device, it outputs the control code of the target device corresponding to the command represented by the first information to the target device. When it determines that the command represented by the first information does not correspond to the control code of the target device, it outputs the first information to the learning model unit.

5. The control system according to any one of claims 1 to 4, wherein, The input interface accepts multiple commands from different users as multiple pieces of first information. The learning model unit coordinates the multiple commands represented by the multiple first information and outputs information representing a sequential control of the multiple commands as the second information.

6. The control system according to claim 1, wherein, The control system includes a device operation unit, which is a user interface for operating the target device. The learning model generates an operation screen representing the sequential control as the second information, displays the operation screen on the device operating mechanism, and allows the user to select whether to agree to the sequential control via the operation screen. When the user agrees to the sequential control, the device operating mechanism sends a control code indicating the sequential control agreed upon by the user to the target device.

7. The control system according to claim 6, wherein, The learning model generates an operation screen representing a candidate list of sequential controls as the second information, and displays the operation screen on the device operating device, allowing the user to select sequential control from the candidate list via the operation screen. The device operating mechanism sends the control code of the target device to the target device, treating the sequential control selected by the user as the sequential control agreed upon by the user.

8. The control system according to claim 6 or 7, wherein, The device operating device is a general-purpose operating device capable of operating the target device and other devices different from the target device. The learning model unit uses display format information stored in the device operation device to display the display representing the sequential control as the second information on the display screen of the device operation device.

9. The control system according to claim 8, wherein, The device operating mechanism is capable of operating multiple devices of different types used by the user.

10. The control system according to any one of claims 6 to 9, wherein, The device operating apparatus has the input interface, which assists the user in inputting commands to the target device by displaying a sequence of controls agreed upon by the user on the display screen.

11. The control system according to any one of claims 1 to 10, wherein, The learning model unit outputs the second information corresponding to a specific environment based on environmental information representing the environment of the object device.

12. The control system according to claim 11, wherein, The environmental information includes at least any one of the following: the state of the object device, the state of the surroundings of the object device, and the state of the user, which includes the user's vital signs data.

13. The control system according to any one of claims 11 to 12, wherein, The learning model unit outputs the second information corresponding to the user based on user history information that includes the user's history of commands to the object device.

14. The control system according to claim 13, wherein, The user history information includes the commands input by the user, the status of the user and the target device, the control codes output by the target device, and changes in the environment. The learning model unit outputs the second information, which corresponds to the state of the user and the object device, based on the user's historical information.

15. The control system according to any one of claims 13 to 14, wherein, The learning model unit has the following features: The first learning model unit inputs the first information into the first learning model and outputs operation information representing the sequential control corresponding to a specific environment based on the environmental information. as well as The second learning model unit inputs the operation information into the second learning model and outputs the control code of the object device used to implement the operation information as the second information to the object device.

16. The control system according to claim 15, wherein, The first learning model unit inputs the first information into the first learning model, and outputs the operation information representing the sequential control corresponding to the state of the user and the object device based on the environmental information and the user history information.

17. The control system according to any one of claims 1 to 16, wherein, The learning model is a large language model or a visual language model.

18. The control system according to any one of claims 1 to 17, wherein, The input interface accepts input of at least one or a combination of signal codes, natural language, and images as the first information.

19. A control method, wherein, The computer accepts input of first information, which represents a user's command to the target device, the command consisting of multiple operations on the target device. The computer inputs the first information into the learning model and outputs the second information representing sequential control, which is a series of controls in the object device used to implement the multiple operations.

20. A control program that causes a computer to perform: Input interface processing accepts input of first information, which represents a user's command to the target device, the command consisting of multiple operations on the target device; and The learning model processes the first information into the learning model and outputs the second information representing sequential control, which is a series of controls in the object device used to implement the multiple operations.

Images