CONTROL SYSTEM, CONTROL METHOD, AND CONTROL PROGRAM

The control system addresses issues with learning model validity and responsiveness by generating sequential control commands, enhancing the efficiency and performance of complex task execution.

JP7595817B1Active Publication Date: 2024-12-06MITSUBISHI ELECTRIC CORP

Patent Information

Application Number
JP2024544530
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2023-11-09
Filing Date
2023-11-30
Publication Date
2024-12-06
Estimated Expiration
2043-11-30

AI Technical Summary

Technical Problem

Existing learning models face challenges in ensuring the validity of outputs, accurately recognizing situations, and maintaining responsiveness and maintainability, particularly in complex tasks, leading to reduced efficiency and performance in controlling equipment and processes.

Method used

A control system utilizing a learning model that integrates an input interface, a learning model unit, and an execution code generation unit to generate sequential control commands based on user inputs, device information, and model learning, enhancing the efficiency and performance of work by improving the accuracy and responsiveness of control operations.

Benefits of technology

The system enhances the efficiency and performance of controlling complex tasks by accurately generating and executing sequential control commands, improving the accuracy and responsiveness of equipment operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007595817000001
    Figure 0007595817000001
  • Figure 0007595817000002
    Figure 0007595817000002
  • Figure 0007595817000003
    Figure 0007595817000003
Patent Text Reader

Abstract

The control system (3000) includes an input interface (311) that accepts input of input information (D31) that is a command for a target device (2) by a user (1). The input information (D31) is information that indicates a command consisting of a plurality of operations for the target device (2). The learning model unit (300) inputs the input information (D31) to a learning model, and outputs an operation command (D32) that indicates sequential control that is a series of controls in the target device (2) to realize the plurality of operations.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a control system, a control method, and a control program. [Background technology]

[0002] In recent years, the use of AI (artificial intelligence) has been progressing. In particular, generative artificial intelligence, an AI that can generate a variety of content, is becoming more widespread, and it is expected that the uses of AI will expand. AI will not only be used in households, but also in various facilities such as buildings, factories, stations, schools, hospitals, and commercial facilities, and outdoors such as roads, outdoor facilities, and in the sky or sea, and in various other places and situations.

[0003] For example, Patent Document 1 describes a machining program generation device that generates a program for controlling a machine by utilizing a large-scale language model. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-060806 Summary of the Invention [Problem to be solved by the invention]

[0005] In order to support work by people or objects, it is considered that a learning model will take on some or all of the tasks involved in the work, not limited to generating control programs for devices. Note that "work by people or objects" includes not only work in the real world performed by people or machines, but also work in the data space, such as information processing performed by a processor such as a CPU (central processing unit).

[0006] Examples of operations using objects include the following: -Work using various devices such as robots, machines, equipment, and sensors -Work using various types of mobility, including cars, trains, buses, aircraft, and ships Such operations may include, for example, operations referred to as controlling, processing, machining, directing, calculating, inputting, outputting, displaying, communicating, testing, manufacturing, converting, generating, measuring, irradiating, emitting, inhaling, dissipating heat, heating, cooling, recording, reading, forming, driving, moving, transporting, flying, investigating, monitoring, measuring, extracting, and the like.

[0007] Examples of tasks performed by humans include the following: Work done by humans on other humans or living creatures Work performed by humans on various equipment Such activities may include, for example, activities known as speaking, listening, checking, operating, monitoring, directing, mediating, interpreting, and the like.

[0008] Note that the above-mentioned examples are merely examples, and the tasks that can be supported by the present disclosure are not limited to these.

[0009] When using some kind of learning model to make an information processing device execute some or all of the tasks involved in work by a person or object, the validity of the model output may be an issue, and the validity of the model input that affects the model output may be an issue.

[0010] Also, depending on the target device, appropriate control may not be possible without first understanding the current situation. In such cases, how to recognize the situation may become an issue. In such cases, it may be necessary to recognize a situation that has continuity, including not only the current situation but also past situations. For example, when determining the next control based on the content of past control, the accuracy of situation recognition may become an issue in order to ensure the continuity of control.

[0011] In addition, in cases where immediacy is required for controlling equipment, the response time between giving instructions to the learning model and obtaining results can be an issue.

[0012] Furthermore, the maintainability of the model may be an issue, since the model must be retrained every time a device is changed or added.

[0013] As described above, the use of learning models still involves various problems. Depending on the scale of the problem, the use of learning models to improve work efficiency or performance may actually result in a decrease in work efficiency or performance.

[0014] These problems when using learning models will become more pronounced, particularly as the task to be supported becomes more complex and the task becomes more advanced.

[0015] Therefore, an object of the present disclosure is to utilize a learning model to improve the efficiency or performance of work performed by people or objects. [Means for solving the problem]

[0016] The control system according to the present disclosure comprises: an input interface that accepts input of first information indicating a command for a target device by a user, the command being composed of a plurality of operations for the target device; a learning model unit that inputs the first information into a learning model and outputs second information indicating a sequential control that is a series of controls in the target device for realizing the plurality of operations; Equipped with. Effect of the Invention

[0017] In the present disclosure, by using a learning model, it is possible to output information indicating sequential control in a target device in response to a complex command that requires multiple operations. Therefore, according to the present disclosure, it is possible to improve the efficiency or performance of work by a user. [Brief description of the drawings]

[0018] [Figure 1] FIG. 1 is a configuration diagram illustrating an example of a control system according to a first embodiment. [Diagram 2] FIG. 2 is an explanatory diagram showing an example of the configuration of a learning model unit. [Diagram 3] FIG. 13 is an explanatory diagram showing another example of the configuration of the learning model unit. [Figure 4] FIG. 13 is an explanatory diagram showing another example of the configuration of the learning model unit. [Diagram 5] FIG. 2 is a configuration diagram showing an example of an information processing device that is the operating environment of a control unit including a learning model unit. [Figure 6] FIG. 11 is an explanatory diagram showing an example of model learning in a model generation unit. [Figure 7] 4 is a flowchart showing an example of the operation of the control system according to the first embodiment. [Figure 8] FIG. 4 is a configuration diagram showing another example of the control system according to the first embodiment. [Figure 9] FIG. 4 is a configuration diagram showing another example of the control system according to the first embodiment. [Figure 10] FIG. 4 is a configuration diagram showing another example of the control system according to the first embodiment. [Figure 11] FIG. 11 is a configuration diagram illustrating an example of a control system according to a second embodiment. [Figure 12] FIG. 11 is a configuration diagram showing another example of the control system according to the second embodiment. [Figure 13] 11 is a flowchart showing an example of the operation of the control system according to the second embodiment. [Figure 14] FIG. 11 is a configuration diagram illustrating an example of a control system according to a third embodiment. [Figure 15] 13 is a flowchart showing an example of the operation of the control system according to the third embodiment. [Figure 16] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 17] 13 is a flowchart showing an operation example of a modified example according to the third embodiment. [Figure 18]FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 19] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 20] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 21] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 22] 13 is a flowchart showing an operation example of a modified example according to the third embodiment. [Figure 23] FIG. 13 is a configuration diagram illustrating an example of a control system according to a fourth embodiment. [Figure 24] 13 is a flowchart showing an example of the operation of the control system according to the fourth embodiment. [Diagram 25] FIG. 13 is a configuration diagram showing another example of the control system according to the fourth embodiment. [Figure 26] 13 is a flowchart showing an operation example of a modified example according to the fourth embodiment. [Figure 27] FIG. 13 is a configuration diagram illustrating an example of a control system according to a fifth embodiment. [Figure 28] 13 is a flowchart showing an example of the operation of the control system according to the fifth embodiment. [Figure 29] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Diagram 30] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Diagram 31] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Diagram 32] FIG. 13 is a configuration diagram showing another example of the control system according to the fifth embodiment. [Diagram 33] 13 is a flowchart showing an operation example of a modified example according to the third embodiment. [Diagram 34] 13 is a flowchart showing an operation example of a modified example according to the third embodiment. [Diagram 35] 13 is a flowchart showing another example of the operation of the modified example according to the third embodiment. [Diagram 36] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 37] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Figure 38] 13 is a flowchart showing an operation example of a modified example according to the third embodiment. [Figure 39] FIG. 11 is a configuration diagram showing another example of the control system according to the third embodiment. [Diagram 40] 13 is a flowchart showing an operation example of a modified example according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] In order to describe the present disclosure in more detail, the embodiments of the present disclosure will be described below with reference to the accompanying drawings. In the following, the same elements are denoted by the same reference numerals and the description thereof will be omitted.

[0020] Embodiment 1 In this embodiment, an example will be described in which a learning model is used to assist in the task of generating code for a target device.

[0021] Fig. 1 is a configuration diagram illustrating an example of a control system 1000 according to the first embodiment. The control system 1000 illustrated in Fig. 1 is a control system for controlling an appliance using a learning model, and includes a learning model unit 100, an appliance information storage unit 110 (referred to as appliance information DB in the figure), and an execution code generation unit 120.

[0022] 1 shows a user 1 and a target device 2, but these may be included in the control system 1000. In that case, the "user 1" may be read as a "user terminal 1." This point is the same in the other embodiments.

[0023] When input information D11 is input, the learning model unit 100 outputs a control description D12. When input information D11 is input, the learning model unit 100 outputs a control description D12 based on model information D102, which will be described later.

[0024] In this embodiment, the learning model unit 100 is a model configured to output a control description D12 corresponding to the input information D11 when the input information D11 is input, and its operating environment. The learning model unit 100 may also be a model configured to generate and output a control description D12 based on the input information D11, device information D13, and / or other information that can be referenced in the learning model unit 100 (such as model reference information D104 described later) when the input information D11 is input.

[0025] In this embodiment, the input information D11 includes information indicating the control content required for the target device 2. The input information D11 may be, for example, text, image, voice, or a combination thereof indicating the control content for the target device 2. The input information D11 may be, for example, text, image, voice, or a combination thereof indicating a plurality of control contents for the target device 2. The input information D11 may also include information indicating the content of control that is performed continuously in time, and in that case, may be time-series data of a predetermined data structure including text, image, voice, or a combination thereof indicating the control content as described above. It is assumed that the way of indicating the control content matches the input format of the model used by the learning model unit 100, but this is not limited to the case where error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 100.

[0026] An example of a method of indicating the control content in the input information D11 is to specify a control to be performed on the target device 2, and then specify a parameter value for performing the control or a state after the control. In this case, the input information D11 may include, for example, information specifying the control and information indicating a parameter value for performing the control or a state after the control. The parameter value for performing the control may include, for example, values ​​related to the type of control (ON / OFF, etc.), direction, amount, and time. Examples of the control content include "turn on function X" for a programmable logic controller (PLC), "move the tip to point A" for a robot arm, and "lower the set temperature by 1 degree" for an air conditioner. In addition, an example of a method of indicating the control content in the input information D11 is to use a document string (docstring) that describes the specifications of a function, a specification, or various information such as a specification, design document, operation command, control code, and source code that are applied to other devices such as other models.

[0027] In addition, as a method of indicating the control contents in the input information D11, not only the above-mentioned explicit method, but also a method of indicating the corresponding control contents by indicating the operation contents when there is a control performed by a certain operation. In addition, for example, a method of implicitly indicating the control contents by the user 1's words and actions associated with a specific control, the operation result of the target device 2, or a similar control command to another model, etc., can be included. In other words, the input information D11 can include not only information directly indicating the control contents for the target device 2, but also information indirectly indicating the operation contents corresponding to the control contents, the user 1's words and actions, images of the target device 2, etc. As an example, the user 1's words such as "It's hot" or the user 1's behavior of feeling hot, such as wiping sweat, rolling up sleeves, and raising hands, can be used to indicate the control contents related to the temperature control of the air conditioner. In this case, the input information D11 can be information such as text, audio, or images indicating the user 1's remarks, or images (videos) indicating the user 1's behavior. As another example, the control content related to the arm control of a robotic device can be indicated by information specifying the posture of the robotic device after control, a destination point of a specified part, or information indicating an imitation of the robot's movements by a person or other object (a simulator that simulates the movements of the robot, including objects on a screen) or an instruction action given to the robot (an instruction to an action by a gesture such as pointing).

[0028] Here, the format of the input information D11 is not particularly limited. For example, the information may be text, image, sound, data written in a predetermined design language, control description (including source code and information written in a predetermined programming platform language), information written in other platform languages, control instructions (including control instructions, control signals, control codes, and controller commands), or execution codes. These information may be appropriately combined. In this disclosure, when "text" is used without any particular distinction, it may include data written in a predetermined design language that cannot be discerned by humans, control descriptions (including source code and information written in a predetermined programming platform language), information written in other platform languages, control instructions (including control instructions, control signals, control codes, and controller commands), and data that can be discerned by machines, such as execution codes, expressed in text, in addition to natural language expressed in text.

[0029] The control description D12 includes information about control written in a predetermined format that can be recognized by the execution code generation unit 120 at the subsequent stage. The control description D12 is, for example, source code written in a predetermined programming language. The control description D12 may also be, for example, a group of commands written in a format (platform language) handled by a predetermined programming platform. Here, the predetermined programming platform may include a no-code programming platform and a low-code programming platform.

[0030] The device information storage unit 110 stores device information D13, which is information related to the target device 2. The device information D13 may include, for example, information indicating the function, performance, structure, dimensions, operation, and / or control method of the target device 2. The device information D13 may also include, for example, information related to a program used to control the target device 2. The device information D13 may be, for example, a digitized manual or instruction manual for the target device 2. The digitization here includes digitization into text, image data, digitization by voice reading, and combinations thereof. The device information D13 is used, for example, as additional information when the learning model unit 100 outputs the control description D12.

[0031] The device information D13 may also include information indicating the state of the target device 2. The information indicating the state of the target device 2 may include not only the current state of the target device 2 but also information indicating the past state. For example, the device information D13 may include time-series data of a predetermined data structure indicating the state of the target device 2. The information indicating the state of the target device 2 may be, for example, information output from the target device 2, or information input by the user 1 or another device. The information indicating the state of the target device 2 may include various types of information (for example, error information, log information, notification information, etc.) output from the target device 2. Hereinafter, in this embodiment, the information indicating the state of the target device 2 may be referred to as state information D15.

[0032] When the control description D12 is input, the execution code generation unit 120 generates and outputs an execution code D14, which is code executable by the target device 2, based on the control description D12. The execution code D14 may be, for example, a group of codes written in machine language. The execution code D14 includes, for example, information used when the target device 2 actually performs control. The execution code D14 may be, for example, information related to control written in a format that can be identified by the target device 2. The execution code generation unit 120 may be, for example, a compiler that converts the control description D12 into the execution code D14.

[0033] The execution code D14 output from the execution code generation unit 120 is input to the target device 2. As a result, the target device 2 operates according to the execution code D14 output from the execution code generation unit 120. The execution code D14 may be input to the target device 2 directly from the execution code generation unit 120, or may be input indirectly via a communication network, another device (such as a server or various conversion devices), or manually.

[0034] There is no particular limitation on the target device 2. It is assumed that the target device 2 is a device that can receive and actually execute the execution code D14, but this is not the case when an interface that allows the target device 2 to read the execution code, such as a writing device, is included between the target device 2 and the target device 2.

[0035] The target device 2 may be, for example, a PLC, a processing machine, a robot, a radar, a sensor, a camera, a projector, or a communication device. The target device 2 may be, for example, an air conditioner, a refrigerator, a television, a light, or a washing machine. The target device 2 may be, for example, an elevator, a mobility, a conveying device, or other machines, or a control device that controls such machines. The target device 2 may be a device that operates in a power generation / transformation / storage plant, a water treatment plant, or other devices, or a control device that controls other devices. The learning model unit 100 may generate the execution code D14. For example, if the control description D12 is an interpreter language and the target device 2 is a device that can receive the control description D12 and execute it as it is, the control description D12 can be regarded as the execution code D14. In this case, the execution code generation unit 120 can be omitted. The control description D12 may be converted into the execution code D14 that is more suitable for the processing of the target device 2 by a compilation or optimization process.

[0036] Fig. 2 is an explanatory diagram showing a configuration example of the learning model unit 100. As shown in Fig. 2, the learning model unit 100 may include a model control unit 101 that operates on the information processing device 10, and a model information storage unit 11 (referred to as model information DB in the figure) that stores model information D102. Here, the model information storage unit 11 may be composed of a plurality of databases connected via a network.

[0037] The model information D102 includes model information. The model information D102 may include, as the model information, information indicating a correlation between the model input data D101 and the model output data D103. The model information D102 may also include, as the model information, information indicating candidates for the model output data D103. The model information D102 may also include, as the model information, information indicating candidates for the model output data D103 and information indicating the relationship between the candidates. The model information D102 may also include model parameters, which are information that specifies the behavior of the learning model, such as constraint conditions, weighting variables, and evaluation functions.

[0038] The model may be a model obtained by machine learning, for example, by supervised learning, reinforcement learning, or unsupervised learning. The model may be a model obtained by performing learning according to, for example, deep learning, genetic programs, functional logic programs, or other known algorithms and methods. The model may also be a model called, for example, a Neural Network (NN) model, a Convolutional Neural Network (CNN) model, a Recurrent Neural Network (RNN), a Variational Autoencoder (VAE), a Generative Adversarial Network (GAN), a Diffusion model, a Transformer model, a Large Language Model (LLM), a Visual Language Model (VLM), a Bidirectional Encoder Representations from Transformers (BERT), a Generative Pre-trained Transformer (GPT), or a Contrastive Language Image Pre-training (CLIP). The model may also be a model described in a rule-based manner that obtains an output result by referring to a predetermined table or making a judgment based on a predetermined condition. The above-mentioned models are not exclusive, and for example, LLM, VLM, BERT, and GPT are included in the Transformer model. For example, the Transformer model is included in the NN model. Furthermore, the learning algorithm and the model may be a combination of multiple types. The model also includes a multimodal model that is learned by combining multiple different types of data.

[0039] When the model control unit 101 receives the model input data D101, it outputs model output data D103 corresponding to the model input data D101 based on the model input data D101 and model information D102. When the model control unit 101 receives the model input data D101, it outputs the model output data D103 corresponding to the model input data D101, for example, by using a model indicated in the model information D102.

[0040] The model control unit 101 is realized by, for example, a CPU that operates according to a program included in the information processing device 10. Hereinafter, the learning model unit 100 may be referred to as an artificial intelligence unit. Hereinafter, the artificial intelligence unit refers to an AI having intelligent functions such as inference and judgment and its operating environment. Therefore, the model control unit 101 may include an AI having intelligent functions such as inference and judgment and its operating environment. The model control unit 101 may be, for example, an AI having a learning model as described above and its operating environment. The model control unit 101 may be one element (module) of the control unit 104 included in the information processing device 10.

[0041] 3, the learning model unit 100 may further include a reference information storage unit 12 (referred to as reference information DB12 in the figure) that stores model reference information D104. Here, the reference information storage unit 12 may be composed of multiple databases connected via a network. The same applies to other storage units (e.g., device information storage unit, etc.) described later.

[0042] The model reference information D104 is information that the model control unit 101 refers to in order to output model output data. The model reference information D104 may include a history of model input data previously input and / or a history of model output data previously output. The model reference information D104 may also include information that associates a feature amount included in a past input with a feature amount included in an output performed for that input. The model reference information D104 may also include information on an evaluation of a result output for a past input.

[0043] The model reference information D104 may also include information related to an expression or concept included in the model input data D101. The model reference information D104 may include, for example, information that associates a specific expression or concept that may be included in the model input data D101 with another expression or concept related to the expression or concept. Here, the other expression or concept related to a certain expression or concept includes an expression or concept that is more specific than the certain expression or concept, and another expression or concept that is evoked based on the certain expression or concept. The model reference information D104 may include, for example, information that associates a specific expression or concept that may be included in the model input data D101 with an expression or concept related to the certain expression or concept. As an example, the model reference information D104 may include, for example, information that associates a specific expression or concept that may be included in the model input data D101 with information related to the expression or concept. The model reference information D104 may include, for example, information that associates a search key extracted from an expression or concept that may be included in the model input data D101 with a value. The model reference information D104 may include information for so-called grounding. The model reference information D104 may also include a so-called knowledge graph that describes real-world entities and the relationships between them. In the knowledge graph, various pieces of information are systematically linked and expressed in a graph structure.

[0044] The model reference information D104 may also include information for so-called attention. For example, the model reference information D104 may include information indicating a correlation between an expression or concept that may be included in the model input data D101 and another expression or concept. The model reference information D104 may also include a feature map in which key information extracted from an expression or concept that may be included in the model output data D103 linked to an expression or concept that may be included in the model input data D101 is used as a feature. The model reference information D104 may also include information in which a query extracted from an expression or concept that may be included in the model input data D101 is associated with key information for search corresponding to the query.

[0045] In the example shown in FIG. 3, when the model control unit 101 receives model input data D101, it outputs model output data D103 based on the model input data D101, model information D102, and model reference information D104.

[0046] The learning model unit 100 may include a search engine that searches for the model reference information D104 or an interface with the search engine, instead of the reference information storage unit 12. In such a case, the search range of the search engine may be an external network or a specific network. Here, a database (e.g., a device information DB, etc.) provided in the control system of the present disclosure may be used as one of the external networks or specific networks.

[0047] The term "learning model" may refer to a computer algorithm that outputs some kind of information based on learned information in response to input information, or the learned information itself. However, when referring to a "learning model" in an operating environment, it often refers to an actual program that operates such a computer algorithm and the operating environment. In the present disclosure, the latter is adopted, and a model that actually operates based on information stored in the model information D102 is called a "learning model" in order to distinguish it from a simple algorithm or a group of learned information. The control system according to the present disclosure includes a learning model unit (particularly, the model control unit 101) that corresponds to such a learning model. Therefore, in the following description of the control system, when referring to a "learning model," it refers to the learning model unit or, in particular, the model control unit 101.

[0048] 4 is an explanatory diagram showing another configuration example of the learning model unit 100. As shown in FIG. 4, the learning model unit 100 may include an input unit 102, an output unit 103, and a control unit 104.

[0049] The input unit 102 accepts the model input data D101. The input unit 102 may accept the model input data D101 input by the user 1 or the like. The input unit 102 may accept the model input data D101 constituting the time-series data. At this time, the input unit 102 may sequentially accept the model input data D101 constituting the time-series data, or may accept the model input data D101 that has been buffered to a certain extent. Furthermore, the input unit 102 may accept the model input data D101 input from a plurality of input sources. At this time, the input unit 102 may accept the model input data D101 to which information of the input source (for example, a user identifier, user attribute information, etc.) is added, or the input unit 102 may determine the input source and add the information of the input source to the model input data D101 before accepting it, or may accept it without doing anything in particular. The input unit 102 is realized, for example, by various input devices (for example, a pointing device, a keyboard, a voice input device, an image input device, a data reading device, a data input device corresponding to various communication interfaces, etc.) provided in the information processing device 10. Note that the input unit 102 may be realized by an external device of the information processing device 10. In that case, it is sufficient for the information processing device 10 to include an interface with the input unit 102.

[0050] The output unit 103 outputs an object generated by the control unit 104. Here, the object includes the model output data D103 or data generated from the model output data D103. In addition, when the object generated by the control unit 104 includes information for a plurality of output destinations, the output unit 103 may output the object to a plurality of output destinations. In this case, the output unit 103 may output the same data to a plurality of output destinations, or may output different data for each output destination. The output unit 103 is realized, for example, by various output devices (for example, a display device, an audio output device, an image output device, a data writing device, a data output device corresponding to various communication interfaces, etc.) provided in the information processing device 10. Note that the output unit 103 may be realized by an external device of the information processing device 10. In that case, the information processing device 10 only needs to include an interface with the output unit 103.

[0051] The control unit 104 operates on the information processing device 10, and includes a pre-processing unit 105 and a post-processing unit 106 in addition to the above-mentioned model control unit 101.

[0052] The pre-processing unit 105 performs processing to improve the accuracy of the object generated by the control unit 104. The pre-processing unit 105 may, for example, add, change, or delete elements from the model input data D101, or convert (including process) the data.

[0053] For example, when the input unit 102 accepts model input data D101, the preprocessing unit 105 may change elements (including addition and deletion) or convert data (including processing) for the model input data D101. Changing elements or converting data (including processing) includes not only changing the data format but also changing the expression or concept represented by the data. The data changed by the preprocessing unit 105 is input to the model control unit 101 at the subsequent stage as the model input data D101. The processing performed by the preprocessing unit 105 includes so-called prompt shaping for the model control unit 101.

[0054] The pre-processing unit 105 may perform, for example, a process of decomposing the model input data D101 into predetermined unit data. The pre-processing unit 105 may also perform, for example, a process of integrating a plurality of model input data D101. Furthermore, the pre-processing unit 105 may change elements or convert data after decomposing the model input data D101 into predetermined unit data, or may integrate a plurality of model input data D101 and then change elements or convert data.

[0055] The post-processing unit 106 corrects the object when there is a problem with the object generated by the control unit 104 (particularly the model control unit 101), for example. The post-processing unit 106 may determine whether there is a problem with the object by using, for example, the above-mentioned knowledge graph. For example, the post-processing unit 106 may compare the similarity between the relationship shown by the knowledge graph and the relationship between the expression or concept included in the model input data and the expression or concept included in the model output data, and / or the relationship between the expressions or concepts included in the model output data, and determine that there is a problem with the object when it is apart from the relationship shown by the knowledge graph by a predetermined distance or more.

[0056] Among the above-mentioned components, components other than the model control unit 101 are not essential, and it is possible to select whether or not to implement them as appropriate.

[0057] Furthermore, the model information D102 and other information used by the learning model may be prepared in advance, or may be acquired via a communication network as necessary.

[0058] 5 is a configuration diagram showing another example of the information processing device 10 as an operating environment of the control unit 104 including the learning model unit 100. The information processing device 10 shown in FIG. 5 may include a control unit 104a including the learning model unit 100 (particularly the model control unit 101), an input processing unit 201, an output confirmation unit 202, and a correction confirmation unit 203.

[0059] The input processing unit 201 receives input information D11 from an input source 1a such as a user 1. The input processing unit 201 also outputs the received input information D11 to the learning model unit 100 as model input data D101.

[0060] At that time, the input processing unit 201 may output, for example, the input information D11 with the elements changed or the data converted, as the model input data D101. The input processing unit 201 may, for example, remove noise from the input information D11. In addition, when the input information D11 includes qualitative information, the input processing unit 201 may convert the information into quantitative information. In addition, when the input information D11 includes quantitative information, the input processing unit 201 may correct the amount according to the device that is the target of the request of the input information D11 and its operating environment. In addition, the input processing unit 201 may, for example, perform so-called grounding processing, that is, change the expression or concept indicated by the input information D11 into a more concrete expression or concept.

[0061] Furthermore, the input processing unit 201 may return an inquiry to the input source when the input information D11 contains unclear or uncertain information. The input processing unit 201 may output, as an inquiry, a message to confirm the input contents, a message proposing a correction proposal for the input information D11, a message requesting re-input of the input information D11 with a different state or expression, or the like. Furthermore, the correction proposal for the input information D11 may be generated by the correction confirmation unit 203 described later. Hereinafter, information indicating correction, addition, or cancellation of contents for the input / output data of the learning model after input / output may be referred to as supplementary information D18. The correction proposal is an example of supplementary information D18.

[0062] The output confirmation unit 202 performs a simulation simulating the control and state of the target device 2 based on the model output data D103 output from the learning model unit 100. The output confirmation unit 202 may perform a simulation after converting the model output data D103 into control information that matches a predetermined simulator (not shown) that can simulate the control and state of the target device 2. The output confirmation unit 202 may have a simulator function. When performing a simulation, the output confirmation unit 202 may use information acquired from an output destination 2a of the model output data D103. Here, the output destination 2a includes an output destination of information generated from the model output data D103. The information acquired from the output destination 2a may include, for example, state information D15 and / or feedback information D16, which will be described later.

[0063] The output confirmation unit 202 may, for example, confirm the state of the target device 2, the state of the system including the target device 2, and / or the state of the work of the target device 2. Furthermore, before performing the operation confirmation, the output confirmation unit 202 may generate and display a human-understandable intermediate product for the model output data D103 or information generated based on the model output data D103. Examples of the intermediate product include source code for a control program and an operation image of the controller of the target device 2 for an operation command to the target device 2. Furthermore, the output confirmation unit 202 may display the result of the simulation together with a reliability index of the learning model.

[0064] The following are examples of reliability indices for a learning model. For example, during pre-learning, etc., a human evaluation of the results of each input to the learning model may be accumulated, an evaluation network that has learned the input and the evaluation results may be provided, and when the learning model is used, the input to the learning model may also be input to the above-mentioned evaluation network, and the output result may be used as the reliability index.

[0065] In addition, for example, a learning device that clusters the output of the learning model during pre-learning, etc., may be provided, and when the learning model is used, the output of the learning model may also be input to the above-mentioned learning device, and the result of the clustering may be used as a reliability index.

[0066] In addition, for example, during pre-learning, etc., a human evaluation of the results of each input to the learning model can be accumulated, and an evaluation network can be provided that learns the features of inputs with high evaluation results.When the learning model is used, the input to the learning model can also be input to the above-mentioned evaluation network, and the similarity between the features that are the output result and the features of the learning result can be used as a reliability index.

[0067] In addition, for example, during pre-learning, etc., a learning device may be provided that accumulates the results of human evaluation of each input to the learning model, and clusters the inputs to the learning model with high evaluation results.When the learning model is used, the input to the learning model may also be input to the above-mentioned learning device, and the result of the clustering may be used as a reliability index.

[0068] The correction checking unit 203 judges the validity of the model output data D103 and / or the model input data D101 using the result of the simulation performed by the output checking unit 202. For example, the correction checking unit 203 may compare the state of the target device 2 shown in the simulation result with the state of the target device 2 specified by the input information D11, the model output data D103 and / or the model input data D101 to judge whether correct control is being performed, thereby judging the validity of the model output data D103 and / or the model input data D101. The correction checking unit 203 may judge that correct control is being performed when the state of the target device 2 shown in the simulation result matches the state of the target device 2 specified by the model output data D103 and / or the model input data D101. The state of the target device 2 to be compared here is not limited to one.

[0069] In addition, the correction confirmation unit 203 may determine the validity of the model output data D103 and / or the model input data D101 by, for example, confirming whether the state or control trajectory of the target device 2 shown in the simulation result matches the control shown in the input information D11, or whether it does not include any content that is prohibited in advance.

[0070] In addition, the correction confirmation unit 203 may present the simulation results to the input source 1a of the input information D11 and ask for a response as to whether the desired control is being performed, thereby judging the validity of the model output data D103 and / or the model input data D101.

[0071] When the correction confirmation unit 203 determines that the model output data D103 and / or the model input data D101 are incorrect, the correction confirmation unit 203 may correct the model input data D101. Instead of correcting the model input data D101, the correction confirmation unit 203 may generate supplemental information D18 for the input information D11 and output it to the input source 1a.

[0072] The control system 1000 may have, for example, any of the configurations shown in Fig. 1 to Fig. 5 as the operating environment of the learning model unit 100. In this case, as in the case of the learning model unit 100, a part or all of the configuration may be an internal configuration of the control system 1000 or an external configuration.

[0073] It should be noted that the above-mentioned learning model unit 100 and its peripheral configuration are merely examples, and not all of the components are essential, and implementation may be selected appropriately depending on the desired functions.

[0074] Fig. 6 is an explanatory diagram showing an example of model learning. As shown in Fig. 6, model information D102 may be generated by, for example, the model generating unit 107 performing machine learning using model learning data D105.

[0075] The model generation unit 107 is a processing unit that generates or updates the model information D102 based on the input model learning data D105 according to a predetermined algorithm. The model generation unit 107 is realized by, for example, a CPU that operates according to a program included in the information processing device 20. Here, the algorithm followed by the model generation unit 107 may be a machine learning algorithm corresponding to the learning model, such as supervised learning, reinforcement learning, or unsupervised learning, or may be deep learning, a genetic program, a functional logic program, or other known algorithms.

[0076] Furthermore, the model generation unit 107 may generate or update model information D102 for the input model learning data D105, further based on model reference information D104. Furthermore, the model generation unit 107 may generate or update model information D102 for the input model learning data D105, further based on model output data D103 from the model control unit 101.

[0077] The model learning data D105 is not particularly limited. For example, when supervised learning is used as a learning algorithm, the model learning data D105 may include candidates for model input data D101 that can be input and candidates for model output data D103 corresponding thereto. The model learning data D105 may also include the model input data D101 that is actually input and / or the model output data D103 that is actually output. Feedback control can be performed by appropriately using the actual model input data D101 and / or the model output data D103. The model learning data D105 may also include information acquired from a device or a processing unit included in a system in which the learning model actually operates.

[0078] The model information D102 generated or updated by the model generation unit 107 is stored in the model information storage unit 11, and is thereby provided to the model control unit 101. Alternatively, the model generation unit 107 can output the model information D102 directly to the model control unit 101.

[0079] The model generation unit 107 may generate the model information D102 by pre-learning using the input model learning data D105 before the model control unit 101 uses the model information D102, and store the model information D102 in the model information storage unit 11, for example.

[0080] The update of the model information D102 by the model generating unit 107 may be a process called FineTune.

[0081] The model generating unit 107 may be included in the control system 1000 or may be included in a system separate from the control system 1000.

[0082] In addition, in FIG. 1, the learning model unit 100, the device information storage unit 110, and the device information D13 are shown separately, but the device information storage unit 110 and the device information D13 may be part of the learning model unit 100. That is, the learning model unit 100 may include the device information storage unit 110 and the device information D13. For example, the learning model unit 100 may include the device information storage unit 110 as one of the reference information storage units 12 described later. In addition, the device information D13 may be used in a model learning phase in which the model used by the learning model unit 100 learns, so that the device information D13 is incorporated in the model in advance. In that case, the device information storage unit 110 may be omitted.

[0083] In addition, a part or all of the learning model unit 100 may be an internal configuration of the control system 1000, or may be an external configuration of the control system 1000. When it is an external configuration of the control system 1000, the control system 1000 may have an interface capable of exchanging information with an external system having a part or all of the learning model unit 100, instead of a part or all of the learning model unit 100. For example, the control system 1000 may have the model information storage unit 11, which is called the core of the learning model, as an external configuration. In addition, for example, the control system 1000 may have the model information storage unit 11, which is called the core of the learning model, and the model control unit 101, which is responsible for the algorithm of the model, as external configurations.

[0084] Hereinafter, in the control system 1000, in order to distinguish between the model control unit 101 that handles the algorithm of the learning model and a unit that performs processing to send a request to the model control unit 101 and obtain a response, the latter processing unit may be referred to as a "model processing unit". More specifically, the model processing unit corresponds to the above-mentioned information processing device 10, the control unit 104, or the control unit 104a, other than the model control unit 101. Note that, for example, when the model control unit 101 exists in an internal environment, the model processing unit may be realized by an OS (Operating System) or a prompt application (and a control unit that is considered to be the operating environment thereof) that runs on the information processing device 10 and calls the learning model application. Note that, for example, when the model control unit 101 exists in an external environment, the model processing unit may be realized by a browser or a client application (and a control unit that is considered to be the operating environment thereof) that runs on the information processing device 10.

[0085] The above-described learning model and the configuration of the information processing device serving as its operating environment, as well as the relationship between the learning model and the control system including the same, are similar in other embodiments.

[0086] In this embodiment, the input information D11 corresponds to the model input data D101. Moreover, the control description D12 corresponds to the model output data D103. For example, the learning model unit 100 (particularly, the model control unit 101) may be configured to output the control description D12 corresponding to the input information D11 based on the model information D102 and, if necessary, the model reference information D104 when receiving the input information D11.

[0087] In such a case, the model generation unit 107 provided corresponding to the learning model unit 100 may perform machine learning using, for example, model learning data D105 including candidates for input information D11 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D11 that may be input to the model control unit 101 and candidates for the corresponding control description D12, to generate or update the model information D102.

[0088] In this embodiment, the learning model unit 100 may be, for example, a language learning model such as LLM, which inputs a natural language and obtains an output result, and its operating environment. The learning model unit 100 may also be, for example, an image learning model such as VLM, which inputs an image and obtains an output result, and its operating environment. The learning model unit 100 may also be, for example, a multimodal model, which inputs a natural language and an image and obtains an output result, and its operating environment. In this case, the input information D11 may be input in the form of text data, image data, a combination of text data and image data, or a data format that can be converted into them (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0089] In this embodiment, the input information D11 received by the control system 1000 can be said to be information about a request (here, the control content required for the target device 2) in the work environment, here, the environment in which the target device 2 operates. Therefore, the input information D11 received by the control system 1000 can be said to be an example of first information indicating a request in the work environment. Also, the control description D12 and the execution code D14 can be said to be information used for the work (the work related to the control of the target device 2) corresponding to such input information D11. Hereinafter, the control description D12 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D11 is input may be referred to as second information.

[0090] Here, in the relationship between the input information D11 and the model input data, model input data based on the input information D11 may include the input information D11 itself, the input information D11 converted into a format that matches the input of the learning model, and the supplement of the input information D11. Also, in the relationship between the model output data and the second information, second information based on the model output data may include the model output data itself, the model output data converted into a format that matches the input of the output destination, and the supplement of the model output data. The same applies to the relationship between the input / output information and the model input / output data in the other embodiments.

[0091] Next, a description will be given of the operation of the control system 1000 according to this embodiment.

[0092] 7, first, the control system 1000 receives input information D11 (step S110). For example, the above-mentioned input unit 102 or the input processing unit 201 may receive the input information D11. The received input information D11 is input to the learning model unit 100 as model input data D101.

[0093] In step S110, the control system 1000 may receive a plurality of pieces of input information D11. The control system 1000 may also receive the input information D11 that better matches the desires of the user 1 interactively with the user 1, that is, by repeatedly inputting and outputting information related to the input information D11 between the control system 1000 and the user 1.

[0094] Next, the control system 1000 performs a process of generating a control description D12 using the learning model unit 100 (step S111). In step S111, the learning model unit 100 generates and outputs a control description D12 corresponding to the input information D11 that has been input. For example, the learning model unit 100 (more specifically, the model control unit 101) outputs the control description D12 corresponding to the input information D11 based on the model information D102, the input input information D11, and model reference information D104 including device information D13 as necessary. The learning model unit 100 may generate the control description D12 of text data from the input information D11 that has been input, for example, by using a learning model capable of generating text data.

[0095] In step S111, the learning model unit 100 (more specifically, the pre-processing unit 105 or the input processing unit 201) may further add, change or delete elements, or convert (including processing) data from the input information D11, in order to improve the accuracy of the control description D12, before the processing by the model control unit 101. Also, in step S111, the learning model unit 100 (more specifically, the post-processing unit 106) may further determine whether or not there is a problem with the control description D12 after the processing by the model control unit 101, and perform a process of correcting the control description D12 if it is determined that there is a problem.

[0096] The control description D12 output from the learning model unit 100 is input to the execution code generation unit 120. When the control description D12 is input, the execution code generation unit 120 generates the execution code D14 based on the input control description D12 (step S112).

[0097] Next, the execution code D14 generated by the execution code generating unit 120 is input to the target device 2 (step S113). As already described, the execution code D14 may be input to the target device 2 directly from the control system 1000 (more specifically, the execution code generating unit 120), or may be input indirectly via a communication network, another device (a server, various conversion devices, etc.), or manually.

[0098] As a result, the target device 2 operates in accordance with the input execution code D14.

[0099] If the state of the target device 2 is changed as a result of outputting the execution code D14, for example, by controlling the target device 2, the control system 1000 may acquire state information D15 (step S114). The acquired state information D15 is stored in the device information storage unit 110 as a part of the device information D13, for example. The control system 1000 may update the device information D13 stored in the device information storage unit 110 using the acquired state information D15, for example. In addition, the control system 1000 may output the acquired state information D15 to the user 1, the learning model unit 100, or another device (not shown) as information indicating the control result, for example. Note that, if the control system 1000 does not use the state information D15, the process of step S114 may be omitted.

[0100] The control system 1000 may repeat the processes of steps S110 to S114 multiple times (for example, until the desired control of the target device 2 is completed).

[0101] In addition, the control system 1000 may output the control description D12 to an operation terminal of the user 1, etc., so that the user 1 can confirm the contents and then execute subsequent processing (such as code generation in the execution code generation unit 120) through the operation of the user 1.

[0102] The state information D15 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on, for example, the input state information D15.

[0103] As described above, according to this embodiment, the execution code D14 can be generated from the input information D11 entered by the user 1 without the user 1 creating a control description D12, thereby improving the efficiency of the work of controlling the target device 2.

[0104] Furthermore, in this embodiment, the input information D11 may be text, an image, audio, or a combination thereof that explicitly or implicitly indicates the control content for the target device 2, thereby making it possible to further reduce the effort required for inputting the input information D11 while improving the efficiency of the work of controlling the target device 2.

[0105] Furthermore, according to this embodiment, since the control description D12 can be generated from the input information D11 by utilizing a learning model, even if the user 1 does not know information for controlling the target device 2 such as detailed specifications of the target device 2 or specifications of the control description D12, the control description D12 corresponding to the input information D11 can be generated, thereby improving the performance of the operation of controlling the target device 2. Here, improving the performance of the operation of controlling the target device 2 also includes high accuracy of control of the target device 2.

[0106] Furthermore, in this embodiment, the status information D15 acquired after controlling the target device 2 based on the input information D11 can be used for generating the next control description D12, etc., thereby making it possible to further improve the performance of the task of controlling the target device 2.

[0107] In the above example, only one target device 2 is shown, but there may be multiple target devices 2 controlled by the control system 1000. In such a case, for example, the input information D11 may include information capable of identifying the target device 2, the input side (input unit 102, preprocessing unit 105, input processing unit 201) to the learning model unit 100 may perform processing to identify the target device 2 based on the input information D11, or the learning model unit 100 may output control content in which the target device 2 has been identified as a result of learning.

[0108] Variation 1-1. Next, a modified example of the control system 1000 will be described. Fig. 8 is a configuration diagram showing an example of a control system 1000a which is a modified example of the control system 1000 according to the present embodiment. Note that the same elements as those in the control system 1000 are given the same reference numerals and description thereof will be omitted.

[0109] In the control system 1000a shown in FIG. 8, the output from the learning model unit 100 is confirmed by the user 1 and then input to the execution code generation unit 120 at the subsequent stage.

[0110] In this embodiment, the user 1 can check the control description D12 output from the learning model unit 100 and input the input information D11 based on the check result. The user 1 may check the feedback information D16 from the execution code generation unit 120 and / or the target device 2 in addition to the control description D12 output from the learning model unit 100 and input the input information D11 based on the check result. In this case, the user 1 may input the input information D11 of new contents, or may input the input information D11 indicating the correction, addition, or cancellation of the contents already input. At this time, the input information D11 may include a command for the learning model unit 100. For example, the user 1 may input, as the input information D11, a command for removing a defect included in the input information D11 or a defect included in the output control description D12 together with the feedback information D16. Here, the command for removing the defect includes an input for finding the cause of the defect and a solution to the defect.

[0111] The feedback information D16 may include a response to a request returned from a processing unit downstream of the learning model unit 100 when control is requested from the processing unit. The feedback information D16 may also include information obtained from a processing unit downstream of the learning model unit 100 after control is requested. For example, the feedback information D16 may include a response to the request returned from the execution code generation unit 120 when the control description D12 is input to the execution code generation unit 120 and a request is made to generate the control description D12. The feedback information D16 may also include a response to the request returned from the target device 2 when the execution code D14 is input to the target device 2 and a request is made to execute the code. The feedback information D16 may include state information D15. The feedback information D16 may be output directly to the user 1, or may be output to the user 1 via an output device (not shown) included in the execution code generation unit 120 or the control system 1000.

[0112] The feedback information D16 may also include information for determining whether or not the control requested of a processing unit downstream of the learning model unit 100 is executed correctly in the processing unit. The information is not limited to information obtained directly from the processing unit. For example, the information may be information obtained from another person, a device, a network, or an AI (none of which are shown). The feedback information D16 may include analysis information for determining whether or not the execution code D14 can execute the target control correctly, such as execution time or control trajectory information. For example, the user 1 may instruct the learning model unit 100 to control the timing of the flow in the control description D12 or adjust the lead time, based on such information included in the feedback information D16.

[0113] Also, the user 1 may use, for example, the feedback information D16 to exchange information with the learning model unit 100 multiple times, and each time determine the validity (presence or absence of a problem) of the output control description D12. When the user 1 determines that there is no problem with the control description D12, the user 1 may output the control description D12 to the execution code generation unit 120.

[0114] The feedback information D16 can be obtained, for example, in the above-mentioned step S114.

[0115] In addition, while Figure 8 shows an example in which user 1 inputs control description D12 to the execution code generation unit 120, the input of control description D12 to the execution code generation unit 120 can also be performed by the learning model unit 100 receiving instructions from user 1.

[0116] In this example, the control description D12 may include, for example, a description corresponding to low-code or no-code.

[0117] In this example, the exchange of information between user 1 and the learning model unit 100 may be performed, for example, via a terminal provided by user 1, or via a user interface (e.g., input unit 102) provided by the information processing device 10 on which the learning model unit 100 operates.

[0118] Further, the input information D11 in this example may be updated not by the user 1 but by the control system 1000 (for example, the correction confirmation unit 203, etc.).

[0119] Furthermore, the feedback information D16 may be input to the learning model unit 100. The feedback information D16 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on the input feedback information D16.

[0120] Other points may be the same as those of the other control systems according to the present embodiment.

[0121] As described above, in this modification, the user 1 can correct the input information D11 while checking the control description D12 output from the learning model unit 100 and exchanging additional instructions, bug consultations, and the like with the learning model unit 100, thereby improving the accuracy of the control description D12 to be output. As a result, the efficiency and performance of the task of controlling the target device 2 can be improved.

[0122] Variation 1-2. Next, a second modified example of the control system 1000 will be described. Fig. 9 is a configuration diagram showing an example of a control system 1000b which is a modified example of the control system 1000. Note that the same elements as those in the control systems 1000 and 1000a are given the same reference numerals and description thereof will be omitted.

[0123] The control system 1000b shown in FIG. 9 is different in that the learning model unit 100 returns a query D17 to the user 1. Examples of the query D17 include a query that questions unclear or uncertain input information D11, a query that questions a solution, and a query that requests re-input of a changed state or expression. The learning model unit 100 may output a query D17 to the user 1, which asks for more specific information along with a reference point, as a query for the unclear or uncertain input information D11. The learning model unit 100 may also output a query D17 to the user 1, which asks for a solution along with a reference point, and gives solution candidates as options. The learning model unit 100 may also output a query D17 to the user 1, which asks for a solution along with a reference point, and asks for the correctness of the most likely solution. In addition, the learning model unit 100 may first generate an intermediate control description, which is an intermediate control description that is easy for humans to understand, and then output the generated intermediate control description along with a query D17 asking whether it is correct or not to the user 1.

[0124] The output of the query D17 may be performed, for example, after the above-mentioned step S110.

[0125] When the learning model unit 100 receives a response from the user 1 to the query D17, the learning model unit 100 may update the input information D11 or determine the interpretation (meaning) of the input information D11.

[0126] The processing of the learning model unit 100 described above in this example can also be implemented, for example, as part of the function of the input unit 102 or pre-processing unit 105 of the learning model unit 100, or the input processing unit 201 (not shown) provided in the information processing device 10.

[0127] As described above, in this modification, the inquiry D17 is output to the user 1 in response to the input information D11, and the input information D11 is updated or its interpretation is confirmed based on the response, thereby eliminating uncertainty in the input information D11. As a result, the control description D12 to be output can be made more accurate, and the efficiency and performance of the operation of controlling the target device 2 can be improved.

[0128] Variation 1-3. Next, a third modified example of the control system 1000 will be described. Fig. 10 is a configuration diagram showing an example of a control system 1000c which is a modified example of the control system 1000. Note that the same elements as those in the control systems 1000, 1000a, and 1000b are given the same reference numerals and description thereof will be omitted.

[0129] The control system 1000c shown in Fig. 10 further includes a state acquisition unit 130. The state acquisition unit 130 acquires feedback information D16 indicating the processing result or state information D15 indicating the state of the device after the processing from the control description D12 output from the learning model unit 100 and the processing destination of the execution code D14 generated therefrom. Here, the feedback information D16 or the state information D15 can include information for determining whether the execution code D14 has correctly executed the target control, such as execution time or control trajectory information.

[0130] The state acquisition unit 130 may, for example, input the acquired information to the learning model unit 100. Also, the state acquisition unit 130 may, for example, update the device information D13 based on the acquired information. Also, the state acquisition unit 130 may, for example, generate information that supplements (including addition, correction, and cancellation) the input information D11 based on the acquired information, and input the information to the learning model unit 100 as supplement information D18. Also, the state acquisition unit 130 may, for example, generate information that supplements (including addition, correction, and cancellation) the control description D12 based on the acquired information, and input the information to the learning model unit 100 as supplement information D18.

[0131] The state acquisition unit 130 may generate, for example, a control command of new content or a command indicating addition, correction, or cancellation of content indicated in the already input input information D11 as supplementary information D18 and input it to the learning model unit 100. In addition, the state acquisition unit 130 may input, for example, a command for removing a defect included in the input information D11 or a defect included in the output control description D12 to the learning model unit 100 as supplementary information D18 together with the acquired information.

[0132] The status acquisition unit 130, for example, determines whether the acquired information indicates normal processing or a normal status at the processing destination, and if not, may input supplementary information D18 indicating correction, addition, or cancellation of the content indicated in the already input input information D11 together with the acquired information to the learning model unit 100.

[0133] The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on, for example, input information (state information D15, feedback information D16, supplementary information D18, etc.).

[0134] The supplemental information D18 may be generated, for example, in step S115 described above. The output destination of the supplemental information D18 may include a destination other than the learning model unit 100. For example, the control system 1000 may output the supplemental information D18 generated by the state acquisition unit 130 to the user 1 or another device (not shown).

[0135] Furthermore, the state acquisition unit 130 may acquire the operation result of a simulator (not shown) of the target device 2 or the operation result of the target device 2 in a debug mode without actually operating the target device 2. The debug mode of the target device 2 refers to a mode in which an execution code is executed on a control board of the target device 2 but actual device control is not performed and only the internal state is updated, and is also called an idle operation mode. By using the debug mode, the execution code D14 can be safely tried on the target device 2 in a state close to actual control.

[0136] As a method for determining the validity of the control description D12 output from the learning model unit 100 and the execution code D14 generated from the control description D12 without actually operating the target device 2, the execution code generation unit 120 may be connected to switch between the target device 2 and a simulator as the output destination of the execution code D14. The simulator includes one that operates an icon of the target device 2 in an augmented reality space. In addition, when outputting the execution code D14 to the target device 2, the execution code generation unit 120 may add information instructing whether to execute in normal mode or debug mode.

[0137] The processing of the state acquisition unit 130 in this example can also be implemented, for example, as part of the functions of the input unit 102, pre-processing unit 105 and post-processing unit 106 of the learning model unit 100, or the input processing unit 201, output confirmation unit 202 and correction confirmation unit 203 (none of which are shown in the figure) provided in the information processing device 10.

[0138] Other points may be the same as those of the other control systems according to the present embodiment.

[0139] As described above, in this modification, the state acquisition unit 130 acquires feedback information D16 indicating the processing result or state information D15 indicating the state of the device after processing from the target device 2 or the execution code generation unit 120 that is the output destination of the model output data D103 and / or information generated based on the model output data, for the input information D11 that has been input, and issues supplemental information D18 to the learning model unit 100 as appropriate based on the acquired information. This makes it possible to improve the accuracy of the control description D12, and in turn to improve the efficiency and performance of the task of controlling the target device 2.

[0140] In addition, in this modified example, for example, a person and a machine (state acquisition unit 130) can cooperate to improve the accuracy of input to the learning model unit 100, which can also contribute to reducing the workload of the user 1.

[0141] Embodiment 2 Next, a description will be given of a second embodiment of the present invention. In this embodiment, an example will be described in which a learning model is used to assist in an operation related to the control of a target device.

[0142] In the following, we consider the control of various control devices such as PLCs, processing machines, robots, sensors, transport devices, and other machine control devices in a factory. Experienced workers may be familiar with the control methods for a wide variety of control devices and complex control devices, but due to reassignments and other reasons, it may become necessary for less experienced workers to control control devices. In addition, when new control devices (including version upgrades) are introduced, it is necessary to inform all workers of the control methods corresponding to the new control devices, and insufficient awareness may lead to mistakes.

[0143] In such cases, it is preferable to be able to reliably perform the desired control even without knowing the specific control method, such as the control instructions, control signals, control codes, or commands to a controller corresponding to the control device, as this will lead to more efficient work and higher performance.

[0144] The situation in which devices are controlled is not limited to within a factory, and the situation in which the present embodiment is used is not limited to within a factory.

[0145] Fig. 11 is a configuration diagram illustrating an example of a control system 2000 according to the embodiment 2. The control system 2000 illustrated in Fig. 11 is a control system for controlling an appliance using a learning model, and includes a learning model unit 200 and an appliance information storage unit 210 (referred to as appliance information DB in the figure).

[0146] The learning model unit 200 outputs a control command D22 when input information D21 is input. For example, when input information D21 is input, the learning model unit 200 outputs a control command D22 based on model information D102. The configuration of the learning model unit 200 may be basically the same as that of the learning model unit 100 in the first embodiment.

[0147] In this embodiment, the learning model unit 200 is a model configured to output a control command D22 corresponding to the input information D21 when the input information D21 is input, and its operating environment. The learning model unit 200 may also be a model configured to generate and output a control command D22 based on the input information D21, device information D23, and other information that can be referred to in the learning model unit 200 when the input information D21 is input, and its operating environment.

[0148] In this embodiment, the input information D21 includes information indicating the control contents for the target device 2. The input information D21 may be, for example, text, image, voice, or a combination thereof indicating the control contents for the target device 2. The input information D21 may be, for example, text, image, voice, or a combination thereof indicating a plurality of control contents for the target device 2. The input information D21 may also include information indicating control contents that are performed successively in time, and in that case, may be time-series data of a predetermined data structure including text, image, voice, or a combination thereof indicating the control contents as described above. It is assumed that the way of indicating the control contents matches the input format of the model used by the learning model unit 200, but this is not limited to the case where error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 200.

[0149] The way of indicating the control contents in the input information D21 may be the same as that in the first embodiment, for example. For example, after specifying the control to be performed on the target device 2, the value of a parameter for performing the control or a state after the control may be specified. In this case, the input information D21 may include, for example, information for specifying the control and information indicating the value of a parameter for performing the control or the state after the control. Furthermore, the input information D21 may include not only information directly indicating the control contents on the target device 2, but also information indirectly indicating the control contents using operation contents corresponding to the control contents, the words and actions of the user 1, an image of the target device 2, or a similar control command in another model, etc.

[0150] The control command D22 includes information regarding the control of the target device 2, which is indicated in a predetermined format that enables identification of the target device 2 or an interface requesting control of the target device 2. The control command D22 may include information indicating a control request to the target device 2. The control command D22 is, for example, a control command, a control signal, or a control code for the target device 2. Furthermore, the control command D22 may be, for example, a command written in a format that can be handled by a predetermined controller corresponding to the target device 2.

[0151] The device information storage unit 210 stores device information D23, which is information related to the target device 2. The device information storage unit 210 and the device information D23 are basically handled in the same manner as the device information storage unit 110 and the device information D13 in the first embodiment. Note that the device information D23 in this embodiment may include, for example, information used to control the target device 2. The device information D23 is used, for example, as additional information when the learning model unit 200 outputs the control command D22. Hereinafter, in this embodiment, information particularly indicating the state of the target device 2 may be referred to as state information D25.

[0152] In this embodiment, the learning model unit 200 may be, for example, a language learning model such as LLM, which inputs a natural language and obtains an output result, and its operating environment. The learning model unit 200 may also be, for example, an image learning model such as VLM, which inputs an image and obtains an output result, and its operating environment. The learning model unit 200 may also be, for example, a multimodal model, which inputs a natural language and an image and obtains an output result, and its operating environment. In this case, the input information D21 may be input in the form of text data, image data, a combination of text data and image data, or a data format that can be converted into them (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0153] In this embodiment, in order to simplify the explanation, the components provided in correspondence with the learning model unit 200 may be explained using the same reference numerals of the components provided in correspondence with the learning model unit 100, but it should be noted that these are provided in correspondence with the learning model unit 200. This also applies to the other embodiments.

[0154] In this embodiment, the input information D21 corresponds to the model input data D101. The control command D22 corresponds to the model output data D103. The learning model unit 200 (particularly, the model control unit 101) may be configured to, for example, upon receiving the input information D21, output the control command D22 corresponding to the input information D21 based on the model information D102 and, if necessary, the model reference information D104.

[0155] In such a case, the model generation unit 107 provided corresponding to the learning model unit 200 may perform machine learning using, for example, model learning data D105 including candidates for input information D21 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D21 that may be input to the model control unit 101 and candidates for the corresponding control commands D22, to generate or update the model information D102.

[0156] Reference symbol D26 is feedback information indicating the control result in the target device 2. In this embodiment, too, the state information D25 and / or feedback information D26 may be acquired from the output destination of the model output data D103 and / or information generated based on the model output data. For example, the control system 2000 may output the acquired state information D25 and / or feedback information D26 to the user 1, the learning model unit 200, or another device (not shown) as information indicating the control result. The control system 2000 may also generate supplemental information D28 for the input / output data of the learning model unit 200 based on the acquired state information D25 and / or feedback information D26, and issue the supplemental information D28 to the user 1, the learning model unit 200, or another device (not shown). The control system 2000 may be configured to return a query D27 to the user 1 when the input information D21 includes unclear or uncertain information. The query D27 is handled in the same manner as the query D17 in the first embodiment.

[0157] Fig. 12 is a configuration diagram showing another example of the control system 2000. As shown in Fig. 12, the control system 2000 may further include a status acquisition unit 230 that acquires status information D25 and / or feedback information D26 and issues supplemental information D28. The status acquisition unit 230 is similar to the status acquisition unit 130 in the first embodiment.

[0158] In this embodiment, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device that can actually be controlled by receiving the control command D22, but this is not necessarily the case when a conversion device that converts various signals, such as a controller or a converter, is included between the target device 2 and the target device 2. In that case, the conversion device may receive the control command D22 and control the target device 2.

[0159] In this embodiment, the input information D21 received by the control system 2000 can be said to be information on a request (here, control content required for the target device 2) in the work environment, here the environment in which the target device 2 operates. Therefore, the input information D21 received by the control system 2000 can be said to be an example of first information indicating a request in the work environment. Also, the control command D22 can be said to be information used for the work (work related to control of the target device 2) corresponding to such input information D21. Hereinafter, the control command D22 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D21 is input may be referred to as second information.

[0160] Next, a description will be given of the operation of the control system 2000 of this embodiment.

[0161] 13, first, the control system 2000 accepts input information D21 (step S210). For example, the above-mentioned input unit 102 or the input processing unit 201 may accept the input information D21. The accepted input information D21 is input to the learning model unit 200 as model input data D101.

[0162] Next, the control system 2000 performs a process of generating a control command D22 using the learning model unit 200 (step S211). In step S211, the learning model unit 200 (more specifically, the model control unit 101) outputs a control command D22 corresponding to the input information D21 based on the model information D102, the input information D21 that has been input, and model reference information D104 including device information D23 as necessary.

[0163] In step S211, the learning model unit 200 may generate a control command D22 for binary data from the input information D21, for example, using a learning model capable of generating binary data. The learning model unit 200 may generate a control command D22 for text data from the input information D21, for example, using a learning model capable of generating text data. The learning model unit 200 may generate a control command D22 for image data from the input information D21, for example, using a learning model capable of generating image data. The learning model unit 200 may generate a control command D22 for voice data from the input information D21, for example, using a learning model capable of generating voice data.

[0164] In step S211, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 200 may further perform the above-mentioned processing.

[0165] The control command D22 output from the learning model unit 200 is input to, for example, the target device 2 (step S212). The control command D22 may be input to the target device 2 directly from the control system 2000 (more specifically, the learning model unit 200 or the information processing device 10 which is its operating environment), or may be input indirectly via a communication network or other devices (servers, various conversion devices, etc.).

[0166] As a result, the target device 2 operates in accordance with the input control command D22.

[0167] When the state of the target device 2 is changed due to the target device 2 being controlled as a result of outputting the control command D22, or when there is feedback from the target device 2, the control system 2000 may acquire state information D25 and feedback information D26 (step S213). Note that the process of step S213 is not essential and may be omitted as appropriate.

[0168] The control system 2000 may repeat the processes of steps S210 to S213 multiple times (for example, until the desired control of the target device 2 is completed).

[0169] As described above, according to this embodiment, even if the user 1 does not know the specific control method for the target device 2, the control command D22 can be generated from the input information D21 input by the user 1, and the target device 2 can be controlled based on the generated control command D22, thereby making it possible to improve the efficiency and sophistication of the work involved in controlling the target device 2.

[0170] Furthermore, according to this embodiment, even with ambiguous information, the device can be controlled to an appropriate state.

[0171] Embodiment 3 Next, a description will be given of a third embodiment of the present invention. In this embodiment, an example will be described in which a learning model is used to assist in a task related to operating a target device.

[0172] In the following, for example, we consider operating various devices in a home or building, such as air conditioners, refrigerators, televisions, lighting, washing machines, projectors, various sensors, and communication devices. In recent years, even these consumer devices have been provided with increasingly sophisticated functions, and control has become more complex. Although controllers such as operation screens and remote controllers have been designed to make complex control easier, it is still difficult to remember all the operations, and even if a desired function is provided, it may be difficult to reach that function.

[0173] Furthermore, even when the functions are the same, the names of the functions provided vary depending on the model, there are often differences in the detailed functions, and the control methods are also different. Therefore, when introducing a different model, such as by replacing the previous model, it is necessary to learn all of these differences from scratch, which is cumbersome.

[0174] Some devices automatically control themselves to an appropriate state by remembering past operating history and understanding the operating environment, but accurate control can be difficult in situations where multiple people gather and the appropriate state varies from person to person, or in situations where the appropriate state varies even for one person due to changes in physical condition, etc.

[0175] In such cases, it is preferable if the operator can easily perform operations to achieve the desired state, even if he or she does not know the specific operating method or does not understand what the appropriate state is, as this leads to improved work efficiency and performance.

[0176] The situation in which the device is operated is not limited to within a home or a building, and the situation in which the present embodiment is used is not limited to within a home or a building.

[0177] Fig. 14 is a configuration diagram showing an example of a control system 3000 according to the third embodiment. The control system 3000 shown in Fig. 14 is a control system for operating a device using a learning model, and includes a learning model unit 300, a device information storage unit 310 (referred to as device information DB in the figure), an input interface 311 (referred to as input IF in the figure), and an output interface 312 (referred to as output IF in the figure).

[0178] When input information D31 is input, the learning model unit 300 outputs an operation command D32. When input information D31 is input, for example, the learning model unit 300 outputs an operation command D32 based on model information D102. The configuration of the learning model unit 300 may be basically the same as that of the learning model unit 100 in the first embodiment.

[0179] In this embodiment, the learning model unit 300 is a model configured to output an operation command D32 corresponding to the input information D31 when the input information D31 is input, and its operating environment. The learning model unit 300 may also be a model configured to generate and output an operation command D32 based on the input information D31, the device information D33, and other information that can be referred to in the learning model unit 300 when the input information D31 is input, and its operating environment.

[0180] In this embodiment, the input information D31 includes information indicating operation contents requested to the target device 2. The input information D31 may be, for example, text, image, voice, or a combination thereof indicating operation contents for the target device 2. The input information D31 may be, for example, text, image, voice, or a combination thereof indicating a plurality of operation contents for the target device 2. The input information D31 may also include information indicating operation contents performed successively in time, and in that case, may be time-series data of a predetermined data structure including text, image, voice, or a combination thereof indicating the operation contents as described above. It is assumed that the way of indicating the operation contents matches the input format of the model used by the learning model unit 300, but this is not limited to the case where error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 300.

[0181] As an example of a way of indicating the operation contents in the input information D31, an operation to be performed on the target device 2 may be specified, and then a parameter value for performing the operation or a state after the operation may be specified. In this case, the input information D31 may include, for example, information for specifying the operation and information indicating the parameter value for performing the operation or the state after the operation. The parameter value for performing the operation may include, for example, values ​​related to the type of operation (ON / OFF, etc.), direction, amount, and time. Furthermore, the input information D31 may include not only information directly indicating the operation contents on the target device 2, but also information indirectly indicating the operation contents using control contents corresponding to the operation contents, the words and actions of the user 1, an image of the target device 2, or a similar operation command in another model.

[0182] The operation command D32 includes information on the operation of the target device 2, which is shown in a predetermined format that allows the target device 2 or an interface (including a person) requesting control of the target device 2 to be identified. The operation command D32 may include information indicating an operation request or a control request to the target device 2. The operation command D32 is, for example, an operation command, an operation signal, an operation code, a control command, a control signal, or a control code for the target device 2. The operation command D32 may also be, for example, a command written in a format that is handled by a predetermined controller corresponding to the target device 2. The operation command D32 can be said to be a concept in which information on the operation is added to the above-mentioned control command D22. The operation command D32 may also be information indicating a method of operating the target device 2, which is shown in a format that allows the person to be identified, when the interface is a person, that is, when control is requested of the target device 2 via a person.

[0183] The device information storage unit 310 stores device information D33, which is information related to the target device 2. The device information storage unit 310 and the device information D33 are basically handled in the same manner as the device information storage unit 110 and the device information D13 in the first embodiment. The device information D33 in this embodiment may include, for example, information used to operate the target device 2. The device information D33 may include, for example, information indicating the procedure of an operation actually performed on the target device 2 for the operation content. The device information D33 may also include, for example, a command, a signal, a code, etc. issued to the target device 2. The device information D33 is used, for example, as additional information when the learning model unit 300 outputs the operation command D32. Hereinafter, in this embodiment, information indicating the state of the target device 2 may be referred to as state information D35. For example, the device information D33 includes mechanical information such as a manual for the target device 2, and control code information such as communication specifications for a remote controller that operates the target device 2. The manual may include images.

[0184] The input interface 311 is an interface that receives input information D31 from the user 1 and inputs it to the learning model unit 300. The input interface 311 may be, for example, an interface that converts the input information D31 input from the user 1 into data that matches the input to the learning model unit 300 and outputs it. The input interface 311 may be provided, for example, as an example of the input unit 102 described above.

[0185] The output interface 312 is an interface that receives the operation command D32 from the learning model unit 300 and outputs it to a predetermined output destination. The output interface 312 may be provided, for example, as an example of the above-mentioned output unit 103. The output interface 312 may be, for example, an interface that converts the operation command D32 output from the learning model unit 300 into data that matches a predetermined output destination and outputs it. In this embodiment, the output destinations of the output interface 312 may include the target device 2, the controller 4 (not shown), a predetermined display 7 (not shown), an operation terminal (not shown) of the user 1, and a spatial projection display such as a hologram or projection, rather than a physical one.

[0186] In this embodiment, the learning model unit 300 may be, for example, a language learning model such as LLM, which inputs a natural language and obtains an output result, and its operating environment. The learning model unit 300 may also be, for example, an image learning model such as VLM, which inputs an image and obtains an output result, and its operating environment. The learning model unit 300 may also be, for example, a multimodal model, which inputs a natural language and an image and obtains an output result, and its operating environment. In this case, the input information D31 may be input in the form of text data, image data, a combination of text data and image data, or a data format that can be converted into them (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0187] In this embodiment, the input information D31 corresponds to the model input data D101. The operation command D32 corresponds to the model output data D103. The learning model unit 300 (particularly, the model control unit 101) may be configured to, for example, upon receiving the input information D31, output the operation command D32 corresponding to the input information D31 based on the model information D102 and, if necessary, the model reference information D104.

[0188] In such a case, the model generation unit 107 provided corresponding to the learning model unit 300 may perform machine learning using, for example, model learning data D105 including candidates for input information D31 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D31 that may be input to the model control unit 101 and candidates for the corresponding operation commands D32, to generate or update the model information D102.

[0189] Although not shown in the figure, in this embodiment, the state information D35 and / or feedback information D36 may be acquired from the model output data D103 of the learning model unit 300 and / or the output destination of information generated based on the model output data D103. For example, the control system 3000 may output the acquired state information D35 and / or feedback information D36 to the user 1, the learning model unit 300, or another device (not shown) as information indicating a response result. The control system 3000 may be configured to return a query D37 to the user 1 when the input information D31 includes unclear or uncertain information. The control system 3000 may also generate supplemental information D38 for the input / output data of the learning model unit 300 based on the acquired state information D35 and / or feedback information D36, and issue the supplemental information D38 to the user 1, the learning model unit 300, or another device (not shown). The handling of the state information D35, the feedback information D36, the query D37, and the supplemental information D38 may be basically the same as in the first embodiment.

[0190] The control system 3000 may further include a status acquisition unit 330 (not shown) that acquires the status information D35 and / or the feedback information D36 and issues supplemental information D38 as necessary. The status acquisition unit 330 is similar to the status acquisition unit 130 in the first embodiment.

[0191] In this embodiment, the target device 2 is not particularly limited. It is assumed that the target device 2 is a device that can receive the operation command D32 and control the device according to the operation content indicated by the operation command D32, but this is not necessarily the case when a conversion device that converts various signals, such as a controller 4 or a converter, or an operator is included between the target device 2 and the target device 2. In that case, the conversion device or the operator can receive the operation command D32 and operate the target device 2.

[0192] In this embodiment, the input information D31 received by the control system 3000 can be said to be information on a request (here, operation content requested to the target device) in the work environment, here, the environment in which the target device 2 operates. Alternatively, the input information D31 can be said to be an instruction on a request (here, operation content requested to the target device) in the work environment, here, the environment in which the target device 2 operates. Therefore, the input information D31 received by the control system 3000 can be said to be an example of first information indicating a request or instruction in the work environment. Also, the operation command D32 can be said to be information used for the work (work related to the operation of the target device 2) corresponding to such input information D31. Hereinafter, the operation command D32 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D31 is input may be referred to as second information.

[0193] Next, a description will be given of the operation of the control system 3000 of this embodiment.

[0194] 15, first, the input interface 311 of the control system 3000 receives input information D31 (step S310). The received input information D31 is input to the learning model unit 300 as model input data D101. The input information D31 is a command from the user 1 to the target device 2. The command may be composed of multiple operations on the target device 2. That is, the command may be composed of simple operations such as "raise the temperature by 1 degree Celsius," or may be a complex command requiring multiple operations on the target device 2.

[0195] Next, the control system 3000 performs a process of generating the operation command D32 using the learning model unit 300 (step S311). In step S311, the learning model unit 300 inputs the input information D31 (first information) to the learning model, and outputs an operation command D32 (second information) indicating a sequential control, which is a series of controls in the target device 2. The sequential control is a series of controls in the target device 2 for realizing a plurality of operations required to realize the command input as the input information D31. For example, the learning model unit 300 may output information indicating the sequential control by the control code of the target device 2 as the operation command D32 (second information) to the target device 2 via the output interface 312. Specifically, the following applies:

[0196] In step S311, the learning model unit 300 (more specifically, the model control unit 101) generates and outputs an operation command D32 corresponding to the input information D31 based on the model information D102 and the input information D31 that has been input, and model reference information D104 including device information D33 as necessary.

[0197] In step S311, the learning model unit 300 may generate an operation command D32 for binary data from the input information D31, for example, using a learning model capable of generating binary data. The learning model unit 300 may also generate an operation command D32 for text data from the input information D31, for example, using a learning model capable of generating text data. The learning model unit 300 may also generate an operation command D32 for image data from the input information D31, for example, using a learning model capable of generating image data. The learning model unit 300 may also generate an operation command D32 for voice data from the input information D31, for example, using a learning model capable of generating voice data.

[0198] In step S311, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 300 may further perform the above-mentioned processing.

[0199] The operation command D32 output from the learning model unit 300 is output to a predetermined output destination via the output interface 312, for example. Here, the predetermined output destination may be the target device 2, the controller 4, a predetermined display 7 (not shown), an operation terminal (not shown) of the user 1, or may not be a physical destination but a spatial projection display such as a hologram or projection. When the operation command D32 is input to the predetermined output destination, the target device 2 is operated according to the input operation command D32 (step S312).

[0200] For example, the output interface 312 may output the operation command D32 to the target device 2. In this case, the target device 2 that has received the operation command D32 (for example, an operation command, an operation signal, an operation code, a control command, a control signal, or a control code) may execute actual control according to the operation command D32. The output interface 312 may also output the operation command D32 to the controller 4 corresponding to the target device 2. In this case, the controller 4 that has received the operation command D32 (for example, indirect control information for the target device 2, such as a command for the controller 4, an operation command, an operation signal, or an operation code) may operate the target device 2 according to the operation command D32. The controller 4 may operate the target device 2 by outputting direct control information, such as a control code, to the target device 2 based on the control information indicated by the received operation command D32. Here, the controller 4 may be, for example, an operation panel provided on the target device 2, or a remote controller corresponding to the target device 2 that is directly operated by a user. The controller 4 includes a controller specific to the target device 2 and a general-purpose controller. Furthermore, the output interface 312 may output the operation command D32 to an operation terminal or a predetermined display device of the user 1. In this case, the operation terminal or display device of the user 1 that has received the operation command D32 (e.g., information indicating an operation method) displays the operation command D32. The user 1 may then operate the target device 2 or the controller 4 by referring to the displayed operation command D32.

[0201] The operation command D32 may be input to the output destination directly from the control system 3000 (more specifically, the learning model unit 300 or the information processing device 10 which is its operating environment), or may be input indirectly via a communication network or other equipment (servers, various conversion devices, etc.).

[0202] As a result, the target device 2 operates in accordance with the operation command D32.

[0203] As a result of the control system 3000 outputting the operation command D32, the target device 2 is operated, etc., and the state of the target device 2 changes. When there is feedback from the target device 2 whose state has changed, the control system 3000 may acquire state information D35 and feedback information D36 (step S313). Note that the process of step S313 is not essential and may be omitted as appropriate.

[0204] The control system 3000 may repeat the processes of steps S310 to S313 multiple times (for example, until a desired operation on the target device 2 is completed).

[0205] As described above, according to this embodiment, even if the user 1 does not know a specific operation method for the target device 2, the operation command D32 can be generated from the input information D31 input by the user 1, and the target device 2 can be operated based on the generated operation command D32, thereby making it possible to improve the efficiency and sophistication of the work involved in operating the target device 2.

[0206] Moreover, according to the present embodiment, even if the information is ambiguous, the device can be operated to an appropriate state. Moreover, according to the present embodiment, the device can be operated to an appropriate state without depending on the device and without learning how to operate the device.

[0207] In addition, even if an operation is simple for the user, a complex command requiring multiple operations may be input. An example of such an input command is, for example, an input command in natural language such as "make it natural with low humidity in the morning and cool it down in the afternoon." Another example is an input command using an image such as "an image showing a graph of the desired changes in room temperature and humidity." Even when a complex command requiring multiple operations is input, the learning model unit can output information indicating sequential control, which is a series of controls in the target device to realize the command. Thus, according to this embodiment, the efficiency or performance of the user's work can be improved.

[0208] Variation 3-1. Next, a modified example of the control system 3000 will be described. Fig. 16 is a configuration diagram showing an example of a control system 3000a which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0209] The control system 3000a shown in Fig. 16 further includes an input determination unit 31. Upon receiving input information D31, the input determination unit 31 is a means for analyzing the input information D31 and switching the control destination for the input information D31. In this modification, the input determination unit 31 switches the control destination for the input information D31 between the learning model unit 300 and the output interface 312 or the target device 2 via the output interface.

[0210] The input determination unit 31 uses a command rule to determine whether or not the command indicated by the input information D31 (first information) matches a command included in the command rule. A command rule is information that associates a command for a device with a control code to be sent to the device to realize the command. For example, a command for the target device 2 is associated with a control code for the target device 2 in a command rule. Note that, in a command rule, information in a format to be input as input information D31 is set for the command for the target device 2. In a command rule, information associated with a command for the target device 2, i.e., the control code for the target device 2, may be information in a format to be output as an operation command D32. For example, in a command rule, information associated with a command for the target device 2 may be a control signal, a parameter, or the like.

[0211] If the command indicated by the input information D31 matches a command included in the command rule, the input judgment unit 31 outputs a control code corresponding to the command indicated by the input information D31 to the target device 2 via the output interface 312. In addition, if the command indicated by the input information D31 does not match a command included in the command rule, the input judgment unit 31 outputs the input information D31 to the learning model unit 300.

[0212] Alternatively, the input determination unit 31 may determine whether or not a command for the target device 2 corresponds to a control code of the target device 2 by machine learning. Then, when the input determination unit 31 determines that the command indicated by the input information D31 corresponds to the control code of the target device 2, it outputs the control code corresponding to the command indicated by the input information D31 to the target device 2 via the output interface 312. On the other hand, when the input determination unit 31 determines that the command indicated by the input information D31 does not correspond to the control code of the target device 2, it outputs the input information D31 to the learning model unit 300. Further explanation is given below.

[0213] The input determination unit 31 may switch the control destination for the input information D31 depending on whether or not the input information D31 matches the command rule for the operation on the target device 2. When the input information D31 matches the command rule for the operation on the target device 2, the input determination unit 31 may input the input information D31 as it is to the output interface 312. On the other hand, when the input information D31 does not match the command rule for the operation on the target device 2, the input determination unit 31 may input the input information D31 to the learning model unit 300.

[0214] Whether or not the input matches the command rules of the operation may be determined, for example, by using a model described in a rule base. Here, the input determination unit 31 may be a learning model that is relatively light compared to the learning model unit 300. In other words, if the model used in the learning model unit 300 is a higher-level AI, the model used in the input determination unit 31 may be a lower-level AI that is lighter and has lower processing power. In this way, in the control system 3000a, the lower-level AI and the higher-level AI may work together to perform processing.

[0215] In the output interface 312, multiple possible outputs as the operation command D32 may be ranked according to their validity, and may be displayed in order from most valid to least valid, or more valid commands may be displayed in larger letters or with a larger area than less valid commands.

[0216] Hereinafter, in order to distinguish the information input at the output interface 312, the operation command D32 output from the learning model unit 300 may be referred to as the operation command D32a, and the input information D31 output to the output interface 312 may be referred to as the operation command D32b.

[0217] In this example, the output interface 312 may be any interface that receives the operation command D32a or D32b and outputs it to a predetermined output destination.

[0218] Next, the operation of the control system 3000a of this modified example will be described below. Fig. 17 is a flow chart showing an example of the operation of the control system 3000a.

[0219] 17, first, the input interface 311 of the control system 3000a receives the input information D31 (step S310). The received input information D31 is input to the input determination unit 31.

[0220] Next, the input determination unit 31 determines whether the input information D31 matches the command rule for the operation on the target device 2 (step S321). If it is determined that the input information D31 matches the command rule for the operation on the target device 2 (Yes in step S321), the input information D31 is input to the output interface 312 (proceed to step S322). On the other hand, if it is determined that the input information D31 does not match the command rule for the operation on the target device 2 (No in step S321), the input information D31 is input to the learning model unit 300 (proceed to step S311).

[0221] The processes in steps S311 to S313 are similar to those in the example shown in FIG.

[0222] In step S322, the output interface 312 outputs information such as a control code corresponding to the input information D31 in the command rule to a predetermined output destination as an operation command D32b, whereby the target device 2 operates in accordance with the operation command D32b.

[0223] Other points may be the same as those of the other control systems according to the present embodiment.

[0224] As described above, according to this modification, when the input from the user 1 matches the command rule for the operation of the target device 2, the target device 2 can be operated according to the input. On the other hand, when the input does not match, the learning model can be used to operate the target device 2. This makes it possible to further improve the efficiency of the work involved in operating the target device 2 and reduce costs.

[0225] Variation 3-2. Next, a description will be given of another modified example of the control system 3000. In this modified example, a learning model is used to generate an operation command including arbitration of multiple inputs.

[0226] 18 is a configuration diagram showing an example of a control system 3000b which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0227] In a control system 3000b shown in FIG. 18, an input interface 311 receives input information D31 from a plurality of users 1. For example, the input interface 311 accepts a plurality of commands from different users 1-1, 1-2, . . . , 1-n (n is a natural number equal to or greater than 2) as a plurality of pieces of input information D31 (first information). Then, the learning model unit 300 arbitrates the multiple commands indicated by the multiple pieces of input information D31. As a result of the arbitration, the learning model unit 300 outputs information indicating one sequential control for realizing the multiple commands as an operation command D32 (second information).

[0228] The input interface 311 accepts input information D31 from multiple users 1 and inputs it to the learning model unit 300. At this time, the input interface 311 may accept the input information D31 to which information about the user 1 who is the input source is attached, or the input interface 311 may identify the user 1 who is the input source and accept the input information D31 after attaching information about the input source, or may accept the input information without doing anything in particular.

[0229] The learning model unit 300 may be a model and its operating environment configured to output an operation command D32 corresponding to the input information D31 group when the input information D31 group received by the input interface 311 is input. The operation command D32 corresponding to the input information D31 group is information indicating one sequential control that realizes multiple commands indicated by multiple pieces of input information D31. The learning model unit 300 may be a model and its operating environment configured to generate and output the operation command D32 based on the input information D31 group, the device information D33, and other information that can be referred to in the learning model unit 300 when the input information D31 group is input.

[0230] Next, the operation of the control system 3000b of this modified example will be described below. Fig. 33 is a flowchart showing an example of the operation of the control system 3000b.

[0231] The input interface 311 receives a plurality of pieces of input information D31 from a plurality of users 1, and inputs the information to the learning model unit 300 (step S330). The learning model unit 300 arbitrates a plurality of commands indicated by a plurality of pieces of input information D31, and outputs information indicating one sequential control for realizing the plurality of commands as an operation command D32 (step S331). The processes in steps S312 and S313 are similar to those in the example shown in FIG.

[0232] For example, the learning model unit 300 may use a language learning model such as LLM, which inputs a natural language and obtains an output result, to perform processing to extract a suitable solution in a language space (more specifically, in a feature vector space having information on the language space), thereby generating and outputting an operation command D32 that is a compromise for different operation contents indicated by a group of input information D31. At this time, the learning model unit 300 may refer to the history of the input information D31 for each user 1 who is the input source and / or the history of the operation command D32 for each user 1 who is the input source.

[0233] Other points may be the same as those of the other control systems according to the present embodiment.

[0234] As described above, according to this modified example, even if information on different operation contents is input from multiple users, the learning model unit 300 can be used to reconcile the contents and generate a more appropriate operation command D32, thereby making it possible to further improve the functionality of the work related to operating the target device 2.

[0235] According to this modification, for example, it is possible to reconcile the needs of multiple users with different preferences for thermal environments and output control codes and parameters for suitable devices that are best or better for the multiple needs. When conflicting words appear in the learning model section, a sort of averaged output is output, making it possible to issue a command that makes use of both conflicting words. In this way, the control system according to this modified example can output more complicated sequential control as a result of arbitration. Specifically, it can output complicated sequential control such as "blow air only to the person who is saying it is hot (for example, on the right side), and on the left side, direct the airflow directly downward so that it does not blow on the person, and rather raise the room temperature by 0.5°C."

[0236] Variation 3-3. Next, a description will be given of another modified example of the control system 3000. In this modified example, a learning model is used to generate an operation screen user interface.

[0237] 19 is a configuration diagram showing an example of a control system 3000c which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0238] 19 further includes an operation screen user interface 3 (referred to as operation screen UI in the figure). In addition, the learning model unit 300 generates an operation screen for actually performing an operation of the target device 2 according to the operation content corresponding to the input information D31 as an operation command D32.

[0239] The operation screen generated by the learning model unit 300 may be, for example, a screen API having a function of receiving an operation input from a user together with an explanation of the operation content, and outputting a control command D34 such as a control code corresponding to the received operation input. API is an abbreviation of Application Programming Interface. Here, the output of a control code or the like corresponding to an operation input includes a mode in which a plurality of control commands D34 are sequentially output in response to one operation input. The operation screen may also be a screen API having an operation explanation corresponding to two or more different operation contents, an operation input acceptance, and a control command output. The learning model unit 300 may, for example, extract operation information indicating two or more different operation contents as an operation command D32 corresponding to the input information D31, and generate a screen API having an operation input acceptance and a control command output corresponding to each operation information.

[0240] In addition, the operation screen generated by the learning model unit 300 may be an operation screen whose display mode has been changed so that the operation points corresponding to the corresponding operation content are highlighted, the operation functions are displayed with restricted functions, or the position and form (shape, size, color, etc.) of UI components on the screen are changed and displayed.

[0241] The operation screen user interface 3 is an interface that displays an operation screen for the target device 2 and receives input related to user operation on the operation screen. The operation screen user interface 3 may be realized by, for example, a touch panel display, or a controller equipped with operation buttons and a display unit. The operation screen user interface 3 may also be realized by a display device such as a display that cooperates with an operation input device such as a mouse.

[0242] The operation screen user interface 3 may be provided in an equipment operation device that operates the target equipment 2. The operation screen user interface 3 may be an example of an equipment operation device that operates the target equipment 2. The equipment operation device may also include an input interface 311. In other words, the equipment operation device may be a remote controller through which the user 1 inputs a command to the target equipment 2. The appliance operation device may also have a function of assisting the user 1 in inputting commands to the target appliance 2 by displaying the sequential control approved by the user 1 on a display screen.

[0243] Further, the output interface 312 in this modification outputs the operation command D32 (operation screen) output from the learning model unit 300 to the operation screen user interface 3.

[0244] In addition, in this modification, the learning model unit 300 may have a function of interactively confirming the operation expected by the user 1. In such a case, when the learning model unit 300 receives information requesting reacquisition of the operation command D32 after presenting an operation screen as the operation command D32, for example, the learning model unit 300 may change a part of the input information, a part of the model parameters, or a reference destination of the reference information, and then reacquire the operation command D32.

[0245] Next, the operation of the control system 3000c of this modified example will be described below. Fig. 34 is a flowchart showing an example of the operation of the control system 3000c.

[0246] The process of step S310 is similar to step S310 in FIG. As described above, the appliance operating device may assist the user 1 in inputting commands to the target appliance 2 by displaying sequential controls approved by the user 1 in the past on the display screen.

[0247] In step S341, the learning model unit 300 generates an operation screen showing sequential control as an operation command D32 (second information) showing sequential control. An operation screen showing sequential control is an operation screen on which sequential control is displayed. As a specific example, it is assumed that the following sequential control is output as the operation command D32 in response to the input information D31 "Use natural conditions with low humidity in the morning and cool conditions in the afternoon." Examples of sequential control include "automatic operation in dry mode from 8:00AM to 12:00AM, and automatic operation at -2°C lower than the current state from 0:00PM to 5:00PM." The learning model unit 300 generates an operation screen for presenting such sequential control to the user 1.

[0248] In step S342, the learning model unit 300 displays an operation screen on the operation screen user interface 3 (device operation device). The learning model unit 300 allows the user 1 to select whether or not to approve the sequential control via the operation screen.

[0249] In step S343, it is determined in the operation screen user interface 3 (device operation device) whether or not the sequential control has been approved by the user 1. For example, an approval button or the like may be displayed on the operation screen, and it may be determined that the sequential control has been approved when the approval button is pressed. If sequential control is approved, processing proceeds to step S344. If sequential control is not granted, processing returns to step S310.

[0250] In step S344, the operation screen user interface 3 (device operation device) transmits to the target device 2 a control code for the target device 2 indicating the sequential control approved by the user 1. As a result, the target device 2 is operated.

[0251] The process of step S313 is similar to step S313 in FIG. In step S310, the appliance operating device may obtain information on the sequential control approved by the user 1 as feedback information D36. When the user 1 inputs a command, the feedback information D36 can be displayed on the display screen to assist the user 1 in inputting a command to the target appliance 2.

[0252] Next, a description will be given of another example of the operation of the control system 3000c of this modified example. Fig. 35 is a flow chart showing another example of the operation of the control system 3000c. The process of step S310 is similar to step S310 in FIG.

[0253] In step S351, the learning model unit 300 generates, as second information, an operation command D32 (operation screen) indicating a list of candidates for sequential control. In step S52, the learning model unit 300 displays an operation screen on the operation screen user interface 3 (device operation device). The learning model unit 300 allows the user 1 to sequentially select controls from the candidate list via the operation screen.

[0254] In step S353, it is determined whether or not the user 1 has selected a sequential control from the candidate list on the operation screen user interface 3 (device operation device). For example, a selection button may be displayed for each candidate in the candidate list, and when the selection button is pressed, it may be determined that the corresponding sequential control has been selected. Also, when the "Next" button is pressed without selecting anything, it may be determined that the user 1 does not approve any of the sequential controls in the candidate list. If sequential control is selected, processing proceeds to step S354. If no selection is made from sequential control, processing returns to step S310.

[0255] In step S354, the operation screen user interface 3 (device operation device) transmits a control code for the target device 2, which indicates the sequential control selected by the user 1, to the target device 2. As a result, the target device 2 is operated.

[0256] The process of step S313 is similar to step S313 in FIG.

[0257] As described above, in this modification, the learning model unit 300 can be used to generate an operation screen on which measures (construction of a screen API or change of display mode, etc.) have been taken to enable a desired operation to be performed simply or clearly, thereby improving the efficiency of the work involved in operating the target device 2. In addition, according to this modification, the user 1 can perform the actual operation while checking the explanation of the operation command generated by the learning model unit 300, and therefore the operation can be performed without mistakes. Moreover, according to this modification, the user 1 can confirm whether the sequential control operation command generated by the learning model unit 300 is really acceptable. Therefore, the state of the target device 2 can be brought closer to the state desired by the user 1.

[0258] For example, if various functions are arranged on the screen of a remote controller, the screen of the remote controller may become very cluttered. According to this modification, the operation screen of the remote controller can be simplified by limiting the functions that the user 1 desires. Furthermore, according to this modification, when the user 1 inputs the function he or she wants to perform by voice or text, the corresponding control menu or a candidate for a similar control menu can be displayed on the operation screen. This has the effect of allowing the user 1 to select an appropriate operation.

[0259] In this modified example, the candidate with the highest probability among several candidates displayed on the operation screen may be displayed largest, followed by smaller candidates in order to provide a display more suitable for the user 1. For sequential control, it is preferable to display a set of multiple operations all at once. For example, sequential control is a set of multiple operations such as "operate in cooling mode with room temperature at 28°C 2 hours after the timer is turned on, and turn off the power after 5 hours" or "initially operate in cooling mode with strong wind, but after 1 hour switch to dry mode with weak wind."

[0260] In addition, in this modification, an operation command with a high input frequency may be added as a candidate to the operation screen outputted in response to the input information based on the past operation history, the frequency of operation, etc. If the operation contents are stored on the device operation device (remote controller) side, it is possible to display the additional operation command without inquiring the learning model unit about the past operation history.

[0261] Variation 3-4. Next, a description will be given of another modified example of the control system 3000. In this modified example, the learning model further uses environmental information to generate an operation command.

[0262] 20 is a configuration diagram showing an example of a control system 3000d which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted.

[0263] A control system 3000d shown in FIG. 20 further includes an environment information storage unit 313 (referred to as environment information DB in the drawing).

[0264] The learning model unit 300 outputs, based on the environment information D33a representing the environment of the target device 2, an operation command D32 (second information) indicating a sequential control corresponding to a specific environment. The environmental information D33a includes, as the environment, at least one of the state of the target device 2, the state of the surroundings of the target device 2, and the state of the user 1 including the vital data of the user 1. The environmental information D33a may include all of the state of the target device 2, the state of the surroundings of the target device 2, and the state of the user 1 including the vital data of the user 1. Further explanation is provided below.

[0265] The environment information storage unit 313 stores environment information D33a, which is information on the environment in which the target device 2 operates. The environment information D33a may include information on the space in which the target device 2 operates. In this modification, information on objects or people present in the space in which the target device 2 operates and the user 1 who is the operator of the target device 2 are also considered to be part of the environment. Therefore, the environment information D33a may include information on the objects or people or the user 1.

[0266] The environment information D33a may include, for example, information about the person, such as the person's attributes, temperature, position, posture, and heart rate. The environment information D33a may also include, for example, information about the space, such as the location, temperature, humidity, and brightness. The environment information D33a may also hold information indicating a transition of information about such a space or person when the information about the space or person changes. Here, the information indicating the transition is also called time-series data or history information. The environment information D33a may be configured, for example, as a part of the model reference information D104 of the learning model.

[0267] The environmental information D33a may be acquired, for example, by a sensor (not shown).

[0268] The learning model unit 300 is a model and its operating environment that are configured, for example, when input information D31 is input, to generate and output an operation command D32 based on the input information D31, device information D33, environmental information D33a, and other information that can be referenced in the learning model unit 300.

[0269] As described above, according to this modified example, the learning model can generate the operation command D32 by also using the environmental information D33a related to the space in which the target device 2 is operating, thereby making it possible to further improve the functionality of the work related to operating the target device 2.

[0270] Variation 3-5. Next, another modified example of the control system 3000 will be described. In this modified example, two learning models are combined to generate an operation command. Fig. 21 is a configuration diagram showing an example of a control system 3000e which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and will not be described.

[0271] A control system 3000e shown in FIG. 21 includes a learning model unit 300a as the first learning model unit 300 and a learning model unit 300b as the second learning model unit 300, instead of the learning model unit 300 shown in FIG.

[0272] The learning model unit 300a outputs operation information D320 when input information D31 is input. The learning model unit 300a may be a model and its operating environment configured to generate and output operation information D320 based on at least the input information D31 and environmental information D33a when input information D31 is input.

[0273] Here, the operation information D320 includes information on the operation of the target device 2, which is shown in a predetermined format that can be discerned by the learning model unit 330b at the subsequent stage. Here, the operation information D320 may be information in which the operation content shown in the input information D31 is supplemented (including addition, correction, and cancellation) according to the environment information D33a. The operation information D320 may be information in which the operation content shown in the input information D31 or its expression is changed according to the situation of the space in which the target device 2 is driven. The learning model unit 300a may be a model that mainly performs grounding on the input information D31.

[0274] For example, even if the desired operation is the same, differences in linguistic expression and in how events are recognized may occur depending on the environment in which the target device 2 is operating. For example, the operation content implied by the input information D31 may differ depending on dialects, idiosyncratic expressions, the use of in-house or household terms, differences in perception of hot / cold, and the like.

[0275] The learning model unit 300a, for example, absorbs such differences in linguistic expressions and / or differences in perception of events and modifies the contents to be more generalized or specific. The learning model unit 300a may be a local learning model that obtains output results based on local information, such as a limited reference database.

[0276] The operation information D320 generated by the learning model unit 300a is input to the learning model unit 300b.

[0277] The learning model unit 300b may basically be the same as the above-mentioned learning model unit 300. However, instead of the input information D31, the operation information D320 generated by the learning model unit 300a is input.

[0278] When the operation information D320 is input, the learning model unit 300b outputs the operation command D32. The learning model unit 300b may be a model and its operating environment configured to generate and output the operation command D32 based on the operation information D320, the device information D33, and other information that can be referred to in the learning model unit 300b when the operation information D320 is input. The learning model unit 300b may be a global learning model that obtains an output result based on global information such as freely accessible to an external network.

[0279] Fig. 22 is a flowchart showing an operation example of this modified example. In the example shown in Fig. 22, when the input interface 311 of the control system 3000e receives the input information D31 in step S310, the input information D31 is input to the learning model unit 300a.

[0280] Next, the control system 3000e performs a process of generating operation information D320 using the learning model unit 300a (step S331). In step S331, the learning model unit 300a (more specifically, the model control unit 101) generates and outputs operation information D320 corresponding to the input information D31 based on the model information D102, the input information D31, and model reference information D104 including the environment information D33a as necessary. The operation information D320 output from the learning model unit 300a is input to the learning model unit 300b.

[0281] Next, the control system 3000e performs a process of generating the operation command D32 using the learning model unit 300b (step S332). In step S332, the learning model unit 300b (more specifically, the model control unit 101) generates and outputs the operation command D32 corresponding to the operation information D320 based on the model information D102, the input operation information D320, and the model reference information D104 including the device information D33 as necessary.

[0282] The subsequent processing may be similar to that of the other control systems according to this embodiment.

[0283] As described above, the learning model unit 300 includes the first learning model unit 300a and the second learning model unit 300b. The first learning model unit 300a inputs the input information D31 (first information) to the first learning model, and outputs the operation information D320 indicating the sequential control according to a specific environment based on the environmental information D33a. The second learning model unit 300a inputs the operation information D320 to the second learning model, and outputs the control code of the target device 2 for realizing the operation information D320 as the second information to the target device 2 via the output interface 312.

[0284] As described above, according to this modified example, the input information D31 input by the user 1 can be modified to more general or specific content by absorbing differences in linguistic expression and / or differences in perception of an event, and then an operation command D32 can be generated, thereby making it possible to further improve the functionality of the tasks related to operating the target device 2.

[0285] Incidentally, even in the configuration shown in the modified example 3-4, the learning model unit 300 can generate the operation command D32 that averages out the difference in linguistic expression and / or the difference in recognition of an event based on the model reference information D104 including the environment information D33a, the device information D33, and the past operation history, etc. However, according to this modified example, the role of the learning model can be clearly divided into absorbing the difference in expression and converting it into an operation command, so that the learning model can be specialized for learning and can be designed compactly, such as by suppressing the scale of learning.

[0286] Variation 3-6. Next, another modified example of the control system 3000 will be described. Fig. 36 is a configuration diagram showing an example of a control system 3000f which is a modified example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are given the same reference numerals and description thereof will be omitted. The control system 3000f includes a general-purpose operating device 4a and display format information 350 in addition to the components of the control system 3000.

[0287] The general-purpose operating device 4a is a general-purpose operating device capable of operating the target device 2 and other devices different from the target device 2. The general-purpose operating device 4a is capable of operating a plurality of different types of devices used by the user 1. For example, the general-purpose operating device 4a may be an operating device capable of operating each type of device, such as all air conditioners or all kitchen appliances used by the user 1. The general-purpose operating device 4a may also be an operating device for each area, such as an operating device capable of operating all devices installed in the living room, such as an air conditioner, air purifier, lighting, circulator, television, and audio equipment used by the user 1. The display format information 350 is information that stores the display format, that is, the format, to be displayed on the general-purpose operating device 4a.

[0288] The learning model unit 300 displays a display indicating the sequential control as the second information on the display screen of the general-purpose operating device 4a using the display format information 350. The learning model unit 300 may obtain approval or selection of the sequential control by the user 1 as described in FIG. 19 of the modified example 3-3 via the display screen of the general-purpose operating device 4a, and control the target device 2. The general-purpose operating device 4a in this modification may have the same functions as the device operating device described in modification 3-3. For example, the input interface 311 and the general-purpose operating device 4a may be the same device. If the general-purpose operating device 4a has, for example, a microphone and a communication means for making an inquiry to the learning model unit 300 (LLM·VLM), it can be configured as a remote controller device.

[0289] As described above, according to this modification, the user 1 can operate the equipment using a device that is familiar to him / her, and the efficiency of the work involved in operating the target equipment can be improved.

[0290] Variation 3-7. Next, a description will be given of another modified example of the control system 3000. In this modified example, the learning model further uses user history information to generate an operation command.

[0291] 37 is a configuration diagram showing an example of a control system 3000g which is a further modification of the control system 3000d according to the modification 3-4 of the present embodiment. Note that the same elements as those in the control system 3000d according to the modification 3-4 (FIG. 20) are denoted by the same reference numerals and will not be described.

[0292] A control system 3000g shown in FIG. 37 further includes a user historical information storage unit 360 (referred to as a user historical information DB in the drawing).

[0293] The learning model unit 300 outputs an operation command D32 (second information) indicating a sequential control according to the user 1, based on the user history information D361 including a history of commands given to the target device 2 by the user 1.

[0294] The user history information D361 is information in which, for example, a command input by the user 1, the state of the user 1 and the target device 2 at that time, the output control code of the target device 2, and changes in the environment are associated with each other. The learning model unit 300 further outputs, based on the user history information D361, an operation command D32 (second information) indicating sequential control according to the states of the user 1 and the target device 2. That is, based on the environment information D33a and the user history information D361, the learning model unit 300 outputs an operation command D32 (second information) indicating sequential control according to the state and preferences of the user 1 and the state of the target device 2.

[0295] The learning model unit 300 accumulates the user historical information D361 in the user historical information storage unit 360, for example, as follows. (1) When the input information D31 is input, the states of the user 1 and the target device 2 at that time are acquired based on the environmental information D33a. The input information D31 and the states of the user 1 and the target device 2 at that time are associated with each other and stored in the user history information storage unit 360. (2) An operation command D32 output from the learning model in response to the input information D31 is stored in the user history information storage unit 360 in association with the input information D31. (3) The states of the user 1 and the target device 2 after the target device 2 is operated by the operation command D32 are acquired and stored in the user history information storage unit 360 in association with the above-mentioned input information D31, etc. The states of the user 1 and the target device 2 after the target device 2 is operated may be acquired by a sensor of the target device 2, etc. Note that the above (1) to (3) are just an example of how the user history information D361 is generated, and the user history information D361 may be generated by other methods.

[0296] Next, the operation of the control system 3000g of this embodiment will be described below. Fig. 38 is a flowchart showing an example of the operation of the control system 3000g. The processes in step S310 and steps S312 to S313 are similar to those in the example shown in FIG.

[0297] In step S361, the learning model unit 300 outputs an operation command D32 (second information) indicating sequential control according to the state and preferences of the user 1, as well as the state of the target device 2 and the surrounding state, based on the environmental information D33a and the user history information D361.

[0298] For example, when the input information D31 is input, the learning model unit 300 generates and outputs an operation command D32 indicating sequential control based on the input information D31, the device information D33, the environment information D33a, and the user history information D361. As a result, the operation command D32 becomes appropriate according to the state of the target device 2, the state of the surroundings, and the state and preferences of the user 1.

[0299] The learning model unit 300 may also generate and output the operation command D32 based on other information that can be referenced, such as weather and external information related to the weather.

[0300] As described above, according to this modification, the learning model can generate the operation command D32 by using the environment information D33a and the user history information D361. Therefore, according to this modification, it is possible to output an appropriate operation command D32 according to the state and surrounding state of the target device 2, and the state and preference of the user 1. Therefore, it is possible to achieve higher accuracy and higher functionality in the work related to the operation of the target device 2.

[0301] Moreover, according to this modification, the learning model can learn the user's preferences for each environment by using the environmental information and the user history information. Therefore, according to this modification, a more comfortable situation can be provided for the user 1.

[0302] Variation 3-8. Next, another modified example of the control system 3000 will be described. In this modified example, two learning models are combined to generate an operation command. The control system 3000h in FIG. 39 is a modified example of the control system 3000e (FIG. 21) and the control system 3000g (FIG. 37) in the modified examples 3-5 and 3-7. Note that the same elements as those in the control system 3000e and the control system 3000g are denoted by the same reference numerals and will not be described.

[0303] In this modified example, the first learning model unit 300a inputs input information D31 into the first learning model, and outputs operation information D320 indicating sequential control according to the states of the user 1 and the target device 2 based on the environmental information D33a and the user history information D361. The first learning model unit 300b is similar to that of variant example 3-5.

[0304] FIG. 40 is a flowchart showing an operation example of this modified example. The processes in steps S310, S312, and S313 are similar to those in the example shown in FIG.

[0305] In step S371, the control system 3000h performs a generation process of the operation information D320 using the first learning model unit 300a. The first learning model unit 300a inputs the input information D31 to the first learning model, generates and outputs the operation information D320 corresponding to the input information D31. The first learning model is a model that outputs the operation information D320 corresponding to the input information D31 based on the environment information D33a, the user history information D361, and other external information as necessary. That is, the first learning model unit 300a is a model configured to output the operation information D320 according to the state and surrounding state of the target device 2, and the state and preference of the user 1, and its operating environment. The operation information D320 output from the first learning model unit 300a is input to the second learning model unit 300b.

[0306] In step S372, the control system 3000h performs a generation process of the operation command D32 using the second learning model unit 300b. The second learning model unit 300b inputs the operation information D320 to the second learning model, and generates and outputs the operation command D32 corresponding to the operation information D320. The second learning model is a model that outputs the operation command D32 corresponding to the operation information D320 based on the device information D33. That is, the second learning model unit 300b is a model configured to output the operation command D32 such as the control code of the target device 2 based on the device information D33, and its operating environment. The operation command D32 output from the second learning model unit 300b is output to the output interface 312.

[0307] As described above, according to this modification, it is possible to separate the first learning model that considers user-specific information such as the state of the user 1 and the target device 2 and the surrounding environment, from the second learning model that considers general-purpose information about the target device 2. Here, since the second learning model is generated from general-purpose information about the target device 2, it can be installed, for example, at the factory of the target device 2. On the other hand, the user-specific first learning model is a model that is updated according to the user's environment. A user-specific model is easy to customize because it can be realized as a compact model specialized for the user, whereas a device-generic model is large-scale because it is generic to all users. According to this modification, since the user-specific model and the device-generalized model are configured separately, the manufacturer does not need to prepare a model for each user environment. In addition, since it becomes easy to update only the user-specific model, the user is provided with a control system that is dedicated to him or her and that he or she can trust.

[0308] In the above-mentioned embodiment 3 and modifications 3-1 to 3-8, a plurality of parts may be combined and implemented. Alternatively, one part of the embodiment 3 and modifications 3-1 to 3-8 may be implemented. In addition, the embodiment 3 and modifications 3-1 to 3-8 may be combined in any manner, either in whole or in part, and implemented. In other words, any combination of embodiment 3 and variations 3-1 to 3-8 is possible, or any component of embodiment 3 and variations 3-1 to 3-8 is possible, or any component of embodiment 3 and variations 3-1 to 3-8 is possible to be modified, or any component of embodiment 3 and variations 3-1 to 3-8 is possible to be omitted.

[0309] Embodiment 4 Next, a fourth embodiment will be described. In this embodiment, an example will be described in which a learning model is used to support a task related to monitoring a certain task status.

[0310] For example, consider monitoring anomalies in a factory automation (FA) system that includes control devices such as robots and PLCs. For example, if there is an obvious installation error in the target work of the control device, existing monitoring algorithms based on rules, etc. can handle the error, but there may be cases where a slight installation error triggers the discovery of an anomaly in a later process. In such cases, even if an analysis is performed using the detection of an anomaly as a trigger, it is difficult to accurately grasp the situation and obtain a method of improvement.

[0311] In this embodiment, the aim is to improve the efficiency and performance of monitoring operations by supporting the work of monitoring work environments where cases may be anticipated in which a relatively small defect can spread and become a major anomaly, making it difficult to determine the cause because the circumstances surrounding the occurrence do not conform to existing rules.

[0312] Fig. 23 is a configuration diagram showing an example of a control system 4000 according to the fourth embodiment. The control system 4000 shown in Fig. 23 is a control system for monitoring a specific work situation by using a learning model, and includes a sensor 5, a learning model unit 400a, a learning model unit 400b, a device information storage unit 410 (referred to as device information DB in the figure), a model interface 6 (referred to as model IF in the figure), and a display 7.

[0313] The sensor 5 acquires data indicating the status of the work to be monitored. Hereinafter, the data acquired by the sensor 5 is referred to as sensor data. The sensor data may be, for example, image data obtained by capturing the state of the work to be monitored. The sensor data may also be, for example, audio data obtained by recording the state of the work to be monitored. The sensor data may also be, for example, measurement data obtained by measuring the state of the position of a person or object performing the work to be monitored.

[0314] It is assumed that the sensor data is always acquired by the sensor 5, but may be acquired based on a trigger provided by a person or another monitoring system. The sensor data acquired by the sensor 5 is input to the learning model unit 400a as input information D41. The sensor data itself, which is the input information D41, may be provided by a person or another monitoring system. In such a case, the sensor 5 may be omitted.

[0315] When input information D41 is input, the learning model unit 400a outputs an analysis result D42a. When input information D41 is input, for example, the learning model unit 400a outputs an analysis result D42a based on model information D102. The configuration of the learning model unit 400a may be basically the same as that of the learning model unit 100 in the first embodiment.

[0316] In this embodiment, the learning model unit 400a is a model and its operating environment configured to output an analysis result D42a corresponding to the input information D41 when the input information D41 is input. The learning model unit 400a may also be a model and its operating environment configured to generate and output an analysis result D42a based on the input information D41, the equipment information D43, and other information that can be referenced by the learning model unit 400a (such as model reference information D104) when the input information D41 is input. Here, the learning model unit 400a may refer to and use information on the work to be monitored as the model reference information D104. The information on the work to be monitored may be information indicating, for example, the position, person, object, procedure, conditions, etc., where the work is performed. The learning model unit 400a may use, for example, a manual that describes the conditions, installation environment, operation procedures, etc. of the equipment used in the work, as the model reference information D104.

[0317] In this embodiment, the input information D41 includes information indicating the status of the work to be monitored. Here, the work to be monitored includes one or more works performed by a person or a machine. The input information D41 may be, for example, a measurement value, an image, a sound, or a combination thereof indicating the status of the work to be monitored. The input information D41 may be, for example, a measurement value, an image, a sound, or a combination thereof indicating the status of a plurality of works to be monitored. The input information D41 may also include information indicating the status of work performed continuously in time, and in that case, may be time-series data of a predetermined data structure including a measurement value, an image, a sound, or a combination thereof indicating the above-mentioned status. It is assumed that the way of indicating the work status matches the input format of the model used by the learning model unit 400a, but this is not limited to the case where error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 400a.

[0318] The analysis result D42a includes information indicating a situation analysis result obtained by analyzing the work situation indicated by the input information D41. The information indicating the situation analysis result may be information indicating an object (environment) existing in the work situation indicated by the input information D41 and / or an event occurring. The information indicating the situation analysis result may be information indicating an interpretation of the work situation indicated by the input information D41. The analysis result D42a may be, for example, text indicating an interpretation of the work situation indicated by the input information D41. The analysis result D42a may also be, for example, text indicating an interpretation of a part of the work situation indicated by the input information D41 that is different from the normal situation, focusing on the part. The analysis result D42a may be in a format other than text. The analysis result D42a may be in any format as long as it is described in a predetermined format that can be discerned by the learning model unit 400b at the subsequent stage, and may be in a format of, for example, text, image, sound, or a combination of these.

[0319] Examples of interpreting a work situation include expressing objects present in the work situation using their attributes, expressing events occurring in the work situation in a predetermined syntax format such as 5W1H or 7W1H, or summarizing them after making them concrete expressions. Other examples include breaking down the work being performed in the work situation into multiple viewpoints, interpreting it, and expressing it for each viewpoint, and when the work being performed in the work situation includes multiple small tasks or steps, breaking down the target work into small task units or step units and explaining each small task or step. The analysis result D42a can be said to be the result of concretizing, subdividing, and / or extracting singular points from the work situation shown in the input information D11, and further expressing it in a predetermined format. In this way, the analysis result D42a expresses the work situation in an easy-to-understand and organized state.

[0320] When the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b. When the analysis result D42a is input, for example, the learning model unit 400b outputs the analysis result D42b based on the model information D102. The configuration of the learning model unit 400b may be basically the same as that of the learning model unit 100 in the first embodiment.

[0321] In this embodiment, the learning model unit 400b is a model and its operating environment configured to output an analysis result D42b corresponding to the analysis result D42a when the analysis result D42a is input. Also, the learning model unit 400b may be a model and its operating environment configured to generate and output the analysis result D42b based on the analysis result D42a, the device information D43, and / or information that can be referenced in the learning model unit (such as model reference information D104) when the analysis result D42a is input.

[0322] The analysis result D42b includes information indicating a method for improving the work situation derived from the analysis result of the work situation by the learning model unit 400a. The information indicating a method for improving the work situation may be information indicating a recovery method for recovering an abnormal state to a normal state, or may be information indicating a method for solving a problem when some problem occurs in the environment (work environment) where the work to be monitored is performed, such as a person having trouble or a device being stopped.

[0323] The information indicating the improvement method may be, for example, text, image, or sound indicating the method, or may be a control command (for example, a command, a control signal, a control code, etc.) for a device (target device 2) on which the method is to be implemented, a procedure manual describing the method, a sequence diagram, a source code, an execution code, or a controller command for causing a controller to execute the method. The information indicating the improvement method may be, for example, text, image, sound, data described in a predetermined design language, a control description (including source code and information described in a predetermined programming platform language), information described in other platform languages, a control command (including a control command, a control signal, a control code, and a controller command), an execution code, or a combination of two or more elements thereof indicating the method. An example of the predetermined design language is UML (Unified Modeling Language), but is not limited to this.

[0324] In the following, the learning model unit 400a may be referred to as the first learning model unit 400, and the analysis result D42a may be referred to as the first analysis result D42. Also, the learning model unit 400b may be referred to as the second learning model unit 400, and the analysis result D42b may be referred to as the second analysis result D42.

[0325] As already described, the analysis result D42a includes information indicating the situation analysis result of the work situation indicated by the input information D41. Therefore, the learning model unit 400b may be a model configured to output the analysis result D42b corresponding to the situation analysis result indicated by the analysis result D42a, and its operating environment. Here, when the information indicating the situation analysis result is a text explaining the work situation indicated by the input information D41, the learning model unit 400b may be a model configured to output the analysis result D42b corresponding to the text explaining the work situation, and its operating environment.

[0326] The device information storage unit 410 and the device information D43 are basically handled in the same manner as the device information storage unit 110 and the device information D13 in the first embodiment. In this embodiment, the device information storage unit 410 stores device information D43, which is information on devices related to the work to be monitored as the target device 2. Here, the devices related to the work include devices that are widely required for the above-mentioned situation analysis and derivation of the improvement method. More specifically, the devices include not only devices used in the work but also devices that affect the person or device performing the work. More specifically, the devices that affect the person or device performing the work may be devices that directly or indirectly bring about changes to the person or device performing the work. Examples include devices directly used in the work (including various machines such as processing machines and conveyors, and tools such as workbenches and tools), devices that control the devices directly used in the work (power supplies, relays, switches, controllers, etc.), and devices that bring about changes in the work environment (lighting equipment, air conditioning equipment, vacuum cleaners, purifiers, etc.).

[0327] The device information D43 is used as additional information when the learning model unit 400a and / or the learning model unit 400b outputs the model output data D103 (the analysis result D42a, the analysis result D42b), for example. Hereinafter, in this embodiment, information indicating the state of the target device 2 may be referred to as state information D45.

[0328] In this embodiment, the learning model unit 400a may be an image learning model such as VLM, which inputs an image and obtains an output result, and its operating environment. The learning model unit 400a may also be, for example, a multimodal model, which inputs a natural language and an image and obtains an output result, and its operating environment. The learning model unit 400b may also be, for example, a language learning model such as LLM, which inputs a natural language and obtains an output result, and its operating environment. In this case, the input information D41 may be input in the form of text data, image data, a combination of text data and image data, or a data format that can be converted into them (such as audio data or a video that is a combination of audio data and image data). The learning model used in this embodiment is not limited to the above-mentioned models.

[0329] The model interface 6 is an interface that receives model output data (analysis results D42a and D42b) from the learning model unit 400a and the learning model unit 400b and outputs the data to a predetermined output destination. The model interface 6 may be, for example, an interface that converts the model output data output from the learning model unit 400a and the learning model unit 400b into data that matches a predetermined output destination and outputs the data. The model interface 6 may be provided, for example, as an example of the output unit 103 described above. In this embodiment, the output destinations of the model interface 6 include the target device 2 and the display 7.

[0330] The model interface 6 may, for example, output result information D44a indicating the situation analysis result included in the analysis result D42a and the improvement method included in the analysis result D42b to the display device 7, and output result information D44b indicating the improvement method included in the analysis result D42b to the target device 2. At this time, the model interface 6 may extract some data from the analysis result D42a and / or the analysis result D42b, convert it into a data format matching the output destination, and output it as result information D44a and result information D44b.

[0331] 23, the target device 2 and the display 7 are shown as output destinations of the model interface 6, but the output destinations of the model output data are not limited to the above. For example, when a method for improving the situation indicated by the model output data to be output includes information indicating control over the target device 2, the model interface 6 can output the model output data or information indicating the method directly to the target device 2 as the implementation destination of the method, or can output the model output data or information indicating the method to a conversion device (not shown) that converts the model output data or information indicating the method into information acceptable to the target device 2. The conversion device may be, for example, the control system 1000 of the first embodiment that converts input information into a control description or execution code that can be discerned by the target device 2.

[0332] In addition, the model interface 6 itself may have a function of a conversion device. For example, the model interface 6 may have a function not only to control the output of the model output data, but also to convert the improvement method output by the learning model unit 400b into a code executable by an interpreter, and output the converted code or control an equipment based on the code. In addition, the model interface 6 may have a function to control the process flow, such as immediately executing a process that has a high urgency when the improvement method includes the process. In addition, the model interface 6 may have a function to transmit a prompt input via the display of the display unit 7, such as a response to a proposed method displayed on the display unit 7, to the learning model unit 400b.

[0333] The model interface 6 may also have the functions of the output confirmation unit 202 and the correction confirmation unit 203 described above. For example, the model interface 6 may determine the urgency of the analyzed situation, and when it is determined that the urgency is not high, it may confirm the appropriateness of the improvement method by inquiring of a supervisor or using a simulator or the like, and if the improvement method is not appropriate, it may communicate this to the learning model unit 400b and prompt it to output the improvement method again. At that time, the model interface 6 may issue supplementary information D48 for the model input data of the target learning model unit.

[0334] In this embodiment, the input information D41 corresponds to the model input data D101 of the learning model unit 400a. The analysis result D42a corresponds to the model output data D103 of the learning model unit 400a. The analysis result D42a corresponds to the model input data D101 of the learning model unit 400b. The analysis result D42b corresponds to the model output data D103 of the learning model unit 400b. The learning model unit 400a (particularly, the model control unit 101) may be configured to, for example, upon receiving the input information D41, output the analysis result D42a corresponding to the input information D41 based on the model information D102 and, if necessary, the model reference information D104. The learning model unit 400b (particularly, the model control unit 101) may be configured to, for example, upon receiving the analysis result D42a, output the analysis result D42b corresponding to the analysis result D42a based on the model information D102 and, if necessary, the model reference information D104.

[0335] In such a case, the model generation unit 107 provided in correspondence with the learning model unit 400a may perform machine learning using model learning data D105 including candidates for input information D41 that may be input to the model control unit 101 to generate or update the model information D102, or may perform machine learning using model learning data D105 including candidates for input information D41 that may be input to the model control unit 101 and candidates for the corresponding analysis results D42a to generate or update the model information D102. In addition, the model generation unit 107 provided in correspondence with the learning model unit 400b may perform machine learning using model learning data D105 including candidates for the analysis results D42a that may be input to the model control unit 101 to generate or update the model information D102, or may perform machine learning using model learning data D105 including candidates for the analysis results D42a that may be input to the model control unit 101 and candidates for the corresponding analysis results D42b to generate or update the model information D102.

[0336] Although not shown in the figure, in this embodiment, the state information D45 and / or feedback information D46 may be acquired from the output destination of the model output data D103 of the learning model unit 400a and the learning model unit 400b and / or information generated based on the model output data. For example, the control system 4000 may output the acquired state information D45 and / or feedback information D46 to the user, the learning model unit 400a, the learning model unit 400b, or another device not shown, as information indicating the control result. In addition, the control system 4000 may be configured to return a query D47 to the user when the input information D41 includes unclear or uncertain information. In addition, the control system 4000 can generate supplementary information D48 for the input / output data of the learning model unit 400a and the learning model unit 400b based on the acquired state information D45 and / or feedback information D46, and issue the supplementary information D48 to the user, the learning model unit 400a, the learning model unit 400b, or another device not shown. The state information D45, the feedback information D46, the inquiry D47, and the supplemental information D48 may be handled basically in the same manner as in embodiment 1. Here, the information may be output to the user via, for example, the display device 7 or an input / output interface (not shown) included in the information processing device 10.

[0337] The control system 4000 may further include a status acquisition unit 430 (not shown) that acquires the status information D45 and / or the feedback information D46 and issues supplemental information D48 as necessary. The status acquisition unit 430 is similar to the status acquisition unit 130 in the first embodiment.

[0338] In the present embodiment, there is no particular limitation on the target device 2. Note that, although it is assumed that the target device 2 is a device that can actually be controlled by receiving the analysis result D42b, this is not the case when the above-mentioned conversion device is included between the target device 2 and the target device 2.

[0339] In this embodiment, the input information D41 received by the control system 4000 can be said to be information on the situation in the work environment (here, the situation in the environment where the monitoring work is performed). Therefore, the input information D41 received by the control system 4000 can be said to be an example of first information indicating the situation in the work environment. Also, the analysis result D42a and the analysis result D42b can be said to be information used for the work (monitoring work) corresponding to such input information D41. Hereinafter, the analysis result D42a and / or the analysis result D42b output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D41 is input may be referred to as second information.

[0340] Next, a description will be given of the operation of the control system 4000 of this embodiment.

[0341] 24, first, the control system 4000 accepts input information D41 (step S410). For example, the above-mentioned input unit 102 or the input processing unit 201 may accept the input information D41. The accepted input information D41 is input to the learning model unit 400a as model input data D101.

[0342] Next, the control system 4000 performs a process of generating an analysis result D42a using the learning model unit 400a (step S411). In step S411, the learning model unit 400a (more specifically, the model control unit 101) outputs an analysis result D42a corresponding to the input information D41 based on the model information D102, the input information D41, and model reference information D104 including device information D43 as necessary. The learning model unit 400a may generate an analysis result D42a of text data from the input information D41 using, for example, a learning model capable of generating text data.

[0343] In step S411, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 400a may further perform the above-mentioned processing.

[0344] The analysis result D42a output from the learning model unit 400a is input to the learning model unit 400b. The analysis result D42a output from the learning model unit 400a is also input to the learning model unit 400b and the model interface 6. The analysis result D42a output from the learning model unit 400a may be input to the model interface 6 via the learning model unit 400b. In this case, the learning model unit 400b may output model output data D103 including the analysis result D42a and the analysis result D42b.

[0345] Next, the control system 4000 performs a process of generating the analysis result D42b using the learning model unit 400b (step S412). In step S412, the learning model unit 400b (more specifically, the model control unit 101) outputs the analysis result D42b corresponding to the analysis result D42a based on the model information D102 and the input analysis result D42a, and model reference information D104 including device information D43 as necessary. The learning model unit 400b may generate the analysis result D42b of binary data from the input analysis result D42a using, for example, a learning model capable of generating text data. In addition, the learning model unit 400b may generate the analysis result D42b of text data and binary data from the input analysis result D42a using, for example, a learning model capable of generating text data and binary data.

[0346] In step S412, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 400b may further perform the above-mentioned processing.

[0347] The analysis result D42b output from the learning model unit 400b is input to the model interface 6, for example.

[0348] The model interface 6 controls the target device 2 and / or displays information on the display 7 based on the analysis results by the learning model unit 400a and the learning model unit 400b (step S413). In step S413, for example, the model interface 6 outputs information based on the analysis results D42a and D42b to a predetermined output destination. The model interface 6 outputs, for example, result information D44a indicating the situation analysis result and an improvement method to the display 7 based on the analysis results D42a and D42b, and outputs result information D44b indicating the improvement method based on the analysis results D42b to the target device 2.

[0349] The result information D44a may be, for example, text and audio, indicating the situation that occurred in the work environment and how to improve it, while the result information D44b may be, for example, text or a control signal, indicating how to improve it.

[0350] As a result, the display 7 displays the situation analysis result indicated by the analysis result D42a and the improvement method indicated by the analysis result D42b based on the result information D44a, and the target device 2 implements the improvement method indicated by the analysis result D42b based on the result information D44b. Information input to the display 7 and the target device 2 may be directly input from the control system 4000 (more specifically, the model interface 6), or may be indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or manually.

[0351] When the state of the target device 2 is changed due to the target device 2 being controlled or when there is feedback from the output destination, the control system 4000 may acquire state information D45 and feedback information D46 (step S414). Note that the process of step S414 is not essential and may be omitted as appropriate.

[0352] The control system 4000 may repeat the processes of steps S410 to S414 multiple times (for example, until a desired state is reached in the target work environment).

[0353] As described above, according to this embodiment, understanding of the situation and acquisition of improvement methods are performed in two stages using different learning models, so that the accuracy of the final product can be improved, and as a result, the efficiency of work related to monitoring the work situation can be improved.

[0354] For example, when grasping the situation, it is important to broadly detect abnormal conditions in the work environment, such as "something strange has occurred."On the other hand, when obtaining improvement methods, specific information such as "stop this machine, move the work position to point A, return the machine to state B, and then restart it" is important.

[0355] When the information to be extracted, i.e., the target information, has different levels of abstraction, there is a concern that the accuracy of the output results will decrease if one learning model is used to learn and extract them together. In particular, when acquiring improvement methods, it is necessary to present specific methods based on knowledge and information of the work environment. In such cases, output accuracy can be improved more reliably by separating the learning models and providing appropriate domain knowledge (environmental information).

[0356] In addition, when trying to obtain solutions to different tasks, grasping the situation and acquiring an improvement method, using one learning model, the problem of hallucination is likely to become prominent. This is because a function of adjusting the solution of one task (grasping the situation) so that the solution of the other task (acquiring an improvement method) appears more plausible may work implicitly in the model algorithm. According to this embodiment, it is effective against such a hallucination problem. That is, by separating the learning models corresponding to the two tasks of grasping the situation and acquiring an improvement method, the modalities entering each learning model can be suppressed, and as a result, the magnitude of hallucination can be suppressed, thereby improving the accuracy of the final product.

[0357] Furthermore, in this embodiment, the analysis results D42a and D42b, which are the output results of the learning model units 400a and 400b, can be verbalized and displayed on the display unit 7, so that a person can check the contents to suppress hallucination and more reliably realize a method for improving the situation.

[0358] The control system 4000 of this embodiment can be applied not only to the monitoring of the control system of the devices in the factory described above, but also to the monitoring of distribution objects in a distribution system, for example.

[0359] Variation 4-1. Next, a modified example of the control system 4000 will be described. Fig. 25 is a configuration diagram showing an example of a control system 4000a which is a modified example of the control system 4000 according to the present embodiment. Note that the same elements as those in the control system 4000 are given the same reference numerals and the description thereof will be omitted.

[0360] A control system 4000a shown in FIG. 25 differs from control system 4000 in that it has two analysis means for analyzing and improving a situation using different methods, and switches between the analysis means to be used as appropriate depending on the situation that has occurred.

[0361] The control system 4000a shown in Figure 25 includes a first analysis unit 41-1 that uses the above-mentioned learning model unit 400a and learning model unit 400b to analyze the situation and obtain improvement methods, and also includes a second analysis unit 41-2, a switching unit 42, and an output switching switch 43.

[0362] The second analysis unit 41-2 may be any means for analyzing the situation and acquiring an improvement method for the input information D41 in a manner different from that of the first analysis unit 41-1. As an example, the second analysis unit 41-2 may be a means for analyzing the situation and acquiring an improvement method on a rule basis. For example, when the input information D41 is input, the second analysis unit 41-2 may determine whether the input information D41 matches a predetermined abnormality pattern, and if the input information D41 matches any of the abnormality patterns, may acquire an improvement method according to the abnormality pattern. The second analysis unit 41-2 outputs an analysis result D42c including at least a method for improving the situation.

[0363] The analysis result D42c may include, for example, information corresponding to the result information D44a and information corresponding to the result information D44b described above. In this modification, the analysis result D42c includes at least the result information D44b indicating the improvement method obtained by the second analysis unit 41-2.

[0364] In this modification, the second analysis unit 41-2 may be realized as an internal execution module by being implemented in, for example, a PLC, an information processing device, or the like arranged in the work environment.

[0365] The switching unit 42 is a means for switching the control destination for the input information D41 according to a predetermined condition. In this modification, the switching unit 42 switches the control destination for the input information D41 between the first analysis unit 41-1 and the second analysis unit 41-2. The switching unit 42 may switch the control destination for the input information D41, for example, depending on whether the input information D41 matches an existing rule. At this time, the switching unit 42 may switch the control destination by switching the output destination of the input information D41 to the second analysis unit 41-2 when the input information D41 matches the existing rule, and switching the output destination of the input information D41 to the first analysis unit 41-1 when the input information D41 does not match the existing rule.

[0366] The switching unit 42 may switch the control destination for the input information D41 according to, for example, an instruction from a supervisor. The switching unit 42 may also switch the control destination for the input information D41 according to, for example, time, work content, or the presence or absence of a supervisor. The switching unit 42 may also switch the control destination for the input information D41 according to, for example, whether an abnormality occurs in the work environment. Here, the presence or absence of an abnormality in the work environment may be determined according to, for example, whether an abnormality signal is generated. For example, the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1 when an abnormality occurs. The switching unit 42 may also switch the control destination for the input information D41 according to, for example, the degree or urgency of the abnormality occurring in the work environment.

[0367] In addition, the switching unit 42 may control an output switching switch 43 that switches a connection path (such as a circuit or communication path) connecting the output of the first analysis unit 41-1 or the output of the second analysis unit 41-2 to the target device 2 and display 7 to which the analysis result is to be output, in response to switching of the control destination for the input information D41.

[0368] For example, when the switching unit 42 switches the control destination for the input information D41 to the first analysis unit 41-1, the switching unit 42 may control the output changeover switch 43 to turn on the connection path connecting the output of the first analysis unit 41-1 with the target device 2 and the display 7, and turn off the connection path connecting the output of the second analysis unit 41-2 with the target device 2 and the display 7. Similarly, when the switching unit 42 switches the control destination for the input information D41 to the second analysis unit 41-2, the switching unit 42 may control the output changeover switch 43 to turn on the connection path connecting the output of the second analysis unit 41-2 with the target device 2 and the display 7, and turn off the connection path connecting the output of the first analysis unit 41-1 with the target device 2 and the display 7.

[0369] Fig. 26 is a flowchart showing an operation example of this modified example. In the example shown in Fig. 26, when the control system 4000a accepts the input information D41 in step S410, the switching unit 42 switches the control destination for the input information D41 according to a predetermined condition (step S421). In the example shown in Fig. 26, the switching unit 42 judges whether the input information D41 matches an existing rule, and if it is judged that it does not match (No in step S421), it proceeds to a first analysis process (step S422). On the other hand, if it is judged that the input information D41 matches the existing rule (Yes in step S421), it proceeds to a second analysis process (step S423).

[0370] In the first analysis process in step S422, the learning model unit 400a and the learning model unit 400b as the first analysis unit 41-1 analyze the situation and obtain an improvement method. As a result of the first analysis process, the learning model unit 400a and the learning model unit 400b output an analysis result D42a including an analysis result of the situation and an analysis result D42b including an improvement method of the situation.

[0371] In the second analysis process in step S423, the second analysis unit 41-2 analyzes the situation and obtains an improvement method according to the existing rules. The second analysis unit 41-2 outputs, for example, an analysis result D42c including at least a method for improving the situation as a result of the first analysis process.

[0372] When the result of the analysis process by the first analysis unit 41-1 or the second analysis unit 41-2 is output, the target device 2 is controlled and / or information is displayed on the display 7 based on the result of either analysis process, depending on the state of the output changeover switch 43 (step S424).

[0373] In this example, when the first analysis unit 41-1 performs the analysis process, the connection path connecting the output of the first analysis unit 41-1 to the target device 2 and the display 7 is turned on. In that case, the model interface 6 may output, for example, result information D44a indicating the situation analysis result and the improvement method to the display 7 based on the analysis result D42a and the analysis result D42b, and may output result information D44b indicating the improvement method based on the analysis result D42b to the target device 2. On the other hand, when the second analysis unit 41-2 performs the analysis process, the connection path connecting the output of the second analysis unit 41-2 to the target device 2 and the display 7 is turned on. In that case, based on the analysis result D42c output from the second analysis unit 41-2, result information D44a indicating the situation analysis result and the improvement method may be output to the display 7, and / or result information D44b indicating the improvement method may be output to the target device 2.

[0374] The display 7 may display the result information D44b in a manner that can be confirmed by the worker, for example. In this case, the worker may refer to the result information D44b displayed on the display 7, confirm the improvement method indicated by the result information D44b, and perform the work related to the method. The worker may also confirm the improvement method indicated by the result information D44b and determine its appropriateness. At this time, if the improvement method indicated by the result information D44b is not appropriate, the worker may prompt the learning model unit 400b to acquire another improvement method (reacquire the model output data). For example, when the learning model unit 400b receives information requesting reacquisition of the model output data, the learning model unit 400b may reacquire the model output data after changing a part of the input information, a part of the model parameters, or the reference destination of the reference information.

[0375] The subsequent processing may be similar to that of the other control systems according to this embodiment.

[0376] As described above, this modified example is configured to have a plurality of analysis units that analyze the situation and obtain improvement methods using different methods, and to switch between them depending on the situation. This enables control that is more suited to the situation. For example, for a problem with a clear cause, the second analysis unit with a high processing load can instantly analyze the situation and present and execute an improvement method, and for a problem with an unclear cause, the first analysis unit using a learning model can analyze a complex situation and present and execute a better improvement method.

[0377] In the above example, the first analysis unit 41-1 analyzes the situation and acquires an improvement method by using two learning models, but the configuration of the first analysis unit 41-1 is not limited to the above example. For example, if the situation does not need to be analyzed, the learning model unit 400a can be omitted. Also, if the improvement method does not need to be acquired, the learning model unit 400b can be omitted. Also, the situation can be analyzed and the improvement method can be acquired by one learning model unit.

[0378] For example, in the case where the first analysis unit 41-1 includes a learning model unit 400b that acquires a method for improving the situation based on the input information D41, the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1 when an abnormality occurs. In this case, the learning model unit 400b of the first analysis unit 41-1 may be configured to output information indicating an improvement method corresponding to the occurrence situation of the abnormality indicated by the input information D41 when the input information D41 is input. At this time, the learning model unit 400b may refer to the device information storage unit 410 accessible by the control system and output information indicating an improvement method corresponding to the situation.

[0379] Embodiment 5. Next, the fifth embodiment will be described. In this embodiment, an example will be described in which a learning model is used to support a response task of returning a response to information sent by a user in a call center, a product site, etc. Here, the information sent by a user may include an inquiry or an opinion regarding a certain service, information, event, or object.

[0380] Fig. 27 is a configuration diagram showing an example of a control system 5000 according to the fifth embodiment. The control system 5000 shown in Fig. 27 includes a learning model unit 500, a reference information storage unit 12, a database search unit 511 (referred to as DB search unit in the figure), a control generation unit 512, a voice recognition unit 513v, and a voice synthesis unit 514v. Here, the reference information storage unit 12, the database search unit 511, and the control generation unit 512 may be included as part of the learning model unit 500.

[0381] When input information D51 is input, the learning model unit 500 outputs response information D52 indicating the response content. For example, when input information D51 is input, the learning model unit 500 outputs response information D52 based on model information D102. The configuration of the learning model unit 500 may be basically the same as that of the learning model unit 100 in the first embodiment.

[0382] In this embodiment, the learning model unit 500 is a model configured to output response information D52 corresponding to the input information D51 when the input information D51 is input, and its operating environment. The learning model unit 500 may also be a model configured to generate and output response information D52 based on the input information D51 and other information that can be referred to in the learning model unit 500 when the input information D51 is input, and its operating environment.

[0383] In this embodiment, the input information D51 includes information indicating the contents transmitted from the user 1, etc. The input information D51 may include information indicating the contents for which a reaction is required in the working environment. The input information D51 may be, for example, a text, an image, a voice, or a combination thereof indicating an inquiry or opinion regarding a certain service, information, event, or thing. The input information D51 may be, for example, a text, an image, a voice, or a combination thereof indicating a plurality of inquiries or opinions regarding a certain service, information, event, or thing. The input information D51 may also include information indicating the transmitted contents that are continuous in time, and in that case, may be time-series data of a predetermined data structure including the text, image, voice, or a combination thereof indicating the transmitted contents as described above. It is assumed that the way of indicating the transmitted contents matches the input format of the model used by the learning model unit 500, but this is not limited to the case where an error processing, correction processing, or conversion processing is provided in the front stage of the learning model unit 500.

[0384] The response information D52 includes information indicating a response to the transmission content included in the input information D51. The response information D52 may be, for example, information indicating a response to an inquiry or opinion regarding a service, information, event, or thing indicated by the transmission content included in the input information D51.

[0385] The reference information storage unit 12 stores model reference information D104 that the model control unit 101 of the learning model unit 500 refers to in order to output the response information D52. The model reference information D104 includes, for example, information related to a service, information, event, or object that may be included in the input information D51. Here, the reference information storage unit 12 may particularly store information related to a specific service, information, event, or object as the model reference information D104. The model reference information D104 may include, for example, a response manual that has been digitized. In addition, the model reference information D104 may include, for example, the history of the input information D51 input in the past or the transmission content included therein. At this time, the reference information storage unit 12 may store, as the model reference information D104, history information indicating the input information D51 input in the past or the transmission content included therein, together with information on the user 1 of the transmission source (for example, a user identifier, user attribute information, etc.). Hereinafter, in this embodiment, information indicating the state of the user 1 of the transmission source may be referred to as state information D55.

[0386] The database search unit 511 is a search engine for the reference information storage unit 12 and other databases. In response to a request from the model control unit 101 of the learning model unit 500, the database search unit 511 searches a database that is connected so that the database search unit 511 can access it, and outputs a search result. At this time, the database that the database search unit 511 accesses may be limited.

[0387] The control generation unit 512 is an interface for setting preconditions when the learning model unit 500 (particularly, the model control unit 101) generates model output data. The control generation unit 512 may be, for example, an interface used to recognize information to be controlled by the learning model unit 500 and / or to set the tendency of output. Here, the control target information is information indicating a target to be focused on in the model control unit 101. The model control unit 101 may be configured to generate model output data D103 from the model input data D101 based on the control target information indicated by the control generation unit 512, for example. The control generation unit 512 may, for example, recognize a part of the model input data input by the user as the control target information, recognize information generated by the model control unit 101 as the control target information, or recognize information generated by the model control unit 101 and modified by another control unit as the control target information. The control target information and / or the tendency setting of the output may be designated by the user, may be designated by an external processing unit, or may be designated by the control generation unit 512 according to a predetermined algorithm.

[0388] When the input from the user 1 includes the input information D51v in a voice format, the voice recognition unit 513v recognizes the voice indicated by the input information D51v, converts it into a format that matches the data format of the learning model unit 500, and outputs it. The voice recognition unit 513v may convert the input information D51v in a voice format into the input information D51 in a text format, for example.

[0389] The voice synthesis unit 514v converts the content indicated by the response information D52 into a voice format and outputs it. For example, when the response information D52 output from the learning model unit 500 includes a data format other than voice, the voice synthesis unit 514v converts the content of the part indicated by the response information D52 into a voice format and outputs it. For example, when the response information D52 is a data structure including a designation of a data format, the voice synthesis unit 514v may convert a data element for which a voice format is designated in the designation into a voice format and output it. For example, the voice synthesis unit 514v may convert the response information D52 in a text format into response information D52v in a voice format.

[0390] In the above example, data in voice format is used for input / output with the user 1, but the data format used for input / output with the user 1 is not limited to voice format. In that case, instead of the voice recognition unit 513v and the voice synthesis unit 514v, a processing unit that converts the data format used for input from the user 1 into a data format used for input to the learning model unit 500 and a processing unit that converts the data format used for output from the learning model unit 500 into a data format used for input to the user 1 may be provided.

[0391] Furthermore, if the learning model unit 500 can accept the data format used for input from the user 1, the voice recognition unit 513v can be omitted. Furthermore, if the user 1 can accept the data format used for output from the learning model unit 500, the voice synthesis unit 514v can be omitted.

[0392] In this embodiment, the input information D51 corresponds to the model input data D101. The response information D52 corresponds to the model output data D103. The learning model unit 500 (particularly, the model control unit 101) may be configured to, for example, upon receiving the input information D51, output the response information D52 corresponding to the input information D51 based on the model information D102 and, if necessary, the model reference information D104.

[0393] In such a case, the model generation unit 107 provided corresponding to the learning model unit 500 may perform machine learning using, for example, model learning data D105 including candidates for input information D51 that may be input to the model control unit 101, to generate or update the model information D102. In addition, the model generation unit 107 may perform machine learning using, for example, model learning data D105 including candidates for input information D51 that may be input to the model control unit 101 and candidates for response information D52 corresponding thereto, to generate or update the model information D102.

[0394] Although not shown in the figure, in this embodiment, the state information D55 and / or feedback information D56 may be acquired from the model output data D103 of the learning model unit 500 and / or the output destination of information generated based on the model output data D103. The control system 5000 may output the acquired state information D55 and / or feedback information D56 to a predetermined supervisor, the learning model unit 500, or another device not shown as information indicating a response result. The control system 5000 may be configured to return a query D57 to the user 1 when the input information D51 includes unclear or uncertain information. The control system 5000 may also generate supplemental information D58 for the input / output data of the learning model unit 500 based on the acquired state information D55 and / or feedback information D56, and issue the supplemental information D58 to the user 1, a predetermined supervisor, the learning model unit 500, or another device not shown. The handling of the state information D55, the feedback information D56, the query D57, and the supplemental information D58 may be basically the same as in the first embodiment.

[0395] The control system 5000 may further include a status acquisition unit 530 (not shown) that acquires the status information D55 and / or the feedback information D56 and issues supplemental information D58 as necessary. The status acquisition unit 530 is similar to the status acquisition unit 130 in the first embodiment.

[0396] In this embodiment, the input information D51 received by the control system 5000 can be said to be information on a request in the work environment (here, the transmitted content that requests a reaction called a response in an environment where a response task is performed to an inquiry). Therefore, the input information D51 received by the control system 5000 can be said to be an example of first information indicating a request in the work environment. Also, the response information D52 can be said to be information used for the task (response task) in response to such input information D51. Hereinafter, the response information D52 output to a predetermined output destination from the operating environment of the learning model to which model input data based on the input information D51 is input may be referred to as second information.

[0397] Next, a description will be given of the operation of the control system 5000 of this embodiment.

[0398] 28, first, the control system 5000 accepts the input information D51v (step S510). For example, the above-mentioned input unit 102 or the input processing unit 201 may accept the input information D51v. The accepted input information D51v is input to the voice recognition unit 513v.

[0399] When the voice recognition unit 513v receives the input information D51v, it recognizes the voice contained in the input information D51v and converts it into input information D51 that matches the data format of the input of the learning model unit 500 (step S511). The converted input information D51 is input to the learning model unit 500 as model input data D101.

[0400] When the voice recognition unit 513v is omitted, the received input information D51v may be input to the learning model unit 500 as the model input data D101.

[0401] Next, the control system 5000 performs a process of generating response information D52 using the learning model unit 500 (step S512). In step S512, the learning model unit 500 (more specifically, the model control unit 101) outputs response information D52 corresponding to the input information D51 based on the model information D102 and the input information D51 that was input, and model reference information D104 as necessary.

[0402] In step S512, the pre-processing unit 105 and / or the post-processing unit 106 of the learning model unit 500 may further perform the above-mentioned processing.

[0403] The response information D52 output from the learning model unit 500 is input to, for example, the voice synthesis unit 514v (step S513). The response information D52 may be input to the voice synthesis unit 514v directly from the control system 5000 (more specifically, the learning model unit 500 or the information processing device 10 as its operating environment), or indirectly via a communication network, other devices (servers, various conversion devices, etc.), or human hands.

[0404] Next, the voice synthesis unit 514v converts the input response information D52 into voice-format response information D52v and outputs it (step S514). The voice synthesis unit 514v may generate the response information D52v, for example, by synthesizing a voice that speaks the response content indicated by the response information D52, which is a data format other than voice. The response information D52v is output to the user 1, who is the source of the input information D51v (step S515).

[0405] When the voice synthesis unit 514v is omitted, the response information D52 output from the learning model unit 500 may be output to the user 1 who is the source of the input information D51v.

[0406] As described above, in this embodiment, response information D52 can be dynamically generated using the learning model unit 500 and sent back to the user who sent the information from the user 1 without the need to prepare an operator or a site with pre-embedded response content, thereby making it possible to improve the efficiency and performance of the response work.

[0407] Variation 5-1. Next, a modified example of the control system 5000 will be described. Fig. 29 is a configuration diagram showing an example of a control system 5000a which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control system 5000 are given the same reference numerals and description thereof will be omitted.

[0408] A control system 5000a shown in FIG. 29 differs from the control system 5000 in that a true / false determination unit 515 is provided.

[0409] The correctness determining unit 515 determines whether the content indicated by the response information D52 output from the learning model unit 500 is correct or not. For example, the correctness determining unit 515 may output the response information D52 to the user 1 or update the content of the reference information storage unit 12 only when it determines that the content indicated by the response information D52 is correct.

[0410] In addition, for example, when the correctness determining unit 515 determines that the content indicated by the response information D52 is not correct, the correctness determining unit 515 may prompt the learning model unit 500 to acquire another response information D52 (reacquire the model output data). The correctness determining unit 515 may be provided as an example of the above-mentioned post-processing unit 106, for example.

[0411] Other points may be the same as those of the other control systems according to the present embodiment.

[0412] As described above, according to this modified example, it is determined whether the content indicated by the response information output from the learning model unit 500 is correct, and based on the result, whether or not to output to the user, re-acquire the response information, and update the reference information are performed, thereby achieving higher performance in the response process.

[0413] Variation 5-2. Next, a second modified example of the control system 5000 will be described. Fig. 30 is a configuration diagram showing an example of a control system 5000b which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control systems 5000 and 5000a are given the same reference numerals and will not be described.

[0414] As shown in FIG. 30, the control system 5000b may further include an emotion determination unit 516.

[0415] The emotion determination unit 516 uses the input information D51 and other information to determine the emotion of the sender, user 1. The emotion determination unit 516 may also determine the emotion of user 1 after response information D52 from the learning model unit 500 is output to the user 1.

[0416] The emotion of user 1 determined by the emotion determination unit 516 may be input to the learning model unit 500 as state information D55 included in the model reference information D104, or may be recorded as history in the reference information storage unit 12 together with the input and output data of the model.

[0417] As a method of recording in the reference information storage unit 12, for example, the control system 5000b may further include a registration determination unit 518, and the registration determination unit 518 may determine whether or not to record in the reference information storage unit 12 based on the result of the determination of the emotion of user 1 by the emotion determination unit 516.

[0418] For example, if the determined emotion of user 1 is positive, the registration determination unit 518 may record the input / output data of the model as history information in the reference information storage unit 12 as a good example. At this time, if there is a determination result of the emotion of user 1 before the output of the response information D52 from the learning model unit 500, the registration determination unit 518 may record the input / output data of the model including emotion information before and after the response as history information in the reference information storage unit 12.

[0419] Furthermore, for example, if the determined emotion of user 1 is negative, the registration determination unit 518 may record the input / output data of the model as history information in the reference information storage unit 12 as a bad case. At this time, if there is a determination result of the emotion of user 1 before the output of the response information D52 from the learning model unit 500, the registration determination unit 518 may record the input / output data of the model including emotion information before and after the response as history information in the reference information storage unit 12.

[0420] In addition, the control system 5000b may further include an additional learning unit 519, and when updating the contents of the reference information storage unit 12, the control system 5000b may reconstruct (additionally learn) the model reference information D104 stored in the reference information storage unit 12 and other information referenced by the model control unit 101 based on the update information.

[0421] Furthermore, the control system 5000b may include an evaluation acquisition unit 517 instead of the emotion determination unit 516 or in addition to the emotion determination unit 516.

[0422] The evaluation acquisition unit 517 inquires of the user 1 about the evaluation of the response information D52, and acquires evaluation information D59 as a response. The evaluation information D59 can be used for updating information referred to by the model, additional learning, and the like, similar to the above-mentioned emotion of the user 1, for example.

[0423] The control system 5000 b may further include a control determination unit 520 .

[0424] The control determination unit 520 designates information to be controlled and / or designates output tendency settings for the control generation unit 512 based on the voice recognition result for the input information from the user 1, the emotion determination result and / or the evaluation result of the response information D52, instructions from an operator (not shown), etc. Here, the voice recognition result for the input information from the user 1 may include information such as the attributes, emotions, region, language, past use, and frequency of use of the user 1. In addition, the control determination unit 520 may set synthetic voice for the voice synthesis unit 514v based on the voice recognition result for the input information from the user 1, the emotion determination result and / or the evaluation result of the response information D52, instructions from an operator (not shown), etc.

[0425] The control determination unit 520 can specify, for example, the difficulty of explanation in a response, the way of speaking (tone of speech), the language, the level of grammar, politeness, the speaker's position, the destination of the story, etc., as an example of output tendency settings. It can specify the gender, tone of speech, etc., of the synthetic voice. In addition, the control determination unit 520 can specify, for example, the gender, way of speaking, the language, the level of grammar, politeness, etc., of the synthetic voice, as an example of synthetic voice settings. The control determination unit 520 may perform these settings, for example, based on predetermined setting rules.

[0426] The elements of the control system 5000b shown in FIG. 30 can be appropriately selected depending on the desired functions.

[0427] Other points may be the same as those of the other control systems according to the present embodiment.

[0428] As described above, according to this modification, the control determination unit 520 designates the control target information and / or designates the output tendency setting based on the information obtainable from the control system 5000b, so that it is possible to generate response information that is likely to match the request of the sender. This makes it possible to further improve the performance of the response work to the user.

[0429] Variation 5-3. Next, a third modified example of the control system 5000 will be described. Fig. 31 is a configuration diagram showing an example of a control system 5000c which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control systems 5000, 5000a, and 5000b are given the same reference numerals and will not be described.

[0430] As shown in FIG. 31, the control system 5000c may further include an image analysis unit 513i, an image generation unit 514i, and a program generation unit 514p.

[0431] When the input from the user 1 includes the input information D51i in an image format, the image analysis unit 513i analyzes the image indicated by the input information D51i, converts it into a format that matches the data format of the learning model unit 500, and outputs it. The image analysis unit 513i may convert the input information D51i in an image format into the input information D51 in a text format, for example.

[0432] For example, when the input from the user 1 includes an image capturing an operation screen of a product owned by the user 1, the image analysis unit 513i may analyze the image, identify which operation screen of which product it is, and what operation state it is in, convert it into text explaining it, and output it. Also, when the input from the user 1 includes an image capturing a purchase site that the user 1 is browsing, the image analysis unit 513i may analyze the image, identify which operation screen of which site it is, and what operation state it is in, convert it into text explaining it, and output it.

[0433] The image generating unit 514i generates and outputs an image based on the response information D52. For example, when the response information D52 output from the learning model unit 500 includes a data format other than an image, the image generating unit 514i may generate and output an image showing the contents of the part indicated by the response information D52. For example, when the response information D52 is a data structure including a designation of a data format, the image generating unit 514i may convert a data element for which an image format is designated in the designation into an image format and output the image. For example, the image generating unit 514i may generate response information D52v in an image format based on the response information D52 in a text format. For example, the image generating unit 514i may perform a synthesis process of adding the contents indicated by the response information D52 in a text format as an annotation to an image included in the input information D51. In addition, the image generating unit 514i may perform a process of highlighting a part of the image included in the input information D51 based on the response information D52 in a text format. The image generating unit 514i may generate an image from input information (the response information D52 and, if necessary, the input information D51) using a learning model.

[0434] The program generating unit 514p converts the contents indicated by the response information D52 into a data format of a predetermined program and outputs the converted contents. For example, when the response information D52 output from the learning model unit 500 includes a data format other than the data format of the predetermined program, the program generating unit 514p converts the contents of the part indicated by the response information D52 into the data format of the predetermined program and outputs the converted contents. For example, when the response information D52 is a data structure including a designation of a data format, the program generating unit 514p may convert a data element, the data format of which is designated in the designation, into the data format of the predetermined program and output the converted contents. For example, the program generating unit 514p may convert the response information D52 in a text format into response information D52p in a data format of the predetermined program. The program generating unit 514p may generate a predetermined program from input information using a learning model.

[0435] The image analysis process by the image analysis unit 513i is performed, for example, in the above-mentioned step S511. The image generation process by the image generation unit 514i and the program generation process by the program generation unit 514p are performed, for example, in the above-mentioned step S514.

[0436] Other points may be the same as those of the other control systems according to the present embodiment.

[0437] As described above, according to this modification, inquiries and responses can be made not only by voice but also by voice and images, so that it is possible to more effectively respond to inquiries about, for example, an operation screen. Also, according to this modification, in addition to voice and images, a program can be provided to the sender as response information, so that it is possible to more effectively respond to inquiries about malfunctions, etc.

[0438] Variation 5-4. Next, a fourth modified example of the control system 5000 will be described. Fig. 32 is a configuration diagram showing an example of a control system 5000d which is a modified example of the control system 5000 according to the present embodiment. Note that the same elements as those of the control systems 5000 to 5000c are given the same reference numerals and description thereof will be omitted.

[0439] In this modified example, there is a function to switch between a response by the operator 8 or a response by another learning model based on the contents of an inquiry from the user 1 and / or the output result from the learning model.

[0440] As shown in FIG. 32, the control system 5000d may further include a call confirmation unit 531 and an output selection unit 532.

[0441] Here, the control system 5000d is assumed to include a learning model unit 500a as a first response function, and an operator 8 and a communication channel with the operator 8 as a second response function. The control system 5000d may further include another learning model unit 500b as a third response function, which has a different algorithm or data used from the learning model unit 500a. Note that the second response function may include another learning model unit 500b having a different algorithm or data used from the learning model unit 500a. In that case, the third response function may further include an operator 8 and a communication channel with the operator 8. Note that the type and number of response functions are not particularly limited. For example, the response function to be switched to may be a response system that does not use a learning model.

[0442] In this example, we will explain the case where the learning model unit 500a, which is the first response function, is the above-mentioned learning model unit 500, the second response function is an operator 8 and a communication channel with the operator 8, and the third response function is another learning model unit 500b that has a different algorithm or data used from the learning model unit 500a.

[0443] Here, the learning model unit 500a may be a local learning model that obtains output results based on local information, such as a restricted reference database, and the learning model unit 500b may be a global learning model that obtains output results based on global information, such as free access to an external network.

[0444] The call confirmation unit 531 switches the processing destination for performing the response process based on the contents of the inquiry from the user 1 and / or the output result from the learning model.

[0445] The call confirmation unit 531 may call the operator 8 as the second response function, for example, when it is determined that the accuracy of the output by the first response function is not expected based on the contents of the inquiry from the user 1 and / or the output result from the learning model. The call confirmation unit 531 may call the operator 8 using, for example, a communication channel with the operator 8, and input the input information D51 to an operation device of the operator 8. The call confirmation unit 531 may also call the operator 8 using a communication channel with the operator 8, and input the input information D51 to an operation terminal (not shown) of the operator 8.

[0446] Furthermore, when it is determined that a call to the second response function is impossible or that the output accuracy is not expected, the call confirmation unit 531 may call the learning model unit 500b as a third response function. The call confirmation unit 531 may call the learning model unit 500b by inputting the input information D51 to the learning model unit 500b using an interface with the learning model unit 500b, for example.

[0447] Here, the output accuracy may be judged, for example, by using an evaluation value or likelihood output by the response function itself, or by using the reliability evaluation described above. In addition, if the response function itself outputs a message indicating that it does not understand or requesting the invocation of another function, it is also possible to judge the output accuracy by the presence or absence of such a message.

[0448] The output selection unit 532 selects response information D52 to be output to the user 1 based on the switching result of the response process by the call confirmation unit 531. When the execution subject of the response process is determined to be the first response function as a result of the switching of the response process by the call confirmation unit 531, the output selection unit 532 outputs response information D52a to be output from the first response function to the user 1. When the execution subject of the response process is determined to be the second response function as a result of the switching of the response process by the call confirmation unit 531, the output selection unit 532 outputs response information D52b to be output from the second response function to the user 1. When the execution subject of the response process is determined to be the third response function as a result of the switching of the response process by the call confirmation unit 531, the output selection unit 532 outputs response information D52c to be output from the third response function to the user 1.

[0449] The output selection unit 532 may output the output from the selected response function to user 1 by controlling an output changeover switch (not shown) that switches the connection path (such as a circuit or communication path) connecting the response function selected as the execution subject and user 1, the output destination.

[0450] Here, the connection path between the response function and the user 1 may include various conversion devices such as the above-mentioned voice synthesis unit, image generation unit, program generation unit, and a predetermined interface, as necessary.

[0451] For example, when the character input by the operator 8 using the operation terminal is output as the response information D52b, the connection path between the response function and the user 1 may include a voice synthesis unit that converts text into voice. The output selection unit 532 may also accept, as the output of the second response function, etc., a modified version of the response information D52a output by the first response function. In this case, the operation terminal of the operator 8 includes a text display unit and a text input unit, and the control system 5000d may accept, for example, the response information D52b output from the operation terminal of the operator 8, which is partially modified.

[0452] Other points may be the same as those of the other control systems according to the present embodiment.

[0453] As described above, according to this modified example, in addition to generating responses using the above-mentioned learning model unit 500, responses can also be generated by an operator or using other learning models (e.g., a tandem structure model in which multiple models are connected, a multimodal model, or a model learned specifically for a specific device or service), thereby making it possible to further improve the performance of response operations to users.

[0454] In robot control, such as operation instructions for a robot or collaborative work between humans and a robot, the complexity and sophistication of the control also depend on the relative relationship between the human and the machine. By implementing the configurations of the above-mentioned embodiments and the configurations of the modifications, the complexity and sophistication of the robot control can be improved or reduced, thereby improving the variation in the relative operating ability of humans with respect to machines, or better reflecting the differences in the operating preferences of humans with respect to machines.

[0455] In the above-mentioned embodiments, an example of a system configuration according to a task of interest is illustrated, but the control system according to the present disclosure is not limited to the above-mentioned example. For example, the control system according to the present disclosure may be a combination of one or more of the above-mentioned embodiments.

[0456] As an example, the control system of the present disclosure can combine the configuration of embodiment 1 with the configuration of embodiment 4, and utilize the functions of embodiment 4 to input information indicating a solution obtained from sensor data into the control system of embodiment 1, convert it into a program, and directly control the target device 2.

[0457] Furthermore, the embodiments and modifications are not limited to the examples described above, and can be modified as appropriate within the scope of the disclosure.

[0458] Furthermore, the control system and control method according to the present disclosure include the control systems and control methods described in the following appendices.

[0459] (Appendix 1) A control system for supporting an operation by a person or an object using equipment, comprising: an input interface that accepts an input of first information indicating a situation or a request in a work environment in which the work is performed; A model processing unit that is provided to be accessible to a predetermined learning model; and an output interface that outputs second information for supporting the task based on an output from the learning model; the model processing unit inputs model input data based on the first information to the learning model, and receives model output data corresponding to the model input data from the learning model; the model output data includes information used in the operation; The output interface outputs the second information based on the model output data. A control system comprising: (Appendix 2) the first information includes information indicating a control content or an operation content required for the device, The model input data is data indicated in a format in which the control content or operation content indicated by the first information matches the input of the learning model, the model output data includes information used to control or operate the device, the information corresponding to the control content or operation content indicated by the model input data; The second information includes information in which information used for controlling or operating the device, which is included in the model output data, is described in a predetermined format that can be recognized at an output destination of the output interface. 2. The control system of claim 1. (Appendix 3) an output destination of the output interface is the device or an interface that requests control of the device; The second information is output to the device or an interface that requests control of the device, and as a result, the device is controlled. 3. The control system of claim 2. (Appendix 4) an execution code generating unit that generates and outputs an execution code that is executable by the device; an output destination of the output interface is the execution code generation unit; The device is controlled by an execution code generated as a result of the second information being output to the execution code generating unit. 3. The control system of claim 2. (Appendix 5) The output destination of the output interface is a terminal operated by a user, As a result of the second information being output to the terminal, the device is controlled. 3. The control system of claim 2. (Appendix 6) the first information includes information indicating a situation in the work environment, The model input data is data in a format in which the situation of the work environment indicated by the first information matches the input of the learning model, the model output data includes an analysis result of the working environment situation represented by the model input data and / or information on how to improve the situation; The second information includes information in which an analysis result of a situation in the work environment and / or information on a method for improving the situation is described in a predetermined format that can be recognized at an output destination of the output interface. 2. The control system of claim 1. (Appendix 7) The model processing unit is provided to be able to access a first learning model and a second learning model, the model processing unit inputs first model input data based on the first information to the first learning model, and receives first model output data corresponding to the first model input data from the first learning model; the model processing unit inputs second model input data based on the first model output data to the second learning model, and receives second model output data corresponding to the second model input data from the second learning model; The output interface outputs the second information based on the second model output data. 2. The control system of claim 1. (Appendix 8) the first information includes information indicating a control content or an operation content required for the device, The first model input data is data indicated in a format in which the control content or operation content indicated by the first information matches the input of the first learning model, The first model output data includes information in which the control content or operation content indicated by the first model input data is more generalized or more specific, The second model input data is data indicated in a format in which the control content or operation content indicated in the first model output data matches the input of the second learning model, the second model output data includes information used to control or operate the device, the information corresponding to the control content or operation content indicated by the second model input data; The second information includes information in which information used for controlling or operating the device, which is included in the second model output data, is described in a predetermined format that can be recognized at an output destination of the output interface. 8. The control system of claim 7. (Appendix 9) the first information includes information indicating a situation in the work environment, The first model input data is data in which the situation of the work environment indicated by the first information is in a format that matches the input of the first learning model, the first model output data includes an analysis result of the conditions of the working environment represented by the model input data; The second model input data is data in a format in which the analysis result of the situation of the working environment indicated by the first model output data matches the input of the second learning model, the second model output data includes information on how to improve a situation in the work environment corresponding to the analysis result of the work environment represented by the second model input data; The second information includes information that is included in the second model output data and is about at least the method for improving the situation in the work environment, and is described in a predetermined format that is recognizable at the output destination of the output interface. 8. The control system of claim 7. (Appendix 10) The task is a response by a person or object using a device; The first information includes information indicating a reaction request content, which is a content that requires a reaction in the work environment, The model input data is data indicated in a format in which the response request content indicated by the first information matches the input of the learning model, the model output data includes information used in the response corresponding to the response request content indicated by the model input data; The second information includes information in which information used for the response included in the model output data is described in a predetermined format that can be recognized at an output destination of the output interface. 2. The control system of claim 1. (Appendix 11) an output destination of the output interface is a screen operation interface that requests control of the device via an operation screen; The model output data is an operation screen for actually performing an operation corresponding to the control content or operation content indicated by the model input data on the device, and includes information on the operation screen described in a predetermined format that can be recognized at the output destination of the output interface. 4. The control system of claim 3. (Appendix 12) the input interface accepts input of the first information indicating requests in the work environment from a plurality of users; The model processing unit inputs the model input data including the first information input from the multiple users to the learning model, and receives the model output data corresponding to the model input data from the learning model. 12. A control system according to any one of claims 1 to 11. (Appendix 13) The learning models are a language learning model that inputs natural language and obtains an output result, an image learning model that inputs an image and obtains an output result, and a multimodal model that inputs natural language and an image and obtains an output result. 13. A control system according to any one of claims 1 to 12. (Appendix 14) The model processing unit is provided to be able to access a first learning model and a second learning model, One of the first learning model and the second learning model is a local learning model in which a reference database is limited to internal information; The other of the first learning model and the second learning model is a global learning model in which the reference database is not limited to internal information. 14. A control system according to any one of claims 1 to 13. (Appendix 15) The model processing unit is provided to be able to access a first learning model and a second learning model, One of the first learning model and the second learning model is a learning model that can refer to information specifically defined in the work environment, The other of the first learning model and the second learning model is a learning model in which information specifically defined in the work environment cannot be referenced. 14. A control system according to any one of claims 1 to 13. (Appendix 16) An output confirmation unit that performs a simulation simulating the control and state of the device based on model output data output from the learning model. 16. The control system of any one of claims 1 to 15. (Appendix 17) Based on the information collected from the output destination of the output interface, additional learning of the learning model, determination of the accuracy of the output information, and flow control of the output information are performed. 17. The control system of any one of claims 1 to 16. (Appendix 18) An input processing unit that queries the input source when the first information includes unclear or uncertain information. 18. A control system according to any one of claims 1 to 17. (Appendix 19) The inquiry includes information indicating correction, addition, or cancellation of contents of the input / output data of the learning model. 19. The control system of any one of claims 1 to 18. (Appendix 20) The first information is time-series data that indicates, together with time information, a situation or a request in a work environment in which the work is performed. 19. The control system of claim 1. (Appendix 21) The learning model execution environment includes a model information storage unit that stores model information, and a model control unit that receives the model input data and outputs the model output data based on the model input data and the information stored in the model information storage unit. 19. The control system of claim 1. (Appendix 22) A control method for assisting a task performed by a person or an object using equipment, comprising: an input interface for receiving an input of first information indicating a situation or a request in a work environment in which the work is performed; a model processing unit provided to be accessible to a predetermined learning model inputs model input data based on the first information into the learning model, and receives model output data from the learning model corresponding to the model input data, the model output data including information used in the work; An output interface outputs, based on an output from the learning model, second information based on the model output data, the second information being for supporting the task. A control method comprising: [Industrial Applicability]

[0460] The control system according to the present disclosure can be suitably applied as a part of a work support system that supports work by a person or an object. Also, the control system according to the present disclosure can be suitably applied as a control system that controls an apparatus when performing some control or work using the apparatus. Here, the control system can also be suitably applied as a control system that controls FA apparatus, a control system in a home or building, or a control system that controls an information processing apparatus such as a server apparatus that processes information on a network. [Explanation of symbols]

[0461] 1000, 1000a, 1000b, 1000c, 2000, 3000, 3000a, 3000b, 3000c, 3000d, 3000e, 4000, 4000a, 5000, 5000a, 5000b, 5000d Control system, 100, 200, 300, 300a, 300b, 400, 400a, 400b, 500, 500a, 500b Learning model unit, 10, 20 Information processing device, 11 Model information storage unit, 12 Reference information storage unit, 101 Model control unit, 102 Input unit, 103 Output unit, 105 Preprocessing unit, 106 Postprocessing unit, 107 Model generation unit, 104, 104a Control unit, 201 Input processing unit, 202 Output confirmation unit, 203 Correction confirmation unit, 1 user, 1a input source, 2 target device, 2a output destination, 3 operation screen user interface, 4 controller, 4a general-purpose operation device, 41-1, 41-2 analysis unit, 42 switching unit, 43 output switching switch, 5 sensor, 6 model interface, 7 display, 8 operator, 110, 210, 310, 410 device information storage unit, 120 execution code generation unit, 230 status acquisition unit, 311 input interface, 312 output interface, 313 environment information storage unit, 350 display format information, 360 user history information storage unit, 511 database search unit, 512 control generation unit, 513v voice recognition unit, 513i image analysis unit, 514v image synthesis unit, 514p program generation unit, 515 correct / incorrect judgment unit, 516 emotion judgment unit, 517 evaluation acquisition unit, 518 Registration judgment unit, 519 additional learning unit, 531 call confirmation unit, 532 output selection unit, 533 output switching unit, D101 model input data, D102 model information, D103 model output data, D104 model reference information, D11, D21, D31, D41, D51, D51v, D51i input information, D12 control description, D22, D34 control command, D42a, D42b analysis result, D52, D52v, D52i, D52p, D52a, D52b, D52c response information, D32, D32a, D32b operation command, D320 operation information, D13, D23, D33, D43 device information, D33a environment information, D14 execution code, D44a, D44b Result information, D15, D25, D35, D45 Status information, D16, D26, D36, D46 Feedback information, D17, D27, D37,D47, D57 Inquiry, D18, D28, D38, D48 Supplementary information, D59 Evaluation information, D361 User history information.

Claims

1. an input interface that accepts input of first information indicating a command for a target device by a user, the command being composed of a plurality of operations for the target device; a learning model unit that inputs the first information into a learning model and outputs second information indicating a sequential control, which is a series of controls in the target device for realizing the plurality of operations; an input determination unit that uses a command rule that associates a command for the target device with a control code for the target device to determine whether or not a command indicated by the first information matches a command included in the command rule, The input determination unit is A control system that outputs the first information to the learning model unit if the command indicated by the first information does not match a command included in the command rule.

2. The input determination unit: The control system according to claim 1 , wherein if the command indicated by the first information matches a command included in the command rule, a control code corresponding to the command indicated by the first information is output to the target device.

3. An input interface that accepts input of first information indicating a command by a user to a target device, the command being composed of a plurality of operations on the target device; a learning model unit that inputs the first information into a learning model and outputs second information indicating a sequential control, which is a series of controls in the target device for realizing the plurality of operations; an input determination unit that determines by machine learning whether or not a command for the target device corresponds to a control code for the target device; The input determination unit is A control system that outputs the first information to the learning model unit when it is determined that the command indicated by the first information does not correspond to the control code of the target device.

4. The input determination unit: A control system as described in claim 3, wherein when it is determined that the command indicated by the first information corresponds to the control code of the target device, the control code of the target device corresponding to the command indicated by the first information is output to the target device.

5. The learning model unit includes:

5. The control system according to claim 1, wherein the sequential control outputs, as the second information, information indicated by a control code of the target device to the target device.

6. the input interface accepts a plurality of commands from different users as a plurality of first information; The learning model unit includes: A control system according to any one of claims 1 to 4, which arbitrates the multiple commands indicated by the multiple first information and outputs information indicating a single sequential control for realizing the multiple commands as the second information.

7. The control system includes: a device operation device which is an operation screen user interface for operating the target device; The learning model unit includes: generating an operation screen showing the sequential control as the second information, displaying the operation screen on the appliance operation device, and having the user select whether or not to approve the sequential control via the operation screen; The equipment operation device is The control system according to claim 1 , wherein when the sequential control is approved by the user, a control code for the target device indicating the sequential control approved by the user is transmitted to the target device.

8. The learning model unit includes: generating the operation screen showing a list of candidates for the sequential control as the second information, displaying the operation screen on the appliance operation device, and having the user select a sequential control from the list of candidates via the operation screen; The equipment operation device is The control system according to claim 7 , wherein the sequential control selected by the user is regarded as the sequential control approved by the user, and a control code for the target device is transmitted to the target device.

9. The equipment operation device is a general-purpose operating device capable of operating the target device and another device different from the target device, The learning model unit includes: The control system according to claim 7 , wherein a display indicating the sequential control is displayed as the second information on a display screen of the device operation device using display format information that stores a display format to be displayed on the device operation device.

10. The equipment operation device is The control system according to claim 9 , capable of operating a plurality of different types of devices used by the user.

11. The equipment operation device is The control system according to claim 7 , further comprising the input interface, and supporting the user in inputting commands to the target device by displaying sequential controls approved by the user on a display screen.

12. The learning model unit includes: The control system according to claim 1 , further comprising: a controller configured to output the second information corresponding to a specific environment based on environmental information representing an environment of the target device.

13. The control system according to claim 12 , wherein the environmental information includes at least one of a state of the target device, a state around the target device, and a state of the user including vital data of the user as the environment.

14. The learning model unit includes: The control system according to claim 12 , wherein the second information is output according to the user, based on user history information including a history of commands given by the user to the target device.

15. the user history information includes commands input by the user, states of the user and the target device, control codes output by the target device, and changes in the environment; The learning model unit includes: The control system according to claim 14 , wherein the second information is output according to states of the user and the target device, based on the user history information.

16. The learning model unit includes: a first learning model unit that inputs the first information into a first learning model and outputs operation information indicating the sequential control according to a specific environment based on the environmental information; a second learning model unit that inputs the operation information into a second learning model and outputs a control code of the target device for realizing the operation information as the second information to the target device; 15. The control system of claim 14, comprising:

17. The first learning model unit includes: The control system of claim 16, wherein the first information is input into a first learning model, and the operation information indicating the sequential control according to the state of the user and the target device is output based on the environmental information and the user history information.

18. The control system according to any one of claims 1 to 4, wherein the learning model is a Large Language Model (LLM) or a Vision-Language Model (VLM).

19. The input interface includes: The control system according to claim 1 , wherein the first information is inputted as at least one of a signal code, a natural language, and an image, or a combination thereof.

20. a computer receives input of first information indicating a command for a target device by a user, the command being composed of a plurality of operations for the target device; A computer executes a learning model process in which the first information is input into a learning model and second information indicating a sequential control, which is a series of controls in the target device for realizing the plurality of operations, is output; a computer uses a command rule that associates a command for the target device with a control code for the target device to determine whether or not the command indicated by the first information matches a command included in the command rule; A control method in which a computer outputs the first information to the learning model processing if the command indicated by the first information does not match a command included in the command rule.

21. an input interface process for receiving input of first information indicating a command for a target device by a user, the command being composed of a plurality of operations for the target device; A learning model process of inputting the first information into a learning model and outputting second information indicating a sequential control, which is a series of controls in the target device for realizing the plurality of operations; an input determination process for determining whether or not a command indicated by the first information matches a command included in a command rule that associates a command for the target device with a control code for the target device; A control program for causing a computer to execute the above. The input determination process includes: a control program that outputs the first information to the learning model processing if the command indicated by the first information does not match a command included in the command rule.

22. A method for detecting a user's input of first information indicating a command for a target device, the command being composed of a plurality of operations for the target device, comprising: A computer executes a learning model process in which the first information is input into a learning model and second information indicating a sequential control, which is a series of controls in the target device for realizing the plurality of operations, is output; A computer determines whether a command for the target device corresponds to a control code for the target device through machine learning; A control method in which a computer outputs the first information to the learning model processing when it determines that the command indicated by the first information does not correspond to the control code of the target device.

23. An input interface process for accepting input of first information indicating a command for a target device by a user, the command being composed of a plurality of operations for the target device; A learning model process of inputting the first information into a learning model and outputting second information indicating a sequential control, which is a series of controls in the target device for realizing the plurality of operations; an input determination process for determining by machine learning whether or not a command for the target device corresponds to a control code for the target device; A control program for causing a computer to execute the above. The input determination process includes: A control program that outputs the first information to the learning model processing when it is determined that the command indicated by the first information does not correspond to the control code of the target device.

Citation Information

Patent Citations

  • Sensitivity reaction type controller

    JP1994266403A

  • Control system, record system, information processor and method, program, and recording medium

    JP2007276052A

  • Systems and methods to configure, program, and personalize a medical device using a digital assistant

    JP2021074528A

  • Operation support device, operation support method, and program

    JP2021179655A

  • Information processing device, information processing system, information processing method, and information processing program

    WO2022269869A1

Cited By

  • Elevator risk early warning method based on channel clustering of Transform

    CN120508920A

  • Information processing program, information processing device, and information processing method

    JP2026091213A

  • system

    JP2026155235A

  • system

    JP2026155246A

  • Information generation device, information generation method, computer program, and learning model

    JP2026167988A