Control System and Control Method

Abstract

A user terminal (1) that generates input information (D11) including information indicating control content required for a target device (2) in response to an operation by a user (1), a control description (D12) generated by a learning model unit 100 that generates a control description (D12) based on the input information (D11) generated by the user terminal (1), or a state acquisition unit (130) that acquires information regarding at least any one of the operations of the target device (2) and feeds back the information to at least one of the user terminal (1) or the learning model unit (100).

Description

Technical Field

[0001] The present disclosure relates to a control system and a control method.

Background Art

[0002] In recent years, the use of AI (artificial intelligence) has been progressing. In particular, AI called generative artificial intelligence, which can generate various contents, has also begun to spread, and it is expected that the application areas of AI will expand. The application areas of AI are not limited to work within the home, but also include work in various facilities such as buildings, factories, stations, schools, hospitals, or commercial facilities, and work in various places and scenes such as outdoors on roads, outdoor facilities, in the air, or on the sea.

[0003] For example, Patent Document 1 describes a machining program generation device that generates a program for controlling a machine using a large language model.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In order to assist work by a person or an object, consider that not only the generation of a control program for a device but also a part or all of the tasks included in the work are taken over by a learning model. Note that the “work by a person or an object” includes not only work in the real space performed by a person or a machine but also work in the data space such as information processing performed by a processor such as a CPU (central processing unit).

[0006] Examples of operations by objects include, for example, the following. · Operations by various devices such as robots, machines, apparatuses, sensors, etc. · Operations by various mobilities such as cars, trains, buses, aircraft, ships, etc. Such operations may include operations called, for example, control, processing, machining, instruction, calculation, input, output, display, communication, testing, manufacturing, conversion, generation, measurement, irradiation, emission, inhalation, heat dissipation, heating, cooling, recording, reading, molding, driving, moving, transporting, flying, investigating, monitoring, measuring, extraction, etc.

[0007] Examples of operations by humans include, for example, the following. · Operations performed by a person on a person or another living being · Operations performed by a person on various devices Such operations may include operations called, for example, conversation, viewing, confirmation, operation, monitoring, instruction, mediation, interpretation, etc.

[0008] Note that the above examples are just examples, and the operations to be supported by the present disclosure are not limited to these.

[0009] When causing an information processing apparatus to execute some or all of the tasks included in the operations by a person or an object using some learning model, the validity of the output of the model may become a problem. Also, the validity of the input of the model that affects the output of the model may become a problem.

[0010] Also, depending on the target device, appropriate control may not be possible without grasping the current situation. In such a case, how to perform situation recognition may become a problem. At this time, it is also conceivable that recognition of a situation with continuity including not only the current situation but also the past situation is required. For example, in the case of determining the next control based on the content of the control performed in the past, the accuracy of situation recognition may become a problem in order to ensure the continuity of the control.

[0011] In addition, when immediacy is required for the control of equipment, the response time from giving an instruction to the learning model until the result is obtained may become a problem.

[0012] In addition, the maintainability of the model may become a problem, such as the need to relearn the model every time the equipment is changed or added.

[0013] Thus, there are still various problems scattered in the use of the learning model. Depending on the size of the problem, even if one tries to improve the work efficiency or performance by using the learning model, it may conversely reduce the work efficiency or performance.

[0014] These problems when using the learning model will become more prominent especially when the work to be supported is more complex and more advanced.

[0015] Therefore, the present disclosure aims to further improve the efficiency or performance of work by people or objects using the learning model.

Means for Solving the Problem

[0016] The control system according to the present disclosure includes a user terminal that generates input information including information indicating the control content required for the equipment in response to an operation by the user, and a control description generated by a learning model unit that generates a control description based on the input information generated by the user terminal, or information regarding at least any of the operations of the equipment, and a state acquisition unit that acquires the information and feeds back the information to at least one of the user terminal or the learning model unit. The user terminal makes a confirmation inquiry As an inquiry to re-ask the input information, or an inquiry to request re-entry to something with a changed state or expression for the instruction words included in the input information, and is characterized by receiving from the learning model unit or outputting to the user.

[0017] The control system according to the present disclosure is a control system for assisting work by a person or an object using equipment, and includes an input interface that receives an input of first information indicating a situation or a request in a work environment, which is an environment where the work is performed; a model processing unit that is accessible to a predetermined learning model; an output interface that outputs second information for assisting the work based on an output from the learning model; and an input processing unit that makes a confirmation inquiry about an instruction word to the input source when the first information includes the instruction word. As an inquiry to re-ask the first information, or an inquiry to request re-entry to something with a changed state or expression The model processing unit inputs model input data based on the first information to the learning model, receives model output data corresponding to the model input data from the learning model, the model output data includes information used for the work, and the output interface outputs second information based on the model output data.

[0018] In the control method according to the present disclosure, a user terminal generates input information including information indicating control content required for equipment in response to an operation by a user, and a state acquisition unit acquires control description generated by a learning model unit that generates a control description based on the input information generated by the user terminal, or information regarding at least any one of operations of the equipment, and feeds back the information to at least one of the user terminal or the learning model unit, and the user terminal receives a confirmation inquiry about an instruction word included in the input information As an inquiry to re-ask the input information, or an inquiry to request re-entry to something with a changed state or expression from the learning model unit or outputs it to the user.

[0019] The control method according to the present disclosure is a control method for assisting work by a person or an object using a device, wherein an input interface receives an input of first information indicating a situation or a request in a work environment which is an environment where the work is performed, a model processing unit provided to be accessible to a predetermined learning model inputs model input data based on the first information to the learning model, and receives model output data corresponding to the model input data from the learning model, the model output data including information used for the work, an output interface outputs second information based on the output from the learning model, the second information being second information for assisting the work based on the model output data, and an input processing unit makes a confirmation inquiry about the instruction word to the input source when the first information includes an instruction word. As an inquiry to re-ask the first information, or an inquiry to request re-entry to something with a changed state or expression It is characterized by performing the above.

Effects of the Invention

[0020] According to the present disclosure, by using a learning model, it is possible to further improve the efficiency or performance of work by a person or an object.

Brief Description of the Drawings

[0021]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Figure 32

Mode for Carrying Out the Invention

[0022] Hereinafter, in order to explain the present disclosure in more detail, embodiments for carrying out the present disclosure will be described with reference to the accompanying drawings. Hereinafter, the same elements will be denoted by the same reference numerals and the description thereof will be omitted.

[0023] Embodiment 1. In this embodiment, an example of assisting the work related to code generation of the target device 2 by using a learning model will be described. Note that the “learning model” referred to here includes not only a model that performs learning but also the meaning of a learned model. The same applies to other embodiments in this regard. Also, the “target device” may be simply referred to as “device”.

[0024] FIG. 1 is a configuration diagram showing an example of the control system 1000 according to Embodiment 1. The control system 1000 shown in FIG. 1 is a control system for controlling a device by using a learning model, and includes a learning model unit 100, a device information storage unit 110 (denoted as a device information DB in the figure), and an execution code generation unit 120.

[0025] Note that in FIG. 1, the case where the learning model unit 100 and the device information storage unit 110 are provided inside the control system 1000 is shown. However, it is not limited to this, and the learning model unit 100 and the device information storage unit 110 may be provided outside the control system 1000.

[0026] Note that in FIG. 1, the user 1 and the target device 2 are shown, but the control system 1000 may include these. In that case, "user 1" may be read as "user terminal 1". The user terminal 1 is a terminal that performs operations such as output of the input information D11 in response to an operation by the user 1. This is the same in other embodiments.

[0027] When the input information D11 is input, the learning model unit 100 outputs the control description D12. When the input information D11 is input, the learning model unit 100 outputs the control description D12 based on the model information D102 described later.

[0028] In the present embodiment, the learning model unit 100 is a model and its operating environment configured to output the control description D12 corresponding to the input information D11 when the input information D11 is input. Further, the learning model unit 100 may be a model and its operating environment configured to generate and output the control description D12 based on the input information D11, the device information D13, and / or other information that can be referred to in the learning model unit 100 (such as the model reference information D104 described later) when the input information D11 is input.

[0029] In this embodiment, the input information D11 includes information indicating the control content required for the target device 2. The input information D11 may be, for example, text, an image (including a video), audio, or a combination thereof that indicates the control content for the target device 2. The input information D11 may be, for example, text, an image, audio, or a combination thereof that indicates a plurality of control contents for the target device 2. Further, the input information D11 may include information indicating the content of control performed continuously in time. In that case, it may be time-series data of a predetermined data structure including text, an image, audio, or a combination thereof that indicates the control content as described above. It is assumed that the method of indicating the control content conforms to the input format of the model used by the learning model unit 100, but this is not the case when error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 100.

[0030] As an example of the method of indicating the control content in the input information D11, after specifying the control to be performed on the target device 2, examples include the value of the parameter for performing the control and the method of specifying the state after the control. In that case, the input information D11 may include, for example, information specifying the control and information indicating the value of the parameter for performing the control or the state after the control. The value of the parameter for performing the control may include, for example, values related to the type of control (such as ON / OFF), direction, amount, and time. Examples of the control content include "turn on function X" for a PLC (programmable logic controller), "move the tip to point A" for a robotic arm, and "lower the set temperature by 1 degree" for an air conditioner. Further, as an example of the method of indicating the control content in the input information D11, there is a method of using various types of information such as a document string (docstring) explaining the specifications such as a function, a specification document, or a specification document, design document, operation instruction, control code, source code, etc. applied to other devices of other models.

[0031] In addition, as a way of indicating the control content in the input information D11, not only the explicit indication method as described above, but also, for example, when there is control implemented by a certain operation, indicating the operation content can be cited as a method of indicating the corresponding control content. Also, for example, there is a method of implicitly indicating by the actions and words of user 1 associated with a specific control, the operation result of target device 2, or a similar control command to other models, etc. In other words, the input information D11 may include not only information directly indicating the control content for target device 2, but also information indirectly indicating it using the operation content corresponding to the control content, or the actions and words of user 1, the image of target device 2, etc. As an example, as something indicating the control content related to the temperature control of an air conditioner, words such as "hot" spoken by user 1, or the actions of user 1 showing discomfort from the heat such as wiping sweat, rolling up sleeves, or raising hands can be used. In this case, as the input information D11, information such as text, voice, or image indicating the speech of user 1, or information such as an image (video) indicating the actions of user 1 can be used. As another example, as something indicating the control content related to the arm control of a robot device, the posture of the robot device after control, information specifying the destination point of movement of a predetermined part, or information indicating an imitation action of the robot's movement or an instruction action for the robot (action instruction by a gesture such as pointing) by a person or other object (including a simulator that makes a pseudo movement of the robot. Also includes objects on the screen) can be used.

[0032] Note that generally, in a specification document, although a functional description of the control content is shown, specific parameter values are often not shown. In contrast, in an image or the like showing a specific operation such as the robot device as described above, the control system 1000 can estimate the values of specific parameters. Therefore, the control system 1000 can improve the generation accuracy of the execution code D14 by inputting not only information such as a specification document showing a functional description, but also information such as an image showing a specific operation as the input information D11.

[0033] In addition, as a way to indicate the control content in the input information D11, not only the explicit indication method as described above, but also, for example, when there is a product such as a processed product obtained by control, a method of indicating the corresponding control content by showing an image of the product can be cited. In other words, the input information D11 may include not only information directly indicating the control content for the target device 2, but also information indirectly indicated using an image of the product obtained by the control. For example, as the input information D11, a 3D image showing the product or an image obtained by photographing the product from a plurality of directions can be used.

[0034] Also, the input information D11 may include information indicating conditions for the control content. For example, in the processing of a product, even if the processing content is the same, it is conceivable that the wear condition of the tool used for processing changes according to the processing speed. Therefore, as the input information D11, in addition to the information indicating the control content (processing content), information indicating conditions related to the processing speed or the like may be included. Thereby, the control system 1000 can generate a more suitable execution code D14.

[0035] Here, the format of the input information D11 is not particularly limited. For example, the information may be text, an image, audio, data described in a predetermined design language, control description (including source code and information described in a predetermined programming platform language), information described in other platform languages, a control command (including a control instruction, a control signal, a control code, and a controller command), or an execution code. Note that these types of information can be combined as appropriate. In the present disclosure, when the term "text" is used without particular distinction, in addition to what is expressed in text as a natural language, data described in a predetermined design language that cannot be discriminated by a human, control description (including source code and information described in a predetermined programming platform language), information described in other platform languages, control commands (including control instructions, control signals, control codes, and controller commands), and data such as execution codes expressed in text that can be discriminated by a machine may be included.

[0036] The control description D12 includes control-related information described in a predetermined format that can be recognized by the subsequent execution code generation unit 120. The control description D12 is, for example, source code described in a predetermined programming language. Also, the control description D12 may be, for example, a group of commands described in a format (platform language) handled by a predetermined programming platform. Here, the predetermined programming platform may include a no-code programming platform and a low-code programming platform.

[0037] The device information storage unit 110 stores device information D13, which is information about the target device 2. The device information D13 may include, for example, information indicating the functions, performance, structure, dimensions, operations, and / or control methods of the target device 2. Also, the device information D13 may include, for example, information about the programs used for controlling the target device 2. The device information D13 may be, for example, the manual or handling instructions of the target device 2 converted into data. The conversion into data here includes text conversion, image data conversion, data conversion by read-aloud voice, and combinations thereof. The device information D13 is used, for example, as additional information when the learning model unit 100 outputs the control description D12.

[0038] Also, the device information D13 may include information indicating the state of the target device 2. The information indicating the state of the target device 2 may include not only the current state of the target device 2 but also information indicating the past state. For example, the device information D13 may include time-series data of a predetermined data structure indicating the state of the target device 2. The information indicating the state of the target device 2 may be, for example, information output from the target device 2 or information input by the user 1 or another device. The information indicating the state of the target device 2 may include various types of information output from the target device 2 (for example, error information, log information, notification information, etc.). Hereinafter, in this embodiment, the information indicating the state of the target device 2 may be particularly referred to as state information D15.

[0039] When the execution code generation unit 120 receives the control description D12, it generates and outputs an execution code D14, which is a code executable by the target device 2, based on the control description D12. The execution code D14 may be, for example, a group of codes described in machine language. The execution code D14 includes, for example, information used when the target device 2 actually performs control. The execution code D14 may be, for example, information related to control described in a format distinguishable by the target device 2. The execution code generation unit 120 may be, for example, a compiler that converts the control description D12 into the execution code D14.

[0040] The execution code D14 output from the execution code generation unit 120 is input to the target device 2. As a result, the target device 2 operates according to the execution code D14 output from the execution code generation unit 120. The input of the execution code D14 to the target device 2 may be directly input from the execution code generation unit 120, or may be indirectly input via a communication network, other devices (such as servers and various conversion devices), or a person's hand.

[0041] The target device 2 is not particularly limited. Although it is assumed that the target device 2 is a device that can actually execute after receiving the execution code D14, this is not the case if an interface for causing the target device 2, such as a writing device, to read the execution code is included between the target device 2 and the execution code D14.

[0042] The target device 2 is, for example, a PLC, a processing machine, a robot, a radar, a sensor, a camera, a projector, or a communication device. Further, the target device 2 may be, for example, an air conditioner, a refrigerator, a television, lighting, or a washing machine. Further, the target device 2 may be, for example, an elevator, a mobility device, a conveying device, other machinery, or a control device that controls such machinery. Further, the target device 2 may be a device operating in a power generation, transformation, and power storage plant, a water treatment plant, etc., or a control device that controls other devices. Further, the learning model unit 100 may generate an execution code D14. For example, when the control description D12 is an interpreter language and the target device 2 is a device that can receive and execute the control description D12 as it is, the control description D12 can be regarded as the execution code D14. In this case, the execution code generation unit 120 can be omitted. Note that the control description D12 may be converted into an execution code D14 more suitable for the processing of the target device 2 by compilation or optimization processing.

[0043] FIG. 2 is an explanatory diagram showing a configuration example of the learning model unit 100. As shown in FIG. 2, the learning model unit 100 may include a model control unit 101 that operates on the information processing device 10, and a model information storage unit 11 (denoted as a model information DB in the figure) that stores model information D102. Here, the model information storage unit 11 may be configured by a plurality of databases connected via a network.

[0044] The model information D102 includes information on the model. The model information D102 may include, as information on the model, for example, information indicating a correlation between the model input data D101 and the model output data D103. Further, the model information D102 may include, as information on the model, for example, information indicating candidates for the model output data D103. Further, the model information D102 may further include information indicating candidates for the model output data D103 and information indicating the relationship between those candidates as information on the model. Further, the model information D102 may include model parameters, which are information defining the behavior of the learning model, such as constraint conditions, weighting variables, and evaluation functions.

[0045] The model may be, for example, a model machine-learned by supervised learning, reinforcement learning, or unsupervised learning. The model may be, for example, a model obtained by performing learning according to known algorithms and methods such as deep learning, genetic programs, and functional logic programs. Further, the model may be, for example, a model called an NN (Neural Network) model, a CNN (Convolutional Neural Network) model, an RNN (Recurrent Neural Network), a VAE (Variational Autoencoder), a GAN (Generative Adversarial Networks), a diffusion model, a Transformer model, an LLM (Large Language Model), a VLM (Visual Language Model), a BERT (Bidirectional Encoder Representations from Transformers), a GPT (Generative Pre-trained Transformer), or a CLIP (Contrastive Language Image Pre-training). Further, the model may be a rule-based model that obtains an output result by referring to a predetermined table or making a determination based on predetermined conditions. Note that the models described above are not exclusive. For example, LLM, VLM, BERT, and GPT are included in the Transformer model. Further, for example, the Transformer model is included in the NN model. Further, the learning algorithms and models may be a combination of multiple types. The model also includes what is called a multimodal model that is learned by combining multiple different types of data.

[0046] When the model control unit 101 receives the model input data D101, it outputs the model output data D103 corresponding to the model input data D101 based on the model input data D101 and the model information D102. When the model control unit 101 receives the model input data D101, for example, it outputs the model output data D103 corresponding to the model input data D101 using the model indicated by the model information D102.

[0047] The model control unit 101 is realized by, for example, a CPU that operates according to a program provided in the information processing apparatus 10. Hereinafter, the learning model unit 100 may sometimes be referred to as an artificial intelligence unit. Here, the artificial intelligence unit refers to an AI having intelligent functions such as inference and judgment and its operating environment. Therefore, the model control unit 101 may include an AI having intelligent functions such as inference and judgment and its operating environment. The model control unit 101 may be, for example, an AI having a learning model as described above and its operating environment. The model control unit 101 may be an element (module) of the control unit 104 provided in the information processing apparatus 10.

[0048] Also, as shown in FIG. 3, the learning model unit 100 may further include a reference information storage unit 12 (denoted as reference information DB12 in the figure) that stores the model reference information D104. Here, the reference information storage unit 12 may be composed of a plurality of databases connected via a network. The same applies to other storage units (for example, device information storage units) described later.

[0049] The model reference information D104 is information that the model control unit 101 refers to in order to output the model output data. The model reference information D104 may include a history of past model input data and / or a history of past model output data. Also, the model reference information D104 may include information associating the feature amounts included in the past input with the feature amounts included in the output performed on that input. Also, the model reference information D104 may include information on the evaluation of the results output for the past input.

[0050] Also, the model reference information D104 may include information related to the expressions or concepts included in the model input data D101. The model reference information D104 may include, for example, information that associates a specific expression or concept that may be included in the model input data D101 with other expressions or concepts related to that expression or concept. Here, other expressions or concepts related to a certain expression or concept include expressions or concepts that further specify a certain expression or concept, or other expressions or concepts that are recalled based on a certain expression or concept. The model reference information D104 may include, for example, information that associates a specific expression or concept that may be included in the model input data D101 with an expression or concept related to that expression or concept. As an example, the model reference information D104 may include, for example, information that associates a specific expression or concept that may be included in the model input data D101 with information related to that expression or concept. The model reference information D104 may include, for example, information that associates a search key and value extracted from an expression or concept that may be included in the model input data D101. The model reference information D104 may include so-called grounding information. Also, the model reference information D104 may include a so-called knowledge graph that describes real-world entities and the relationships between them. In a knowledge graph, various information is systematically connected and represented in a graph structure.

[0051] Also, the model reference information D104 may include so-called attention information. For example, the model reference information D104 may include information indicating the correlation between an expression or concept that may be included in the model input data D101 and other expressions or concepts. Also, the model reference information D104 may include a feature map having as a feature quantity key information extracted from an expression or concept that may be included in the model output data D103 associated with an expression or concept that may be included in the model input data D101. Also, the model reference information D104 may include information that associates a query extracted from an expression or concept that may be included in the model input data D101 with search key information corresponding to that query.

[0052] In the example shown in FIG. 3, when the model control unit 101 receives the model input data D101, it outputs the model output data D103 based on the model input data D101, the model information D102, and the model reference information D104.

[0053] Note that the learning model unit 100 can also include a search engine that searches for the model reference information D104 or an interface with the search engine instead of the reference information storage unit 12. In such a case, the search range of the search engine may be an external network or a specific network. Here, as one of the external network or the specific network, a database (for example, device information DB, etc.) provided in the control system of the present disclosure can be used.

[0054] When referring to a "learning model", it may refer to a computer algorithm that performs some output based on the learned information for the input information, or the learned information itself. However, when referring to a "learning model" in an operating environment, it often refers to the actual program that operates such a computer algorithm and its operating environment. In the present disclosure, the latter is adopted, and in order to distinguish it from a mere algorithm or a group of learned information, based on the information stored in the model information D102, etc., the actually operating model is called a "learning model". In the control system according to the present disclosure, a learning model unit (particularly the model control unit 101) is provided as corresponding to such a learning model. Therefore, hereinafter, when referring to a "learning model" in the description of the control system, it shall refer to the learning model unit or particularly the model control unit 101 thereof.

[0055] FIG. 4 is an explanatory diagram showing another configuration example of the learning model unit 100. As shown in FIG. 4, the learning model unit 100 may include an input unit 102, an output unit 103, and a control unit 104.

[0056] The input unit 102 receives the model input data D101. The input unit 102 may receive the model input data D101 input by the user 1 or the like. The input unit 102 may receive the model input data D101 that constitutes the time-series data. At this time, the input unit 102 may sequentially receive the model input data D101 that constitutes the time-series data, or may receive the model input data D101 buffered to some extent. Also, the input unit 102 may receive the model input data D101 input from a plurality of input sources. In that case, the input unit 102 may receive the model input data D101 attached with information of the input source (for example, user identifier, attribute information of the user 1, etc.), or the input unit 102 may determine the input source and attach the information of the input source to the model input data D101 before receiving it, or may receive it without any particular processing. The input unit 102 is realized by, for example, various input devices provided in the information processing apparatus 10 (for example, pointing device, keyboard, voice input device, image input device, data reading device, data input device corresponding to various communication interfaces, etc.). Note that the input unit 102 may be realized by an external device of the information processing apparatus 10. In that case, the information processing apparatus 10 only needs to include an interface with the input unit 102.

[0057] The output unit 103 outputs the object generated by the control unit 104. Here, the object includes the model output data D103 or data generated from the model output data D103. Also, when the object generated by the control unit 104 includes information for a plurality of output destinations, the output unit 103 may output the object to the plurality of output destinations. In that case, the output unit 103 may output the same data to the plurality of output destinations, or may output different data for each output destination. The output unit 103 is realized by, for example, various output devices provided in the information processing apparatus 10 (for example, display device, voice output device, image output device, data writing device, data output device corresponding to various communication interfaces, etc.). Note that the output unit 103 may be realized by an external device of the information processing apparatus 10. In that case, the information processing apparatus 10 only needs to include an interface with the output unit 103.

[0058] The control unit 104 operates on the information processing apparatus 10 and includes a preprocessing unit 105 and a postprocessing unit 106 in addition to the above-described model control unit 101.

[0059] The preprocessing unit 105 performs processing for improving the accuracy of the object generated by the control unit 104. The preprocessing unit 105 may, for example, perform addition, modification, or deletion of elements, or conversion (including processing) of data on the model input data D101.

[0060] For example, when the input unit 102 receives the model input data D101, the preprocessing unit 105 may perform modification of elements (including addition and deletion) or conversion (including processing) of data on the model input data D101. Modification of elements or conversion (including processing) of data includes not only change of data format but also change of the expression or concept represented by the data. The modified data by the preprocessing unit 105 is input to the subsequent model control unit 101 as the model input data D101. The processing performed by the preprocessing unit 105 includes so-called prompt shaping for the model control unit 101.

[0061] For example, the preprocessing unit 105 may perform processing for decomposing the model input data D101 into predetermined unit data. Further, the preprocessing unit 105 may perform processing for integrating a plurality of model input data D101, for example. Furthermore, the preprocessing unit 105 may perform modification of elements or conversion of data after decomposing the model input data D101 into predetermined unit data, or may perform modification of elements or conversion of data after integrating a plurality of model input data D101.

[0062] The post-processing unit 106 corrects the object when there is a problem with the object generated by, for example, the control unit 104 (especially the model control unit 101). The post-processing unit 106 may determine whether there is a problem with the object by using, for example, the above-described knowledge graph. For example, the post-processing unit 106 may compare the similarity between the relationship indicated by the knowledge graph, the relationship between the expressions or concepts included in the model input data and the expressions or concepts included in the model output data, and / or the relationship between the expressions or concepts included in the model output data, and determine that there is a problem with the object when it is separated from the relationship indicated by the knowledge graph by a predetermined distance or more.

[0063] Note that among the above-described components, the components other than the model control unit 101 are not essential, and the presence or absence of implementation can be selected as appropriate.

[0064] Also, the model information D102 and other information used by the learning model may be prepared in advance, or may be acquired via a communication network as necessary.

[0065] FIG. 5 is a configuration diagram showing another example of the information processing apparatus 10 as an operating environment of the control unit 104 including the learning model unit 100. The information processing apparatus 10 shown in FIG. 5 may include a control unit 104a including a learning model unit 100 (especially a model control unit 101), an input processing unit 201, an output confirmation unit 202, and a correction confirmation unit 203.

[0066] The input processing unit 201 receives the input information D11 from the input source 1a such as the user 1. The input processing unit 201 also outputs the received input information D11 to the learning model unit 100 as the model input data D101.

[0067] At that time, the input processing unit 201 may output, for example, the input information D11 with elements modified or data converted as the model input data D101. The input processing unit 201 may, for example, remove noise from the input information D11. Also, the input processing unit 201 may, for example, convert qualitative information into quantitative information when the input information D11 contains qualitative information. Further, the input processing unit 201 may, for example, correct the quantity according to the device targeted by the requirements of the input information D11 and its operating environment when the input information D11 contains quantitative information. Also, the input processing unit 201 may, for example, perform so-called grounding processing, that is, change the expression or concept represented by the input information D11 into a more specific expression or concept.

[0068] In addition, when the input information D11 contains unclear or uncertain information, the input processing unit 201 may send an inquiry to the input source. As the inquiry, the input processing unit 201 may output, for example, a message for confirming the input content, a message for proposing an amendment to the input information D11, or a message for requesting re-entry of the input information D11 with different states or expressions. Also, the amendment to the input information D11 may be generated by a correction confirmation unit 203 described later. Hereinafter, information indicating content correction, addition, and cancellation for the input / output data of the learning model after input / output may sometimes be referred to as supplementary information D18. The amendment is an example of the supplementary information D18.

[0069] The output verification unit 202 performs a simulation that emulates the control and state of the target device 2 based on the model output data D103 output from the learning model unit 100. The output verification unit 202 may convert the model output data D103 into control information that matches a predetermined simulator (not shown) capable of simulating the control and state of the target device 2, and then perform the simulation. The output verification unit 202 may have the function of a simulator. When performing the simulation, the output verification unit 202 may utilize the information acquired from the output destination 2a of the model output data D103. Here, the output destination 2a includes the output destination of the information generated from the model output data D103. The information acquired from the output destination 2a may include, for example, the state information D15 and / or the feedback information D16 described later.

[0070] For example, the output verification unit 202 may confirm the state of the target device 2, the state of the system including the target device 2, and / or the state of the workpiece possessed by the target device 2. Also, before performing the operation confirmation, the output verification unit 202 may generate and display an intermediate product that can be understood by a person for the model output data D103 or the information generated based on it. Examples of the intermediate product include the source code for the control program and the operation image of the controller of the target device 2 for the operation command to the target device 2. Further, the output verification unit 202 may display the result of the simulation together with the reliability index of the learning model.

[0071] The following are examples of the reliability index of the learning model. For example, during pre-learning or the like, each time there is an input to the learning model, the result evaluated by a person is accumulated, an evaluation network that learns the input and the evaluation result is provided, and when using the learning model, the input of the learning model is also input to the above-described evaluation network, and the output result thereof may be used as the reliability index.

[0072] Also, for example, during pre-learning or the like, a learner that clusters the output of the learning model is provided, and when using the learning model, the output of the learning model is also input to the above-described learner, and the result of the clustering may be used as the reliability index.

[0073] Also, for example, during pre-training or the like, every time there is an input to the learning model, the results evaluated by a person are accumulated, and an evaluation network that learns the feature amounts of the inputs with high evaluation results is provided. When using the learning model, the input of the learning model is also input to the above-described evaluation network, and the similarity between the feature amount that is the output result thereof and the feature amount of the learning result may be used as a reliability index.

[0074] Also, for example, during pre-training or the like, every time there is an input to the learning model, the results evaluated by a person are accumulated, and a learner that clusters the inputs of the learning model with high evaluation results is provided. When using the learning model, the input of the learning model is also input to the above-described learner, and the clustering result may be used as a reliability index.

[0075] The correction confirmation unit 203 determines the validity of the model output data D103 and / or the model input data D101 using the result of the simulation performed by the output confirmation unit 202. The correction confirmation unit 203, for example, compares the state of the target device 2 indicated by the simulation result with the state of the target device 2 specified by the input information D11, the model output data D103, and / or the model input data D101, and determines whether correct control is being performed, thereby determining the validity of the model output data D103 and / or the model input data D101. The correction confirmation unit 203 may determine that correct control is being performed when the state of the target device 2 indicated by the simulation result matches the state of the target device 2 specified by the model output data D103 and / or the model input data D101. The state of the target device 2 to be compared here is not limited to one.

[0076] Also, the correction confirmation unit 203, for example, checks whether the state or control trajectory of the target device 2 indicated by the simulation result matches the control indicated by the input information D11, or whether it contains content that is prohibited in advance, etc., thereby determining the validity of the model output data D103 and / or the model input data D101.

[0077] Further, the correction verification unit 203 may determine the validity of the model output data D103 and / or the model input data D101 by presenting the simulation result to the input source 1a of the input information D11 and asking for an answer as to whether the desired control is being performed.

[0078] If the correction verification unit 203 determines that the model output data D103 and / or the model input data D101 is incorrect, it may correct the model input data D101. Alternatively, instead of correcting the model input data D101, the correction verification unit 203 may generate supplementary information D18 for the input information D11 and output it to the input source 1a.

[0079] The control system 1000 may have a configuration shown in any of FIGS. 1 to 5, for example, as the operating environment of the learning model unit 100. Similar to the learning model unit 100, in that case, part or all of the configuration may be an internal configuration or an external configuration of the control system 1000.

[0080] Note that the above-described learning model unit 100 and its surrounding configurations are merely examples, and not all components are essential configurations. The presence or absence of implementation may be appropriately selected according to the desired function.

[0081] FIG. 6 is an explanatory diagram showing an example of model learning. As shown in FIG. 6, the model information D102 may be generated, for example, by the model generation unit 107 performing machine learning using the model learning data D105.

[0082] The model generation unit 107 is a processing unit that generates or updates model information D102 based on the input model learning data D105 according to a predetermined algorithm. The model generation unit 107 is realized, for example, by a CPU or the like that operates according to a program provided in the information processing apparatus 20. Here, the algorithm followed by the model generation unit 107 may be a machine learning algorithm corresponding to the learning model, such as supervised learning, reinforcement learning, or unsupervised learning, or may be a known algorithm such as deep learning, genetic programming, or functional logic programming.

[0083] Further, the model generation unit 107 may generate or update the model information D102 for the input model learning data D105 based on the model reference information D104. Further, the model generation unit 107 may generate and update the model information D102 for the input model learning data D105 based on the model output data D103 from the model control unit 101.

[0084] The model learning data D105 is not particularly limited. For example, when supervised learning is used as the learning algorithm, the model learning data D105 may include candidates for the model input data D101 that can be input and candidates for the corresponding model output data D103. Further, the model learning data D105 may include the actually input model input data D101 and / or the actually output model output data D103. By appropriately using the actual model input data D101 and / or the model output data D103, feedback control can be performed. Further, the model learning data D105 may include information obtained from devices or processing units included in the system in which the learning model actually operates.

[0085] The model information D102 generated or updated by the model generation unit 107 is stored in the model information storage unit 11 and provided to the model control unit 101. Alternatively, the model generation unit 107 can also directly output the model information D102 to the model control unit 101.

[0086] For example, the model generation unit 107 may generate the model information D102 using the input model learning data D105 through pre-training before the model control unit 101 uses the model information D102, and store it in the model information storage unit 11.

[0087] The update of the model information D102 by the model generation unit 107 may be a process called so-called FineTune.

[0088] Note that the model generation unit 107 may be included in the control system 1000 or may be included in a system separate from the control system 1000.

[0089] Also, in FIG. 1, the learning model unit 100, the device information storage unit 110, and the device information D13 are shown separately, but the device information storage unit 110 and the device information D13 may be part of the learning model unit 100. That is, the learning model unit 100 may include the device information storage unit 110 and the device information D13. For example, the learning model unit 100 may include the device information storage unit 110 as one of the reference information storage units 12 described later. Also, by using the device information D13 in the model learning phase for learning the model used by the learning model unit 100, the device information D13 may be pre-embedded in the model. In that case, the device information storage unit 110 may be omitted.

[0090] Further, part or all of the learning model unit 100 may be an internal configuration of the control system 1000 or an external configuration of the control system 1000. When it is an external configuration of the control system 1000, the control system 1000 only needs to be provided with an interface capable of exchanging information with an external system including part or all of the learning model unit 100 instead of part or all of the learning model unit 100. For example, the control system 1000 may use the model information storage unit 11 called the core of the learning model as an external configuration. Further, for example, the control system 1000 may use the model information storage unit 11 called the core of the learning model and the model control unit 101 responsible for the algorithm of the model as external configurations.

[0091] Hereinafter, in the control system 1000, in order to distinguish between the model control unit 101 responsible for the algorithm of the learning model and the part that issues a request to such a model control unit 101 and obtains a response, the part that performs the latter process may be referred to as the "model processing unit". More specifically, the model processing unit corresponds to the part other than the model control unit 101 among the information processing device 10, the control unit 104, or the control unit 104a described above. Note that the model processing unit may be realized by, for example, an OS (Operating System), a prompt application (and the control unit serving as its operating environment) that calls a learning model application operating on the information processing device 10 when the model control unit 101 exists in the internal environment. Note that the model processing unit may be realized by, for example, a browser, a client application (and the control unit serving as its operating environment) operating on the information processing device 10 when the model control unit 101 exists in the external environment.

[0092] Note that the configurations of the learning model and the information processing device as its operating environment, and the relationship between the learning model and the control system including the same are the same in other embodiments.

[0093] In this embodiment, the input information D11 corresponds to the model input data D101. Also, the control description D12 corresponds to the model output data D103. The learning model unit 100 (particularly, the model control unit 101) may be configured to output the control description D12 corresponding to the input information D11 based on, for example, the model information D102 and, if necessary, the model reference information D104 when receiving the input information D11.

[0094] Also, in such a case, the model generation unit 107 provided corresponding to the learning model unit 100 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D11 that can be input to the model control unit 101 to generate or update the model information D102. Further, the model generation unit 107 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D11 that can be input to the model control unit 101 and candidates for the control description D12 corresponding thereto to generate or update the model information D102.

[0095] In this embodiment, the learning model unit 100 may be, for example, a language learning model such as an LLM that inputs natural language to obtain an output result and its operating environment. Also, the learning model unit 100 may be, for example, an image learning model such as a VLM that inputs an image to obtain an output result and its operating environment. Also, the learning model unit 100 may be, for example, a multimodal model that inputs natural language and an image to obtain an output result and its operating environment. In that case, the input information D11 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data, a video that is a combination of audio data and image data). Note that the learning model used in this embodiment is not limited to the models described above.

[0096] In this embodiment, the input information D11 received by the control system 1000 can be information regarding a request in the working environment, here the environment in which the target device 2 operates (here, the control content required for the target device 2). Therefore, the input information D11 received by the control system 1000 can be said to be an example of first information indicating a request in the working environment. Also, the control description D12 and the execution code D14 can be said to be information used for such work (work related to the control of the target device 2) corresponding to such input information D11. Hereinafter, the control description D12 output to a predetermined output destination from the operating environment of the learning model into which the model input data based on the input information D11 is input may be referred to as second information.

[0097] Here, in the relationship between the input information D11 and the model input data, when referring to the model input data based on the input information D11, it may include the input information D11 itself, the input information D11 converted into a format matching the input of the learning model, and the input information D11 supplemented. Also, in the relationship between the model output data and the second information, when referring to the second information based on the model output data, it may include the model output data itself, the model output data converted into a format matching the input of the output destination, and the model output data supplemented. The same applies to the relationship between the input / output information and the model input / output data in other embodiments.

[0098] Next, the operation of the control system 1000 of this embodiment will be described. FIG. 7 is a flowchart showing an operation example of the control system 1000.

[0099] In the example shown in FIG. 7, first, the control system 1000 receives the input information D11 (step S110). For example, the above-described input unit 102 or input processing unit 201 may receive the input information D11. The received input information D11 is input to the learning model unit 100 as the model input data D101.

[0100] In step S110, the control system 1000 may receive a plurality of input information D11. Further, the control system 1000 may interact with the user 1, that is, repeatedly input and output information related to the input information D11 with the user 1, and receive the input information D11 that better meets the requirements of the user 1.

[0101] Next, the control system 1000 performs a generation process of the control description D12 using the learning model unit 100 (step S111). In step S111, the learning model unit 100 generates and outputs a control description D12 corresponding to the input input information D11. For example, the learning model unit 100 (more specifically, the model control unit 101) outputs a control description D12 corresponding to the input information D11 based on the model information D102, the input input information D11, and the model reference information D104 including the device information D13 as needed. The learning model unit 100 may generate the control description D12 of the text data from the input input information D11 using, for example, a learning model capable of generating text data.

[0102] In step S111, further, the learning model unit 100 (more specifically, the preprocessing unit 105 or the input processing unit 201) may perform addition, change, or deletion of elements, or data conversion (including processing) on the input input information D11 in order to improve the accuracy of the control description D12 before the processing of the model control unit 101. Also, in step S111, further, the learning model unit 100 (more specifically, the postprocessing unit 106) may determine whether there is a problem in the control description D12 after the processing of the model control unit 101, and perform a process of correcting the control description D12 when it is determined that there is a problem.

[0103] The control description D12 output from the learning model unit 100 is input to the execution code generation unit 120. When the control description D12 is input, the execution code generation unit 120 generates an execution code D14 based on the input control description D12 (step S112).

[0104] Next, the execution code D14 generated by the execution code generation unit 120 is input to the target device 2 (step S113). As already described, the input of the execution code D14 to the target device 2 may be directly input from the control system 1000 (more specifically, the execution code generation unit 120), or may be indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or a person's hand.

[0105] Thereby, the target device 2 operates according to the input execution code D14.

[0106] If there is a change in the state of the target device 2, such as the target device 2 being controlled as a result of the control system 1000 outputting the execution code D14, the control system 1000 may acquire the state information D15 (step S114). The acquired state information D15 is stored, for example, in the device information storage unit 110 as part of the device information D13. The control system 1000 may update the device information D13 stored in the device information storage unit 110 using the acquired state information D15, for example. Further, the control system 1000 may output the acquired state information D15 to the user 1, the learning model unit 100, or another device (not shown) as information indicating the control result, for example. Note that if the control system 1000 does not use the state information D15, the process of step S114 may be omitted.

[0107] The control system 1000 may repeat the processes of steps S110 to S114 a plurality of times (for example, until the desired control of the target device 2 is completed).

[0108] Note that the control system 1000 may output the control description D12 to the operation terminal of the user 1 or the like, and after the user 1 confirms the content, subsequent processing (such as code generation by the execution code generation unit 120) may be executed by the operation of the user 1.

[0109] The state information D15 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on the input state information D15, for example.

[0110] As described above, according to the present embodiment, the execution code D14 can be generated from the input information D11 input from the user 1 without the user 1 creating the control description D12, so that the efficiency of the work of controlling the target device 2 can be improved.

[0111] Also, in the present embodiment, since the input information D11 may be text, an image, audio, or a combination thereof that explicitly or implicitly indicates the control content for the target device 2, it is possible to further suppress the labor of inputting the input information D11 and improve the efficiency of the work of controlling the target device 2.

[0112] Also, according to the present embodiment, since the control description D12 can be generated using the learning model from the input information D11, even if the user 1 does not know information such as the detailed specifications of the target device 2 and the specifications of the control description D12 for controlling the target device 2, the control description D12 corresponding to the input information D11 can be generated, so that the performance of the work of controlling the target device 2 can be improved. Here, the improvement in the performance of the work of controlling the target device 2 includes high precision in controlling the target device 2.

[0113] Also, in the present embodiment, since the state information D15 obtained after controlling the target device 2 based on the input information D11 can be used for generating the next control description D12 and the like, the performance of the work of controlling the target device 2 can be further improved.

[0114] In the above example, only one target device 2 is shown. However, there may be a plurality of target devices 2 to be controlled by the control system 1000. In such a case, for example, the input information D11 may include information that can identify the target device 2, or the input side (input unit 102, preprocessing unit 105, input processing unit 201) of the learning model unit 100 may perform a process of identifying the target device 2 based on the input information D11, or the learning model unit 100 may output the control content for which the target device 2 has been identified as a result of learning.

[0115] Note that a number of programming languages are used in the target device 2. For example, there are five programming languages defined by international standards in PLCs, several programming languages exist even in a single machine tool manufacturer, and a number of programming languages exist depending on the model of the robot. On the other hand, in the control system 1000, the user 1 can input information that can identify the target device 2 such as a specification document to the learning model unit 100 by including it in the input information D11, and the learning model unit 100 can also refer to the device information D13. Thereby, in the control system 1000, the learning model unit 100 can appropriately select the programming language required for the target device 2 and generate the control description D12, and the program conversion work in the subsequent stage can be omitted.

[0116] Also, even when the learning model unit 100 does not support all programming languages, by adding a conversion unit after the learning model unit 100, the control description D12 from the learning model unit 100 can be changed to the required program by the conversion unit and output to the execution code generation unit 120.

[0117] In addition, the learning model unit 100 outputs control description information related to the control description D12, such as the document string or annotation regarding the generated control description D12, or the information referred to when generating the control description D12, to the user 1 side, and the user 1 may check the control description information displayed on a display device (not shown). As a result, it becomes easier for the user 1 to check the content of the control description D12 generated by the learning model unit 100, and it also becomes easier for the learning model unit 100 itself to perform self-evaluation. This can be realized, for example, when the user 1 outputs the input information D11 to the learning model unit 100, by making a request for output of control description information in addition to the request for generation of the control description D12.

[0118] Also, the user 1 may determine whether the control description D12 satisfies the requirements of the input information D11 by determining whether the control description information satisfies the requirements of the input information D11 based on the control description information from the learning model unit 100. And when the user 1 determines that the control description information does not satisfy the requirements of the input information D11, the user 1 corrects the input information D11 and re-enters it into the learning model unit 100. As a result, in the control system 1000, the accuracy of the control description D12 generated by the learning model unit 100 can be improved. Also, the user 1 may set the display device so as not to display the control description information that does not satisfy the requirements of the input information D11 among the control description information from the learning model unit 100, or may set it to display only a predetermined number of the latest ones. As a result, it becomes easier for the user 1 to check the control description information closer to the requirements.

[0119] Note that in the above, the case where the input information D11 is input to the learning model unit 100 and the control description D12 generated by the learning model unit 100 is input to the execution code generation unit 120 is shown. On the other hand, in the learning model unit 100, cases such as slow response or cost may occur.

[0120] Therefore, User 1 may determine whether the input information D11 matches a specific rule, and based on the determination result, determine whether to input the input information D11 into the learning model unit 100.

[0121] For example, as the above rule, a rule showing the correspondence between existing input information and control descriptions can be used. In this case, when User 1 determines that the input information D11 does not match the above rule, that is, when it is determined that there is no similar input information in the above rule for the input information D11, this input information D11 is output to the learning model unit 100. On the other hand, when User 1 determines that the input information D11 matches the above rule, that is, when it is determined that there is similar input information in the above rule for the input information D11, the control description corresponding to the corresponding existing input information is extracted, and the control description is input to the execution code generation unit 120. That is, in this case, the input information D11 is not input to the learning model unit 100. Further, the execution code generation unit 120 generates an execution code D14 based on the above control description from User 1. Thereby, when the generated input information D11 is similar to the input information for which the correspondence with the control description has already been determined, the control system 1000 can be processed without going through the learning model unit 100, and the processing can be performed more quickly and at a lower cost.

[0122] Also, for example, as the above rule, a rule indicating input information that can be directly input to the execution code generation unit 120 can be used. In this case, when the user 1 determines that the input information D11 does not match the above rule, the user 1 outputs this input information D11 to the learning model unit 100. On the other hand, when the user 1 determines that the input information D11 matches the above rule, the user 1 inputs this input information D11 to the execution code generation unit 120. That is, in this case, the input information D11 is not input to the learning model unit 100. Further, the execution code generation unit 120 generates an execution code D14 based on the above input information D11 from the user 1. Thereby, when the generated input information D11 is input information that can be directly input to the execution code generation unit 120, the control system 1000 can be processed without going through the learning model unit 100, and the processing can be performed more quickly and at a lower cost.

[0123] Note that the user 1 may make the above determination based on a rule such as referring to a preset table, or by using a learning model unit different from the learning model unit 100. Although the processing ability of the above-mentioned other learning model unit is lower than that of the learning model unit 100, a lightweight and high-speed learning model unit or the like is assumed. Note that when the above determination is difficult, the other learning model unit may request the learning model unit 100 to make a determination, that is, the other learning model unit and the learning model unit 100 may jointly perform processing. Note that in other embodiments as appropriate, the same method, particularly the method described in Modification Example 3-1, FIGS. 16 and 17, may be adopted.

[0124] Modification Example 1-1. Next, a modification example of the control system 1000 will be described. FIG. 8 is a configuration diagram showing an example of a control system 1000a which is a modification example of the control system 1000 according to the present embodiment. Note that the same elements as those in the control system 1000 are denoted by the same reference numerals and the description thereof is omitted.

[0125] In the control system 1000a shown in FIG. 8, the output from the learning model unit 100 is input to the subsequent execution code generation unit 120 after being confirmed by the user 1.

[0126] In the present embodiment, the user 1 can check the control description D12 output from the learning model unit 100 and input the input information D11 based on the confirmation result. Further, in addition to the control description D12 output from the learning model unit 100, the user 1 can check the feedback information D16 from the execution code generation unit 120 and / or the target device 2, and input the input information D11 based on the confirmation results thereof. At this time, in addition to inputting the input information D11 with new content, the user 1 may input the input information D11 indicating the correction, addition, or cancellation of the already input content. At this time, the input information D11 can include a command for the learning model unit 100. For example, the user 1 may input, as the input information D11, a command for removing a defect included in the input information D11 or a defect included in the output control description D12, together with the feedback information D16. Here, the command for removing the defect includes an input for searching for the cause of the defect and a solution to the defect.

[0127] The feedback information D16 may include a response to the request returned from the processing unit when a control is requested from the processing unit downstream of the learning model unit 100. Further, the feedback information D16 may include information obtained from the processing unit after requesting control from the processing unit downstream of the learning model unit 100. For example, the feedback information D16 may include a response to the request returned from the execution code generation unit 120 when the control description D12 is input to the execution code generation unit 120 to request generation of the execution code D14. For example, the execution code D14 generated by the execution code generation unit 120 may be included in the feedback information D16. Further, the feedback information D16 may include a response to the request returned from the target device 2 when the execution code D14 is input to the target device 2 to request execution of the code. The state information D15 may be included in the feedback information D16. The feedback information D16 may be output directly to the user 1 or may be output to the user 1 via an output device (not shown) provided in the execution code generation unit 120 or the control system 1000.

[0128] Also, the feedback information D16 can include information for determining whether the control requested from the processing unit downstream of the learning model unit 100 is correctly executed in the processing unit. The information is not limited to the information directly obtained from the processing unit. For example, it may be information obtained from another person, device, network, or AI (all not shown). The feedback information D16 can include, for example, analysis information for determining whether the execution code D14 can correctly execute the target control, such as the execution time or control trajectory information. The user 1 can, for example, also instruct the learning model unit 100 to control the timing of the flow or adjust the lead time in the control description D12 based on such information included in the feedback information D16.

[0129] Also, for example, User 1 may perform multiple exchanges with the learning model unit 100 using the feedback information D16, and each time, determine the validity (presence or absence of problems) of the output control description D12. When User 1 determines that there are no problems with the control description D12, User 1 may output the control description D12 to the execution code generation unit 120.

[0130] The acquisition of the feedback information D16 can be performed, for example, in step S114 described above.

[0131] In addition, in FIG. 8, an example is shown in which User 1 inputs the control description D12 to the execution code generation unit 120. However, the input of the control description D12 to the execution code generation unit 120 can also be performed by the learning model unit 100 that has received an instruction from User 1.

[0132] In this example, the control description D12 may include, for example, a description corresponding to low-code or no-code.

[0133] In this example, the exchange of information between User 1 and the learning model unit 100 may be performed, for example, via a terminal provided by User 1, or may be performed via a user interface (for example, the input unit 102) provided by the information processing apparatus 10 in which the learning model unit 100 operates.

[0134] Also, the update of the input information D11 in this example may be configured to be performed not by User 1 but on the control system 1000 side (for example, the modification confirmation unit 203, etc.).

[0135] In addition, the feedback information D16 may be input to the learning model unit 100. The feedback information D16 input to the learning model unit 100 is used, for example, for additional learning of the learning model unit 100. The learning model unit 100 may update the model information D102 and / or the model reference information D104 based on the input feedback information D16.

[0136] Regarding other points, it may be the same as other control systems according to this embodiment.

[0137] As described above, in this modification example, the user 1 can correct the input information D11 while checking the control description D12 output from the learning model unit 100 and communicating with the learning model unit 100 about additional instructions, bug consultations, etc. Therefore, the accuracy of the output control description D12 can be improved. As a result, the efficiency and performance of the work of controlling the target device 2 can be improved.

[0138] Modification Example 1-2. Next, a second modification example of the control system 1000 will be described. FIG. 9 is a configuration diagram showing an example of a control system 1000b which is a modification example of the control system 1000. Note that the same elements as those in the control system 1000 and the control system 1000a are denoted by the same reference numerals and the description thereof is omitted.

[0139] In the control system 1000b shown in FIG. 9, the learning model unit 100 is different in that it returns an inquiry D17 to the user 1. Examples of the inquiry D17 include, for example, those that re-question unclear or uncertain input information D11, those that ask for solutions, and those that request re-input to those with changed states or expressions. The learning model unit 100 may output an inquiry D17 to the user 1 that requests input of more specific information along with the presentation of the reference location as a re-question of unclear or uncertain input information D11. Also, the learning model unit 100 may output an inquiry D17 to the user 1 that gives candidates for solutions as options along with the presentation of the reference location as an inquiry for solutions. Further, the learning model unit 100 may output an inquiry D17 to the user 1 that asks for the correctness along with the information of the most likely solution as an inquiry for solutions. Additionally, the learning model unit 100 may generate an intermediate control description which is an intermediate control description that is once easy for people to understand, and output an inquiry D17 that asks for its correctness to the user 1 together with the generated intermediate control description.

[0140] The output of Inquiry D17 may be performed, for example, after the above-described step S110.

[0141] When the learning model unit 100 receives a response from the user 1 to Inquiry D17, it may update the input information D11 or determine the interpretation (semantic association) of the input information D11.

[0142] For example, when the input information D11 includes an image, there is often a lot of unclear content as to what is specifically being referred to, such as "this," "that," "this one," or "that one." When the learning model unit 100 detects such unclear content, it may make an inquiry to the user 1 to resolve the unclear content. Alternatively, the learning model unit 100 may select the most likely content for the unclear content, and then update the input information D11 or determine the interpretation of the input information D11. Also, the learning model unit 100 may inquire of the user 1 about the validity of the selected most likely content, and update the input information D11 or determine the interpretation of the input information D11 based on the selection determined by the user 1 to be valid. Also, when there are multiple options for the most likely content for the unclear content, the learning model unit 100 may make a case-by-case analysis and update the input information D11 or determine the interpretation of the input information D11 with each option.

[0143] Note that the inquiry method by the learning model unit 100 is not limited to the above, and various methods can be cited.

[0144] Also, for example, when the specification does not show specific processing, the user 1 may have the learning model unit 100 extract the specific processing in a template based on information such as the device information D13. In this case, for example, first, user 1 causes the learning model unit 100 to extract, using a template, control descriptions that are close to the content of the specification based on information such as device information D13. At this time, the learning model unit 100 may be instructed to present information such as a dog string related to the extracted control description or the device information D13 itself that was referenced to user 1. Next, user 1 causes the learning model unit 100 to extract control descriptions of parts that can be discriminated in the specification from the extracted control descriptions. Next, user 1 checks the correctness of control description D12 obtained by synthesizing the control descriptions extracted by the learning model unit 100.

[0145] The processing of the learning model unit 100 described above in this example can also be implemented, for example, as a part of the function of the input unit 102 or the preprocessing unit 105 of the learning model unit 100, or the input processing unit 201 (not shown) provided in the information processing apparatus 10.

[0146] As described above, in this modification example, for the input input information D11, an inquiry D17 is output to user 1, and based on the answer, the input information D11 is updated or the interpretation of the input information D11 is determined, so that the uncertainty of the input information D11 can be eliminated. As a result, the accuracy of the output control description D12 can be improved, and thus the efficiency and high performance of the work of controlling the target device 2 can be achieved.

[0147] Note that in the above, the case where the learning model unit 100 returns an inquiry D17 to user 1 was shown. However, it is not limited to this. When the control system 1000 includes the user terminal 1, the user terminal 1 may return an inquiry to user 1. In this case, when the user terminal 1 receives an answer from user 1 to the inquiry, it may update the input information D11 or determine the interpretation of the input information D11.

[0148] Also, in the above, the case where the configuration of the control system 1000b is deformed based on the configuration of the control system 1000 was shown. However, it is not limited to this. The configuration of the control system 1000b may be deformed based on the configuration of the control system 1000a, and the same effect as described above can be obtained.

[0149] Modification Examples 1-3 Next, a third modification example of the control system 1000 will be described. FIG. 10 is a configuration diagram showing an example of a control system 1000c which is a modification example of the control system 1000. Note that the same elements as those in the control systems 1000, 1000a, and 1000b are denoted by the same reference numerals and the description thereof will be omitted.

[0150] The control system 1000c shown in FIG. 10 further includes a state acquisition unit 130. The state acquisition unit 130 acquires feedback information D16 indicating a processing result or state information D15 indicating the state of the device after processing from the processing destinations of the control description D12 output from the learning model unit 100 and the execution code D14 generated therefrom. Here, the feedback information D16 or the state information D15 can include information for determining whether the execution code D14 can correctly execute the target control, such as execution time or control trajectory information. Further, the state acquisition unit 130 may acquire the control description D12 output from the learning model unit 100.

[0151] The state acquisition unit 130 may, for example, input the acquired information to the learning model unit 100. Further, the state acquisition unit 130 may, for example, update the device information D13 based on the acquired information. Further, the state acquisition unit 130 may, for example, generate information for supplementing (including adding, modifying, or canceling) the input information D11 based on the acquired information, and input it to the learning model unit 100 as supplementary information D18. Further, the state acquisition unit 130 may, for example, generate information for supplementing (including adding, modifying, or canceling) the control description D12 based on the acquired information, and input it to the learning model unit 100 as supplementary information D18.

[0152] The state acquisition unit 130 may, for example, generate a control command with new content, or a command indicating addition, modification, or cancellation of the content indicated by the input information D11 that has already been input, as supplementary information D18, and input it to the learning model unit 100. Further, the state acquisition unit 130 may, for example, input, as supplementary information D18, a command for removing a defect included in the input information D11 or a defect included in the output control description D12, together with the acquired information, to the learning model unit 100.

[0153] The state acquisition unit 130 may, for example, determine whether the acquired information indicates information on normal processing or a normal state at the processing destination, and if not, input, to the learning model unit 100, supplementary information D18 indicating modification, addition, or cancellation of the content indicated by the input information D11 that has already been input, together with the acquired information.

[0154] The learning model unit 100 may update the model information D102 and / or the model reference information D104, for example, based on the input information (state information D15, feedback information D16, supplementary information D18, etc.).

[0155] The generation of the supplementary information D18 may be performed, for example, in step S115 described above. Further, the output destination of the supplementary information D18 may include not only the learning model unit 100. The control system 1000 may output the supplementary information D18 generated by the state acquisition unit 130 to the user 1 or another device (not shown).

[0156] Further, the state acquisition unit 130 may obtain the operation result by a simulator (not shown) of the target device 2 or the operation result in the debug mode of the target device 2 without actually operating the target device 2. The debug mode of the target device 2 refers to a mode in which the execution code is executed on the control board of the target device 2, but actual device control is not performed and only the internal state is updated, and is also called an idle operation mode. By using the debug mode, the execution code D14 can be safely tested in a state close to actual control on the target device 2.

[0157] Not limited to this modification example, as a method for determining the validity of the control description D12 output from the learning model unit 100 and the execution code D14 generated from the control description D12 without actually operating the target device 2, the execution code generation unit 120 may be connected so that the target device 2 and the simulator can be switched as the output destination of the execution code D14. The simulator includes those that operate the icon of the target device 2 in the extended reality space. Also, when outputting the execution code D14 to the target device 2, the execution code generation unit 120 may attach information indicating whether to execute in the normal mode or the debug mode.

[0158] For example, when a compilation error occurs during compilation in the execution code generation unit 120, the state acquisition unit 130 acquires feedback information D16 indicating that fact and inputs it to the learning model unit 100. As a result, in the learning model unit 100, the control description D12 can be corrected to eliminate this compilation error. Also, for example, when the target device 2 does not operate with the execution code D14 generated by the execution code generation unit 120, the state acquisition unit 130 acquires state information D15 indicating that fact and inputs it to the learning model unit 100. As a result, in the learning model unit 100, the control description D12 can be corrected so that the target device 2 operates.

[0159] The processing of the state acquisition unit 130 in this example can also be implemented as part of the functions of, for example, the input unit 102, the preprocessing unit 105, and the postprocessing unit 106 of the learning model unit 100, or the input processing unit 201, the output confirmation unit 202, and the correction confirmation unit 203 (none of which are shown) provided in the information processing apparatus 10.

[0160] Regarding other points, it may be the same as other control systems according to this embodiment.

[0161] As described above, in this modified example, for the input information D11 that is input, the state acquisition unit 130 acquires feedback information D16 indicating a processing result or state information D15 indicating the state of the device after processing from the target device 2 that is the output destination of the model output data D103 and / or information generated based thereon, or from the execution code generation unit 120, and issues supplementary information D18 to the learning model unit 100 as appropriate based on the acquired information. Thereby, it is possible to improve the accuracy of the control description D12, and as a result, to improve the efficiency and performance of the work of controlling the target device 2.

[0162] Note that, in the above, the case where the state acquisition unit 130 outputs the acquired information or information based on the information to the learning model unit 100 or the like has been shown. However, not limited to this, when the control system 1000 includes the user terminal 1, the state acquisition unit 130 may output the acquired information or information based on the information to the user terminal 1. In this case, the user terminal 1 may update the input information D11 based on the input information such as the state information D15, the feedback information D16, or the supplementary information D18, for example.

[0163] Also, in this modified example, for example, a person and a machine (the state acquisition unit 130) can cooperate to improve the accuracy of the input to the learning model unit 100, which can also contribute to reducing the work load of the user 1.

[0164] Also, in the above, the case where the configuration of the control system 1000c is modified based on the configuration of the control system 1000 has been shown. However, not limited to this, the configuration of the control system 1000c may be modified based on the configuration of the control system 1000a or the control system 1000b, and the same effect as above can be obtained. Note that, in the case of a configuration where the user 1 checks the output from the learning model unit 100 as in the control system 1000a, the state acquisition unit 130 may not acquire the control description D12 output from the learning model unit 100.

[0165] As described above, according to the first embodiment, the control system 1000 includes a user terminal 1 that generates input information D11 including information indicating the control content required for the target device 2 in response to an operation by the user 1, and a control description D12 generated by the learning model unit 100 that generates the control description D12 based on the input information D11 generated by the user terminal 1, or a state acquisition unit 130 that acquires information regarding at least any one of the operations of the target device 2 and feeds back the information to at least one of the user terminal 1 or the learning model unit 100. Further, according to the first embodiment, the control system 1000 includes an execution code generation unit 120 that generates an execution code D14 for the target device 2 based on the control description D12 generated by the learning model unit 100. The learning model unit 100 outputs the generated control description D12 to the execution code generation unit 120. The state acquisition unit 130 acquires information regarding at least any one of the control description D12 generated by the learning model unit 100, the execution code D14 generated by the execution code generation unit 120, or the operation of the target device 2, and feeds back the information to at least one of the user terminal 1 or the learning model unit 100. Further, according to the first embodiment, the control system 1000 includes an execution code generation unit 120 that generates an execution code D14 for the target device 2 based on the control description D12 generated by the learning model unit 100. The learning model unit 100 outputs the generated control description D12 to the user terminal 1. The user terminal 1 outputs the control description D12 output by the learning model unit 400a to the execution code generation unit 120 in response to an operation by the user 1, or causes the learning model unit 400a to output the control description D12 to the execution code generation unit 120. The state acquisition unit 130 acquires information regarding at least any one of the execution code D14 generated by the execution code generation unit 120 or the operation of the target device 2, and feeds back the information to at least one of the user terminal 1 or the learning model unit 100. Further, according to the first embodiment, the learning model unit 100 uses the information stored in the device information storage unit 110 that stores information about the target device 2 as additional information when generating the control description D12. Further, according to the first embodiment, the input information D11 generated by the user terminal 1 is at least one of a document string explaining the specifications, a specification document, or a specification document, design document, operation instruction, control code, or source code applied to other devices. As a result, the control system 1000 according to the first embodiment can improve the accuracy of the control description D12, and thus improve the efficiency and performance of the work of controlling the target device 2.

[0166] Further, according to the first embodiment, when the user terminal 1 determines that the generated input information D11 does not match the rule, the user terminal 1 outputs the generated input information D11 to the learning model unit 100, and when it is determined that the generated input information D11 matches the rule, the user terminal 1 outputs the generated input information D11 or the corresponding control description to the execution code generation unit 120. As a result, the control system 1000 according to the first embodiment can eliminate the need for the processing of the learning model unit 100 when the input information D11 matches the existing rule, and can perform the processing at a higher speed and lower cost.

[0167] Further, according to the first embodiment, the control method includes the user terminal 1 generating input information D11 including information indicating the control content required for the target device 2 in response to an operation by the user 1, and the state acquisition unit 130 obtaining information related to at least one of the control description D12 generated by the learning model unit 100 that generates the control description D12 based on the input information D11 generated by the user terminal 1, or the operation of the target device 2, and feeding back the information to at least one of the user terminal 1 or the learning model unit 100. As a result, the control method according to the first embodiment can improve the accuracy of the control description D12, and thus improve the efficiency and performance of the work of controlling the target device 2.

[0168] Embodiment 2. Next, Embodiment 2 will be described. In this embodiment, an example of assisting the work related to the control of the target device using a learning model will be described.

[0169] Hereinafter, for example, in a factory, it is considered to control various control devices such as PLCs, processing machines, robots, sensors, and control devices for other machines such as transport devices. A skilled worker may be familiar with various control devices and control methods for complex control devices, but there may be cases where a worker who is not so due to rearrangement or the like needs to control the control device. In addition, when a new control device (including version upgrade) is introduced, it is necessary to inform all workers of the control method corresponding to the new control device, and if the notification is not sufficient, there is a risk of leading to mistakes.

[0170] In such a case, even if one does not know a specific control method, for example, a control command, control signal, control code, or command to the controller corresponding to the control device to be performed on the control device, it is preferable because it leads to improvement in work efficiency and high performance and enables reliable desired control.

[0171] Note that the scene of controlling the device is not limited to within the factory, and the application scene of this embodiment is not limited to within the factory either.

[0172] FIG. 11 is a configuration diagram showing an example of a control system 2000 according to Embodiment 2. The control system 2000 shown in FIG. 11 is a control system for controlling a device using a learning model, and includes a learning model unit 200 and a device information storage unit 210 (denoted as a device information DB in the figure).

[0173] When the input information D21 is input, the learning model unit 200 outputs a control command D22. When the input information D21 is input, for example, the learning model unit 200 outputs a control command D22 based on the model information D102. The configuration of the learning model unit 200 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0174] In this embodiment, the learning model unit 200 is a model and its operating environment configured to output a control command D22 corresponding to the input information D21 when the input information D21 is input. Further, the learning model unit 200 may be a model and its operating environment configured to generate and output the control command D22 based on the input information D21, the device information D23, and other information that can be referenced in the learning model unit 200 when the input information D21 is input.

[0175] In this embodiment, the input information D21 includes information indicating the control content for the target device 2. The input information D21 may be, for example, text, an image, voice, or a combination thereof indicating the control content for the target device 2. The input information D21 may be, for example, text, an image, voice, or a combination thereof indicating a plurality of control contents for the target device 2. Further, the input information D21 may include information indicating control contents performed continuously in time, and in that case, it may be time-series data having a predetermined data structure including text, an image, voice, or a combination thereof indicating the control contents as described above. It is assumed that the method of indicating the control content matches the input format of the model used by the learning model unit 200, but this is not the case when error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 200.

[0176] The method of indicating the control content in the input information D21 may be the same as that in the first embodiment, for example. For example, after specifying the control to be performed on the target device 2, the value of the parameter for performing the control or the state after the control may be specified. In that case, the input information D21 may include, for example, information specifying the control and information indicating the value of the parameter for performing the control or the state after the control. Further, the input information D21 may include not only information directly indicating the control content for the target device 2 but also information indirectly indicating the control content using the operation content corresponding to the control content, the actions and words of the user 1, an image of the target device 2, or a similar control command in other models, etc.

[0177] The control command D22 includes information regarding the control of the target device 2, which is indicated in a predetermined format that can distinguish the target device 2 or the interface that requests control of the target device 2. The control command D22 may include information indicating a control request to the target device 2. The control command D22 is, for example, a control instruction, a control signal, or a control code for the target device 2. Also, the control command D22 may be, for example, a command described in a format handled by a predetermined controller corresponding to the target device 2.

[0178] The device information storage unit 210 stores device information D23, which is information regarding the target device 2. The handling of the device information storage unit 210 and the device information D23 is basically the same as that of the device information storage unit 110 and the device information D13 in the first embodiment. Note that the device information D23 in the present embodiment may include, for example, information used for the control of the target device 2. The device information D23 is used, for example, as additional information when the learning model unit 200 outputs the control command D22. Hereinafter, in the present embodiment, information indicating the state of the target device 2 may be particularly referred to as state information D25.

[0179] In the present embodiment, the learning model unit 200 may be, for example, a language learning model such as an LLM that inputs natural language and obtains an output result, and its operating environment. Also, the learning model unit 200 may be, for example, an image learning model such as a VLM that inputs an image and obtains an output result, and its operating environment. Also, the learning model unit 200 may be, for example, a multimodal model that inputs natural language and an image and obtains an output result, and its operating environment. In that case, the input information D21 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data or a video that is a combination of audio data and image data). Note that the learning model used in the present embodiment is not limited to the models described above.

[0180] In this embodiment, for the sake of simplicity of explanation, components provided corresponding to the learning model unit 100 may be used as they are to explain the components provided corresponding to the learning model unit 200. However, it should be noted that they are provided corresponding to the learning model unit 200. The same applies to other embodiments.

[0181] In this embodiment, the input information D21 corresponds to the model input data D101. Also, the control command D22 corresponds to the model output data D103. The learning model unit 200 (particularly, the model control unit 101) may be configured to output, for example, a control command D22 corresponding to the input information D21 based on the model information D102 and, if necessary, the model reference information D104 when receiving the input information D21.

[0182] Also, in such a case, the model generation unit 107 provided corresponding to the learning model unit 200 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D21 that can be input to the model control unit 101 to generate or update the model information D102. Further, the model generation unit 107 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D21 that can be input to the model control unit 101 and candidates for the control command D22 corresponding thereto to generate or update the model information D102.

[0183] The symbol D26 is feedback information indicating the control result in the target device 2. Also in this embodiment, the state information D25 and / or the feedback information D26 may be acquired from the output destination of the model output data D103 and / or the information generated based thereon. The control system 2000 may output, for example, the acquired state information D25 and / or feedback information D26 as information indicating the control result to the user 1, the learning model unit 200, or another device (not shown). Further, the control system 2000 may generate supplementary information D28 for the input / output data of the learning model unit 200 based on the acquired state information D25 and / or feedback information D26, and issue it to the user 1, the learning model unit 200, or another device (not shown). Also, the control system 2000 may be configured to return an inquiry D27 to the user 1 when the input information D21 includes unclear or uncertain information. The handling of the inquiry D27 is the same as that of the inquiry D17 in the first embodiment.

[0184] FIG. 12 is a configuration diagram showing another example of the control system 2000. As shown in FIG. 12, the control system 2000 may further include a state acquisition unit 230 that acquires the state information D25 and / or the feedback information D26 and issues the supplementary information D28. The state acquisition unit 230 is the same as the state acquisition unit 130 in the first embodiment.

[0185] Also in this embodiment, the target device 2 is not particularly limited. Note that although it is assumed that the target device 2 is a device that can actually be controlled upon receiving the control command D22, this is not the case when the target device 2 includes a conversion device that converts various signals such as a controller or a converter between the target device 2 and the control system. In that case, the conversion device may receive the control command D22 and control the target device 2.

[0186] In the present embodiment, the input information D21 received by the control system 2000 can be information regarding a request in the working environment, here the environment in which the target device 2 operates (here, the control content required for the target device 2). Therefore, the input information D21 received by the control system 2000 can be said to be an example of the first information indicating a request in the working environment. Further, the control command D22 can be said to be information used for such work (work related to the control of the target device 2) corresponding to such input information D21. Hereinafter, the control command D22 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D21 is input is called the second information.

[0187] Next, the operation of the control system 2000 of the present embodiment will be described. FIG. 13 is a flowchart showing an operation example of the control system 2000.

[0188] In the example shown in FIG. 13, first, the control system 2000 receives the input information D21 (step S210). For example, the above-described input unit 102 or input processing unit 201 may receive the input information D21. The received input information D21 is input to the learning model unit 200 as the model input data D101.

[0189] Next, the control system 2000 performs a generation process of the control command D22 using the learning model unit 200 (step S211). In step S211, the learning model unit 200 (more specifically, the model control unit 101) outputs a control command D22 corresponding to the input information D21 based on the model information D102, the input input information D21, and the model reference information D104 including the device information D23 as necessary.

[0190] In step S211, the learning model unit 200 may generate a control command D22 for binary data from the input input information D21 using, for example, a learning model capable of generating binary data. Further, the learning model unit 200 may generate a control command D22 for text data from the input input information D21 using, for example, a learning model capable of generating text data. Further, the learning model unit 200 may generate a control command D22 for image data from the input input information D21 using, for example, a learning model capable of generating image data. Further, the learning model unit 200 may generate a control command D22 for audio data from the input input information D21 using, for example, a learning model capable of generating audio data.

[0191] In step S211, the preprocessing unit 105 and / or the postprocessing unit 106 of the learning model unit 200 may further perform the above-described processing.

[0192] The control command D22 output from the learning model unit 200 is input to the target device 2, for example (step S212). The input of the control command D22 to the target device 2 may be directly input from the control system 2000 (more specifically, the learning model unit 200 or the information processing device 10 which is its operating environment), or may be indirectly input via a communication network or other devices (servers, various conversion devices, etc.).

[0193] Thereby, the target device 2 operates according to the input control command D22.

[0194] When there is a change in the state of the target device 2 such that the target device 2 is controlled as a result of the control system 2000 outputting the control command D22, or when there is feedback from the target device 2, the control system 2000 may acquire the state information D25 and the feedback information D26 (step S213). Note that the processing in step S213 is not essential and may be omitted as appropriate.

[0195] The control system 2000 may repeat the processes of step S210 to step S213 a plurality of times (for example, until the desired control for the target device 2 is completed).

[0196] As described above, according to this embodiment, even if the user 1 does not know the specific control method for the target device 2, the control command D22 can be generated from the input information D21 input from the user 1, and the target device 2 can be controlled based on the generated control command D22. Therefore, the efficiency and sophistication of the work related to the control of the target device 2 can be improved.

[0197] Also, according to this embodiment, the device can be controlled to an appropriate state even from ambiguous information.

[0198] Embodiment 3. Next, Embodiment 3 will be described. In this embodiment, an example of assisting the work related to the operation of the target device by using a learning model will be described.

[0199] Hereinafter, for example, in a home or a building, consider operating various devices such as an air conditioner, a refrigerator, a TV, lighting, a washing machine, a projector, various sensors, and communication devices. In recent years, even these consumer-oriented devices have advanced functions and complex controls. Although the controllers such as the operation screen and the remote controller have been improved to enable easy operation of complex controls, it is still difficult to remember all the operations, and sometimes it is difficult to easily reach the desired function even though the desired function is provided.

[0200] Also, even for the same type of function, the function names provided by different models may be different, there may be differences in detailed functions, and the control methods are often different. In the scenario of introducing different models due to replacement, etc., it is cumbersome to have to remember these differences from scratch.

[0201] In addition, some devices can automatically control themselves to an appropriate state by remembering past operation histories and grasping the operating environment. However, in situations where multiple people gather or even for a single person, the appropriate state may vary depending on changes in physical condition, etc., it has been difficult to perform precise control in scenes where the appropriate state differs from person to person.

[0202] In such cases, even if one does not know the specific operation method and even if the operator does not grasp the state considered appropriate, it is preferable because it leads to improved work efficiency and performance if the operation to achieve the desired state can be easily performed.

[0203] Note that the scenes where the device is operated are not limited to within a home or a building, and the application scenes of this embodiment are also not limited to within a home or a building.

[0204] FIG. 14 is a configuration diagram showing an example of a control system 3000 according to Embodiment 3. The control system 3000 shown in FIG. 14 is a control system for operating a device using a learning model, and includes a learning model unit 300, a device information storage unit 310 (denoted as a device information DB in the figure), an input interface 311 (denoted as an input IF in the figure), and an output interface 312 (denoted as an output IF in the figure).

[0205] When the input information D31 is input, the learning model unit 300 outputs an operation command D32. For example, when the input information D31 is input, the learning model unit 300 outputs the operation command D32 based on the model information D102. The configuration of the learning model unit 300 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0206] In the present embodiment, the learning model unit 300 is a model and its operating environment configured to output an operation command D32 corresponding to the input information D31 when the input information D31 is input. Further, the learning model unit 300 may be a model and its operating environment configured to generate and output an operation command D32 based on the input information D31, the device information D33, and other information that can be referred to in the learning model unit 300 when the input information D31 is input.

[0207] In the present embodiment, the input information D31 includes information indicating the operation content required for the target device 2. The input information D31 may be, for example, text, an image, voice, or a combination thereof indicating the operation content for the target device 2. The input information D31 may be, for example, text, an image, voice, or a combination thereof indicating a plurality of operation contents for the target device 2. Further, the input information D31 may include information indicating operation contents performed continuously in time, and in that case, it may be time-series data having a predetermined data structure including text, an image, voice, or a combination thereof indicating the operation contents as described above. It is assumed that the way of indicating the operation content matches the input format of the model used by the learning model unit 300, but this is not the case when error processing, correction processing, or conversion processing is included in the previous stage of the learning model unit 300.

[0208] As an example of the way of indicating the operation content in the input information D31, after specifying the operation to be performed on the target device 2, the value of the parameter for performing the operation or the state after the operation may be specified. In that case, the input information D31 may include, for example, information for specifying the operation and information indicating the value of the parameter for performing the operation or the state after the operation. The value of the parameter for performing the operation may include, for example, values related to the type of operation (such as ON / OFF), direction, amount, and time. Further, the input information D31 may include not only information directly indicating the operation content for the target device 2 but also information indirectly indicating the operation content using control content corresponding to the operation content, the behavior of the user 1, an image of the target device 2, or an operation command similar to that in other models.

[0209] The operation instruction D32 includes information regarding the operation of the target device 2 indicated in a predetermined format that can be discriminated by the target device 2 or an interface (including a person) that requests control of the target device 2. The operation instruction D32 may include information indicating an operation request or a control request to the target device 2. The operation instruction D32 is, for example, an operation command, an operation signal, an operation code, a control command, a control signal, or a control code for the target device 2. Further, the operation instruction D32 may be, for example, a command described in a format handled by a predetermined controller corresponding to the target device 2. The operation instruction D32 can be said to be a concept in which information regarding an operation is added to the above-described control instruction D22. Further, for example, when the interface is a person, that is, when control is requested for the target device 2 via a person, the operation instruction D32 may be information indicating a method of operating the target device 2 indicated in a format that can be discriminated by the person.

[0210] The device information storage unit 310 stores device information D33, which is information regarding the target device 2. The handling of the device information storage unit 310 and the device information D33 is basically the same as that of the device information storage unit 110 and the device information D13 in the first embodiment. Note that the device information D33 in the present embodiment may include, for example, information used for the operation of the target device 2. The device information D33 may include, for example, information indicating a procedure of an operation actually performed on the target device 2 with respect to the operation content. Further, the device information D33 may include, for example, instructions, signals, codes, etc. issued to the target device 2. The device information D33 is used, for example, as additional information when the learning model unit 300 outputs the operation instruction D32. Hereinafter, in the present embodiment, information indicating the state of the target device 2 may be particularly referred to as state information D35.

[0211] The input interface 311 is an interface that receives input information D31 from user 1 and inputs it to the learning model unit 300. The input interface 311 may be, for example, an interface that converts the input information D31 input from user 1 into data that matches the input of the learning model unit 300 and outputs it. Note that the input interface 311 may be provided, for example, as an example of the above-described input unit 102.

[0212] The output interface 312 is an interface that receives an operation command D32 from the learning model unit 300 and outputs it to a predetermined output destination. Note that the output interface 312 may be provided, for example, as an example of the above-described output unit 103. The output interface 312 may be, for example, an interface that converts the operation command D32 output from the learning model unit 300 into data that matches a predetermined output destination and outputs it. In the present embodiment, the output destinations of the output interface 312 may include the target device 2, the controller 4 (not shown), a predetermined display 7 (not shown), and the operation terminal of user 1 (not shown).

[0213] In the present embodiment, the learning model unit 300 may be, for example, a language learning model such as an LLM that inputs natural language and obtains an output result, and its operating environment. Further, the learning model unit 300 may be, for example, an image learning model such as a VLM that inputs an image and obtains an output result, and its operating environment. Further, the learning model unit 300 may be, for example, a multimodal model that inputs natural language and an image and obtains an output result, and its operating environment. In that case, the input information D31 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data, a video that is a combination of audio data and image data). Note that the learning model used in the present embodiment is not limited to the above-described models.

[0214] In this embodiment, the input information D31 corresponds to the model input data D101. Also, the operation command D32 corresponds to the model output data D103. The learning model unit 300 (particularly, the model control unit 101) may be configured to output an operation command D32 corresponding to the input information D31 based on, for example, the model information D102 and, if necessary, the model reference information D104 when receiving the input information D31.

[0215] Also, in such a case, the model generation unit 107 provided corresponding to the learning model unit 300 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D31 that can be input to the model control unit 101 to generate or update the model information D102. Further, the model generation unit 107 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D31 that can be input to the model control unit 101 and candidates for the operation command D32 corresponding thereto to generate or update the model information D102.

[0216] Although not shown in the drawings, in this embodiment as well, the state information D35 and / or the feedback information D36 may be acquired from the output destination of the model output data D103 of the learning model unit 300 and / or the information generated based thereon. The control system 3000 may output, for example, the acquired state information D35 and / or feedback information D36 as information indicating the response result to the user 1, the learning model unit 300, or another device (not shown). Also, the control system 3000 may be configured to return an inquiry D37 to the user 1 when the input information D31 includes unclear or uncertain information. Further, the control system 3000 may generate supplementary information D38 for the input / output data of the learning model unit 300 based on the acquired state information D35 and / or feedback information D36 and issue it to the user 1, the learning model unit 300, or another device (not shown). The handling of the state information D35, the feedback information D36, the inquiry D37, and the supplementary information D38 may be basically the same as in the first embodiment.

[0217] Further, the control system 3000 may further include a state acquisition unit 330 (not shown) that acquires state information D35 and / or feedback information D36 and issues supplementary information D38 as necessary. The state acquisition unit 330 is the same as the state acquisition unit 130 in the first embodiment.

[0218] Also in this embodiment, the target device 2 is not particularly limited. Although it is assumed that the target device 2 is a device capable of receiving the operation command D32 and performing control corresponding to the operation content indicated by the operation command D32, this is not the case when a conversion device that converts various signals such as a controller 4 or a converter or an operator is included between the target device 2. In that case, the conversion device or the operator may receive the operation command D32 and operate the target device 2.

[0219] In this embodiment, the input information D31 received by the control system 3000 can be information regarding the working environment, here the requirements in the environment where the target device 2 operates (here, the operation content required for the target device). Therefore, the input information D31 received by the control system 3000 can be said to be an example of the first information indicating the requirements in the working environment. Also, the operation command D32 can be said to be information used for such work (work related to the operation of the target device 2) corresponding to such input information D31. Hereinafter, the operation command D32 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D31 is input may be referred to as the second information.

[0220] Next, the operation of the control system 3000 of this embodiment will be described. FIG. 15 is a flowchart showing an operation example of the control system 3000.

[0221] In the example shown in FIG. 15, first, the input interface 311 of the control system 3000 receives the input information D31 (step S310). The received input information D31 is input to the learning model unit 300 as model input data D101.

[0222] Next, the control system 3000 performs a generation process of the operation command D32 using the learning model unit 300 (step S311). In step S311, the learning model unit 300 (more specifically, the model control unit 101) generates and outputs an operation command D32 corresponding to the input information D31 based on the model information D102, the input input information D31, and the model reference information D104 including the device information D33 as necessary.

[0223] In step S311, the learning model unit 300 may generate an operation command D32 of binary data from the input input information D31 using, for example, a learning model capable of generating binary data. Also, the learning model unit 300 may generate an operation command D32 of text data from the input input information D31 using, for example, a learning model capable of generating text data. Further, the learning model unit 300 may generate an operation command D32 of image data from the input input information D31 using, for example, a learning model capable of generating image data. Additionally, the learning model unit 300 may generate an operation command D32 of audio data from the input input information D31 using, for example, a learning model capable of generating audio data.

[0224] In step S311, the preprocessing unit 105 and / or the postprocessing unit 106 of the learning model unit 300 may perform the above-described processing.

[0225] The operation command D32 output from the learning model unit 300 is output to a predetermined output destination via, for example, the output interface 312. Here, the predetermined output destination may be the target device 2, the controller 4, a predetermined display 7, or the operation terminal of the user 1 (not shown). When the operation command D32 is input to the predetermined output destination, the target device 2 is operated according to the input operation command D32 (step S312).

[0226] For example, the output interface 312 may output the operation command D32 to the target device 2. In this case, the target device 2 that has received the operation command D32 (for example, an operation instruction, an operation signal, an operation code, a control instruction, a control signal, or a control code, etc.) may execute actual control according to the operation command D32. Also, the output interface 312 may output the operation command D32 to the controller 4 corresponding to the target device 2. In this case, the controller 4 that has received the operation command D32 (for example, a command for the controller 4, an operation instruction, an operation signal, an operation code, etc., which is indirect control information for the target device 2) may operate the target device 2 according to the operation command D32. The controller 4 may operate the target device 2 by outputting direct control information such as a control code to the target device 2 based on the control information indicated by the received operation command D32. Here, the controller 4 may be, for example, an operation panel provided in the target device 2, or a remote controller corresponding to the target device 2 that the user 1 directly operates. The controller 4 includes those specific to the target device 2 and general-purpose ones. Also, the output interface 312 may output the operation command D32 to the operation terminal of the user 1 or a predetermined display. In this case, the operation terminal or display of the user 1 that has received the operation command D32 (for example, information indicating an operation method, etc.) displays the operation command D32. Then, the user 1 may operate the target device 2 or the controller 4 by referring to the displayed operation command D32.

[0227] The input of the operation command D32 to the output destination may be directly input from the control system 3000 (more specifically, the learning model unit 300 or the information processing device 10 that is its operating environment), or may be indirectly input via a communication network or other devices (servers, various conversion devices, etc.).

[0228] As a result, the target device 2 operates according to the operation command D32.

[0229] When the control system 3000 outputs an operation command D32 and as a result, the target device 2 is operated or the like and there is a change in the state of the target device 2, if there is feedback from the target device 2, the state information D35 and the feedback information D36 may be acquired (step S313). Note that the process of step S313 is not essential and may be omitted as appropriate.

[0230] The control system 3000 may repeat the processes of step S310 to step S313 a plurality of times (for example, until a desired operation on the target device 2 is completed).

[0231] As described above, according to the present embodiment, even if the user 1 does not know the specific operation method for the target device 2, the operation command D32 can be generated from the input information D31 input from the user 1, and the target device 2 can be operated based on the generated operation command D32. Therefore, the efficiency and sophistication of the work related to the operation of the target device 2 can be improved.

[0232] Also, according to the present embodiment, the device can be operated in an appropriate state even from ambiguous information. Further, according to the present embodiment, the device can be operated in an appropriate state without depending on the device and without learning the operation method of the device.

[0233] Modification Example 3-1. Next, a modification example of the control system 3000 will be described. FIG. 16 is a configuration diagram showing an example of a control system 3000a which is a modification example of the control system 3000 according to the present embodiment. Note that the same elements as those of the control system 3000 are denoted by the same reference numerals and the description thereof is omitted.

[0234] The control system 3000a shown in FIG. 16 further includes an input determination unit 31. The input determination unit 31 is a means for analyzing the input information D31 when receiving the input information D31 and switching the control destination for the input information D31. In this modification example, the input determination unit 31 switches the control destination for the input information D31 between the learning model unit 300 and the output interface 312.

[0235] For example, the input determination unit 31 may switch the control destination for the input information D31 depending on whether the input information D31 conforms to the operation instruction rule for the target device 2. When the input information D31 conforms to the operation instruction rule for the target device 2, the input determination unit 31 may input the input information D31 as it is to the output interface 312. On the other hand, when the input information D31 does not conform to the operation instruction rule for the target device 2, the input determination unit 31 may input the input information D31 to the learning model unit 300.

[0236] Whether it conforms to the operation instruction rule may be determined, for example, using a model described in a rule-based manner. Here, the input determination unit 31 may be a learning model that is relatively lightweight compared to the learning model unit 300.

[0237] Hereinafter, in order to distinguish the information input in the output interface 312, the operation command D32 output from the learning model unit 300 may be referred to as an operation command D32a, and the input information D31 output to the output interface 312 may be referred to as an operation command D32b.

[0238] In this example, the output interface 312 may be an interface that receives the operation command D32a or the operation command D32b and outputs it to a predetermined output destination.

[0239] Next, the operation of the control system 3000a of this modification will be described. FIG. 17 is a flowchart showing an operation example of the control system 3000a.

[0240] In the example shown in FIG. 17, first, the input interface 311 of the control system 3000a receives the input information D31 (step S310). The received input information D31 is input to the input determination unit 31.

[0241] Next, the input determination unit 31 determines whether the input information D31 conforms to the operation instruction rule for the target device 2 (step S321). Here, when it is determined that the input information D31 conforms to the operation instruction rule for the target device 2 (Yes in step S321), the input information D31 is input to the output interface 312 (proceed to step S322). On the other hand, when it is determined that the input information D31 does not conform to the operation instruction rule for the target device 2 (No in step S321), the input information D31 is input to the learning model unit 300 (proceed to step S311).

[0242] The processing of steps S311 to S313 is the same as the example shown in FIG. 15.

[0243] In step S322, the output interface 312 outputs the input input information D31 as an operation command D32b to a predetermined output destination. As a result, the target device 2 operates according to the operation command D32b.

[0244] Regarding other points, it may be the same as other control systems according to this embodiment.

[0245] As described above, according to this modification example, when the input from the user 1 conforms to the operation instruction rule for the target device 2, the target device 2 can be operated according to the input. On the other hand, when it does not conform, the target device 2 can be operated using the learning model. Therefore, the efficiency of the work related to the operation of the target device 2 can be further improved.

[0246] Modification Example 3-2. Next, another modification example of the control system 3000 will be described. In this modification example, an operation command including mediation of a plurality of inputs is generated using a learning model.

[0247] FIG. 18 is a configuration diagram showing an example of a control system 3000b which is a modification example of the control system 3000 according to this embodiment. Note that the same elements as those in the control system 3000 are denoted by the same reference numerals and the description thereof is omitted.

[0248] In the control system 3000b shown in FIG. 18, the input interface 311 receives input information D31 from a plurality of users 1.

[0249] The input interface 311 receives the input information D31 from a plurality of users 1 and inputs it to the learning model unit 300. At this time, the input interface 311 may receive the input information D31 with the information of the input source user 1 attached, or the input interface 311 may discriminate the input source user 1 and attach the input source information before receiving the input information D31, or may receive it without any particular processing.

[0250] The learning model unit 300 may be a model and its operating environment configured to output an operation command D32 corresponding to the input information D31 group when the input information D31 group received by the input interface 311 is input. The learning model unit 300 may be a model and its operating environment configured to generate and output an operation command D32 based on the input information D31 group, device information D33, and other information that can be referred to in the learning model unit 300 when the input information D31 group is input.

[0251] For example, the learning model unit 300 may perform a process of extracting a preferred solution in the language space (more specifically, on the feature vector space having the information of the language space) using a language learning model such as an LLM that inputs a natural language and obtains an output result, thereby generating and outputting an operation command D32 that is a compromise for different operation contents indicated by the input information D31 group. At this time, the learning model unit 300 may refer to the history of the input information D31 for each input source user 1 and / or the history of the operation command D32 for each input source user 1.

[0252] Regarding other points, it may be the same as other control systems according to this embodiment.

[0253] As described above, according to this modification example, even when information regarding different operation contents is input from a plurality of users 1, it is possible to generate a more appropriate operation command D32 in which those contents are mediated using the learning model unit 300. Therefore, it is possible to further enhance the functionality of the work related to the operation of the target device 2.

[0254] Modification Example 3-3. Next, another modification example of the control system 3000 will be described. In this modification example, an operation screen user interface is generated using a learning model.

[0255] FIG. 19 is a configuration diagram showing an example of a control system 3000c which is a modification example of the control system 3000 according to the present embodiment. Note that the same elements as those of the control system 3000 are denoted by the same reference numerals and the description thereof is omitted.

[0256] The control system 3000c shown in FIG. 19 further includes an operation screen user interface 3 (denoted as operation screen UI in the figure). Further, the learning model unit 300 generates, as the operation command D32, an operation screen for actually performing an operation of the operation content corresponding to the input information D31 on the target device 2.

[0257] The operation screen generated by the learning model unit 300 may be, for example, a screen API (Application Programming interface) having a function of receiving an operation input from the user 1 together with an explanation of the operation content and outputting a control command D34 such as a control code according to the received operation input. Here, the output of a control code or the like according to the operation input includes a mode in which a plurality of control commands D34 are sequentially output according to one operation input. Further, the operation screen may be a screen API having operation explanations, operation input reception, and control command output corresponding to two or more different operation contents. The learning model unit 300 may extract, for example, operation information indicating two or more different operation contents as the operation command D32 corresponding to the input information D31, and generate a screen API having operation input reception and control command output corresponding to each operation information.

[0258] In addition, the operation screen generated by the learning model unit 300 may be such that the display mode of the existing operation screen is changed so that the operation location corresponding to the corresponding operation content is highlighted and displayed, the operation function is restricted and displayed, or the position and form (shape, size, color, etc.) of the UI components on the screen are changed and displayed.

[0259] The operation screen user interface 3 is an interface that displays an operation screen for the target device 2 and receives an input related to a user operation on the operation screen. The operation screen user interface 3 may be realized, for example, by a touch panel display or a controller having an operation button and a display unit. Further, the operation screen user interface 3 may be realized by a display device such as a display that cooperates with an operation input device such as a mouse.

[0260] In addition, the output interface 312 in this modification outputs the operation command D32 (operation screen) output from the learning model unit 300 to the operation screen user interface 3.

[0261] In addition, in this modification, the learning model unit 300 may have a function of interactively confirming the operation expected by the user 1. In such a case, for example, after presenting an operation screen as the operation command D32, when the learning model unit 300 receives information requesting re-acquisition of the operation command D32, it may re-acquire the operation command D32 after making changes to a part of the input information, a part of the model parameters, or the reference destination of the reference information.

[0262] As described above, in this modification, an operation screen subjected to processing (such as constructing a screen API or changing the display mode) so that a desired operation can be performed easily or understandably can be generated using the learning model unit 300, so that the efficiency of the work related to the operation of the target device 2 can be further improved. Further, according to this modification, since the user 1 can perform the actual operation while checking the explanation of the operation command generated by the learning model unit 300, the operation can be carried out without mistakes.

[0263] Modification Example 3-4 Next, another modification example of the control system 3000 will be described. In this modification example, the learning model further uses environmental information to generate operation commands.

[0264] FIG. 20 is a configuration diagram showing an example of a control system 3000d which is a modification example of the control system 3000 according to the present embodiment. Note that the same elements as those in the control system 3000 are denoted by the same reference numerals and the description thereof is omitted.

[0265] The control system 3000d shown in FIG. 20 further includes an environmental information storage unit 313 (denoted as an environmental information DB in the figure).

[0266] The environmental information storage unit 313 stores environmental information D33a which is information on the environment at the operation destination of the target device 2. The environmental information D33a may include information on the space where the target device 2 is operating. In this modification example, information on objects or people existing in the space where the target device 2 is operating and the user 1 who is the operator of the target device 2 are also regarded as part of the environment. Therefore, the environmental information D33a may include information on such objects or people or the user 1.

[0267] The environmental information D33a may include, for example, information such as a person's attributes, temperature, position, posture, and heartbeat as information regarding that person. Also, the environmental information D33a may include, for example, information such as the location, temperature, humidity, and brightness of the space as information regarding that space. Further, the environmental information D33a may hold information indicating the transition when information regarding such a space or person changes. Here, the information indicating the transition is also called time-series data or history information. The environmental information D33a may be configured as part of, for example, the model reference information D104 of the learning model.

[0268] The environmental information D33a may be acquired, for example, by a sensor (not shown) or the like.

[0269] When the learning model unit 300 receives the input information D31, for example, it is configured to generate and output an operation command D32 based on the input information D31, device information D33, environment information D33a, and other information that can be referred to in the learning model unit 300, along with its operating environment.

[0270] As described above, according to this modification example, since the learning model can generate the operation command D32 using the environment information D33a regarding the space where the target device 2 is operating, it is possible to further enhance the functionality of the work related to the operation of the target device 2.

[0271] Modification Example 3-5. Next, another modification example of the control system 3000 will be described. In this modification example, two learning models are combined to generate an operation command. FIG. 21 is a configuration diagram showing an example of a control system 3000e, which is a modification of the control system 3000 according to the present embodiment. The same elements as those in the control system 3000 are denoted by the same reference numerals and the description thereof is omitted.

[0272] In the control system 3000e shown in FIG. 21, instead of the learning model unit 300 shown in FIG. 20, a learning model unit 300a as the first learning model unit and a learning model unit 300b as the second learning model unit are provided.

[0273] When the input information D31 is input, the learning model unit 300a outputs operation information D320. The learning model unit 300a may be a model and its operating environment configured to generate and output the operation information D320 based at least on the input information D31 and the environment information D33a when the input information D31 is input.

[0274] Here, the operation information D320 includes information regarding the operation of the target device 2 indicated in a predetermined format that can be recognized by the subsequent learning model unit 330b. Here, the operation information D320 may be information obtained by supplementing (including adding, modifying, or canceling) the operation content indicated by the input information D31 according to the environmental information D33a. The operation information D320 may be information obtained by changing the operation content or its expression indicated by the input information D31 according to the situation of the space where the target device 2 is driven. The learning model unit 300a may mainly be a model that performs grounding on the input information D31.

[0275] For example, even if the desired operation is the same, it is conceivable that there are differences in language expressions and differences in the way of recognizing events depending on the environment in which the target device 2 is driven. For example, due to dialects, speaking habits, use of in-company or in-house terms, differences in perception such as hot / cold, etc., the operation content meant by the input information D31 may be different.

[0276] The learning model unit 300a, for example, serves to absorb such differences in language expressions and / or differences in the recognition of events and correct them to more generalized or specific content. The learning model unit 300a may be a local learning model that obtains an output result based on local information such as when the reference database is restricted.

[0277] The operation information D320 generated by the learning model unit 300a is input to the learning model unit 300b.

[0278] The learning model unit 300b may basically be the same as the above-described learning model unit 300. However, instead of the input information D31, the operation information D320 generated by the learning model unit 300a is input.

[0279] When the operation information D320 is input, the learning model unit 300b outputs an operation command D32. The learning model unit 300b may be a model and its operating environment configured to generate and output the operation command D32 based on the operation information D320, the device information D33, and other information that can be referred to in the learning model unit 300b when the operation information D320 is input. The learning model unit 300b may be a global learning model that obtains an output result based on global information such as being freely accessible to an external network.

[0280] FIG. 22 is a flowchart showing an operation example of this modification. In the example shown in FIG. 22, when the input interface 311 of the control system 3000e receives the input information D31 in step S310, the input information D31 is input to the learning model unit 300a.

[0281] Next, the control system 3000e performs a generation process of the operation information D320 using the learning model unit 300a (step S331). In step S331, the learning model unit 300a (more specifically, the model control unit 101) generates and outputs the operation information D320 corresponding to the input information D31 based on the model information D102, the input input information D31, and the model reference information D104 including the environment information D33a as needed. The operation information D320 output from the learning model unit 300a is input to the learning model unit 300b.

[0282] Next, the control system 3000e performs a generation process of the operation command D32 using the learning model unit 300b (step S332). In step S332, the learning model unit 300b (more specifically, the model control unit 101) generates and outputs the operation command D32 corresponding to the operation information D320 based on the model information D102, the input operation information D320, and the model reference information D104 including the device information D33 as needed.

[0283] The subsequent processing may be the same as that of other control systems according to this embodiment.

[0284] As described above, according to this modification example, for the input information D31 input from the user 1, after absorbing the differences in language expressions and / or the differences in the recognition of events and changing it to more generalized or specific content, the operation command D32 can be generated. Therefore, it is possible to further enhance the functionality of the work related to the operation of the target device 2.

[0285] Note that even in the configuration shown in Modification Example 3-4, it is possible for the learning model unit 300 to generate an operation command D32 in which the differences in language expressions and / or the differences in the recognition of events are equalized based on the environmental information D33a, the device information D33, the model reference information D104 including the past operation history, and the like. However, according to this modification example, the role of the learning model can be clearly divided, such as absorbing the differences in expressions and converting them into operation commands. Therefore, the learning model can be specialized and learned accordingly, and a compact design can be achieved, such as suppressing the scale of learning.

[0286] Embodiment 4. Next, Embodiment 4 will be described. In this embodiment, an example of assisting the work related to monitoring a certain work situation using a learning model will be described.

[0287] For example, consider monitoring the abnormalities of a FA (Factory Automation) system including control devices such as robots and PLCs in a factory. For example, when there is an obvious installation error in the target work of the control device, existing monitoring algorithms such as rule-based ones can handle it. However, it is also conceivable that a slight installation error triggers an abnormality to be discovered in a subsequent process. In such a case, even if analysis or the like is performed triggered by, for example, abnormality detection, it has been difficult to accurately grasp the situation and obtain a method for improvement.

[0288] In this embodiment, by assisting the work related to monitoring a work environment where cases where the occurrence situation does not conform to existing rules and it is difficult to investigate the cause, such as a relatively small defect spreading to a large abnormality, can be assumed, the efficiency and high performance of the monitoring work are improved.

[0289] FIG. 23 is a configuration diagram showing an example of the control system 4000 according to Embodiment 4. The control system 4000 shown in FIG. 23 is a control system for monitoring a specific working situation using a learning model, and includes a sensor 5, a learning model unit 400a, a learning model unit 400b, a device information storage unit 410 (denoted as a device information DB in the figure), a model interface 6 (denoted as a model IF in the figure), and a display 7.

[0290] The sensor 5 acquires data indicating the situation of the work to be monitored. Hereinafter, the data acquired by the sensor 5 is referred to as sensor data. The sensor data may be, for example, image data obtained by photographing the state of the work to be monitored. Further, the sensor data may be, for example, audio data obtained by recording the state of the work to be monitored. Further, the sensor data may be, for example, measurement data obtained by measuring the state such as the position of a person or an object performing the work to be monitored.

[0291] Although it is assumed that the acquisition of sensor data by the sensor 5 is always performed, it may be performed based on a trigger given by a person or another monitoring system, for example. The sensor data acquired by the sensor 5 is input to the learning model unit 400a as input information D41. Further, the sensor data itself used as the input information D41 may be given from a person or another monitoring system. In such a case, the sensor 5 can be omitted.

[0292] When the input information D41 is input, the learning model unit 400a outputs an analysis result D42a. For example, when the input information D41 is input, the learning model unit 400a outputs an analysis result D42a based on the model information D102. The configuration of the learning model unit 400a may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0293] In the present embodiment, the learning model unit 400a is a model and its operating environment configured to output an analysis result D42a corresponding to the input information D41 when the input information D41 is input. Further, the learning model unit 400a may be a model and its operating environment configured to generate and output the analysis result D42a based on, for example, the input information D41, the device information D43, and other information (such as the model reference information D104) that can be referred to in the learning model unit 400a when the input information D41 is input. Here, the learning model unit 400a may refer to and utilize information regarding the work to be monitored as the model reference information D104. The information regarding the work to be monitored may be, for example, information indicating the position, person, object, procedure, conditions, etc. where the work is performed. The learning model unit 400a may use, for example, as the model reference information D104, data obtained by digitizing a manual in which conditions of devices used in the work, installation environment, operation procedures, etc. are described.

[0294] In the present embodiment, the input information D41 includes information indicating the status of the work to be monitored. Here, the work to be monitored includes one or more operations by a person or a device. The input information D41 may be, for example, a measured value, an image, a voice, or a combination thereof indicating the status of the work to be monitored. The input information D41 may be, for example, a measured value, an image, a voice, or a combination thereof indicating the status of a plurality of works to be monitored. Further, the input information D41 may include information indicating the status of operations performed continuously in time, and in that case, it may be time-series data having a predetermined data structure including a measured value, an image, a voice, or a combination thereof indicating the status as described above. The way of indicating the work status is premised on being consistent with the input format of the model used by the learning model unit 400a, but this is not the case when error processing, correction processing, or conversion processing is included in the stage preceding the learning model unit 400a.

[0295] The analysis result D42a includes information indicating a situation analysis result obtained by analyzing the working situation indicated by the input information D41. The information indicating the situation analysis result may be information indicating an object (environment) existing and / or an event occurring in the working situation indicated by the input information D41. The information indicating the situation analysis result can be said to be information indicating an interpretation of the working situation indicated by the input information D41. The analysis result D42a may be, for example, text indicating an interpretation of the working situation indicated by the input information D41. Also, the analysis result D42a may be, for example, text that focuses on a part different from the normal situation in the working situation indicated by the input information D41 and indicates an interpretation of that part. Note that the format of the analysis result D42a may be other than text. The analysis result D42a is not particularly limited as long as it is described in a predetermined format that can be discriminated by the subsequent learning model unit 400b, and may be, for example, text, an image, audio, or a combination thereof.

[0296] Examples of the interpretation of the working situation include expressing the object existing in the working situation using its attributes, expressing the event occurring in the working situation in a predetermined syntactic form such as 5W1H or 7W1H, or further summarizing after such a concrete expression. In addition, there are also examples such as decomposing the work being done in the working situation into multiple viewpoints for interpretation and expressing it for each viewpoint, and when the work being done in the working situation includes multiple sub-tasks or steps, decomposing the target work into sub-task units or step units and explaining it for each sub-task or step. The analysis result D42a can be said to be the result of adding concretization, subdivision, and / or extraction of singular points to the working situation indicated by the input information D11, and further expressing it in a predetermined format. In this way, in the analysis result D42a, the working situation is expressed in an easy-to-understand and organized state.

[0297] When the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b. For example, when the analysis result D42a is input, the learning model unit 400b outputs the analysis result D42b based on the model information D102. The configuration of the learning model unit 400b may be basically the same as that of the learning model unit 100 in the first embodiment.

[0298] In the present embodiment, the learning model unit 400b is a model and its operating environment configured to output the analysis result D42b corresponding to the analysis result D42a when the analysis result D42a is input. Further, for example, when the analysis result D42a is input, the learning model unit 400b may be a model and its operating environment configured to generate and output the analysis result D42b based on the analysis result D42a, the device information D43, and / or information that can be referred to in the learning model unit 400b (such as the model reference information D104).

[0299] The analysis result D42b includes information indicating a method for improving the working condition derived from the analysis result of the working condition by the learning model unit 400a. The information indicating the method for improving the working condition may be information indicating a recovery method for normally restoring an abnormal state, or may be information indicating a solution method for a problem that occurs in an environment (working environment) where the work to be monitored, such as a person in trouble or a device stopped, is being performed.

[0300] The information indicating the improvement method may be, for example, text, an image, or audio indicating the method, a control command (e.g., an instruction, a control signal, a control code, etc.) for the device (target device 2) to which the method is to be applied, a procedure manual describing the method, a sequence diagram, source code, execution code, or a controller command for causing the controller to execute the method. The information indicating the improvement method may be text, an image, audio, data described in a predetermined design language, control descriptions (including source code and information described in a predetermined programming platform language), information described in other platform languages, control commands (including control instructions, control signals, control codes, and controller commands), execution code, or a combination of two or more of these elements. Examples of the predetermined design language include, but are not limited to, UML (Unified Modeling Language).

[0301] Hereinafter, the learning model unit 400a may be referred to as the first learning model unit 400, and the analysis result D42a may be referred to as the first analysis result D42. Also, the learning model unit 400b may be referred to as the second learning model unit 400, and the analysis result D42b may be referred to as the second analysis result D42.

[0302] As already described, the analysis result D42a includes information indicating the situation analysis result of the working situation indicated by the input information D41. Therefore, the learning model unit 400b may be a model configured to output an analysis result D42b corresponding to the situation analysis result indicated by the analysis result D42a, and its operating environment. Here, when the information indicating the situation analysis result is text explaining the working situation indicated by the input information D41, the learning model unit 400b may be a model configured to output an analysis result D42b corresponding to the text explaining the working situation, and its operating environment.

[0303] The handling of the device information storage unit 410 and the device information D43 is basically the same as that of the device information storage unit 110 and the device information D13 in the first embodiment. In this embodiment, the device information storage unit 410 stores, as the target device 2, the device information D43 which is information on devices related to the work to be monitored. Here, the devices related to the work include the devices required for the situation analysis and the derivation of improvement methods widely described above. More specifically, it includes not only the devices used for the work, but also the devices that affect the person or device performing the work. The devices that affect the person or device performing the work may more specifically be devices that directly or indirectly bring about changes to the person or device performing the work. As an example, the devices directly used for the work (including various machines such as processing machines and conveyors, and tools such as workbenches and tools), the devices that control the devices directly used for the work (such as power supplies, relays, switches, and controllers), and the devices that bring about changes to the working environment (such as lighting devices, air conditioning devices, cleaners, and purifiers) can be mentioned.

[0304] The device information D43 is used, for example, as additional information when the learning model unit 400a and / or the learning model unit 400b outputs the model output data D103 (analysis result D42a, analysis result D42b). Hereinafter, in this embodiment, the information indicating the state of the target device 2 may be particularly referred to as the state information D45.

[0305] In this embodiment, the learning model unit 400a may be an image learning model such as a VLM that inputs an image and obtains an output result, and its operating environment. Further, the learning model unit 400a may be, for example, a multimodal model that inputs natural language and an image and obtains an output result, and its operating environment. Further, the learning model unit 400b may be, for example, a language learning model such as an LLM that inputs natural language and obtains an output result, and its operating environment. In that case, the input information D41 may be input in the form of text data, image data, a combination of text data and image data, or a data format convertible thereto (such as audio data, a video that is a combination of audio data and image data). Note that the learning model used in this embodiment is not limited to the above-described models.

[0306] The model interface 6 is an interface that receives model output data (analysis results D42a and D42b) from the learning model unit 400a and the learning model unit 400b and outputs them to a predetermined output destination. The model interface 6 may be, for example, an interface that converts the model output data output from the learning model unit 400a and the learning model unit 400b into data that matches a predetermined output destination and outputs it. The model interface 6 may be provided, for example, as an example of the above-described output unit 103. In this embodiment, the output destinations of the model interface 6 include the target device 2 and the display 7.

[0307] The model interface 6 may output, for example, result information D44a indicating the situation analysis result included in the analysis result D42a and the improvement method included in the analysis result D42b to the display 7, and output result information D44b indicating the improvement method included in the analysis result D42b to the target device 2. At this time, the model interface 6 may extract some data from the analysis result D42a and / or the analysis result D42b, convert it into a data format that matches the output destination, and output it as the result information D44a and the result information D44b.

[0308] In the example shown in FIG. 23, the target device 2 and the display 7 are shown as the output destinations of the model interface 6. However, the output destination of the model output data is not limited to the above. For example, when the information indicating the control of the target device 2 is included in the improvement method of the situation indicated by the model output data to be output, the model interface 6 directly outputs the model output data or the information indicating the method to the target device 2 as the implementation destination of the method. In addition to this, it is also possible to output the model output data or the information indicating the method to a conversion device (not shown) that converts the information into information that can be received by the target device 2. The conversion device may be, for example, the control system 1000 of Embodiment 1 that converts the input information into a control description or an execution code that can be discriminated by the target device 2.

[0309] Further, the model interface 6 itself may have the function of a conversion device. For example, the model interface 6 may have a function not only of output control of the model output data, but also of converting the improvement method output by the learning model unit 400b into a code that can be executed by an interpreter, and outputting the converted code or controlling a device based on the code. In addition, when the improvement method includes a process with a high degree of urgency, the model interface 6 may have a function of controlling the processing flow such as immediately executing the process. Further, the model interface 6 may have a function of transmitting a prompt input through the display of the display 7, such as an answer to the proposed method displayed on the display 7, to the learning model unit 400b.

[0310] Further, the model interface 6 may have the functions of the above-described output confirmation unit 202 and correction confirmation unit 203. For example, the model interface 6 determines the urgency of the analyzed situation, and when it is determined that the urgency is not high, it confirms the validity of the improvement method by inquiring a monitor or using a simulator or the like. If the improvement method is not valid, it may transmit that fact to the learning model unit 400b and prompt the output of the improvement method again. At that time, the model interface 6 may issue supplementary information D48 for the model input data of the target learning model unit 400b.

[0311] In this embodiment, the input information D41 corresponds to the model input data D101 of the learning model unit 400a. Also, the analysis result D42a corresponds to the model output data D103 of the learning model unit 400a. Further, the analysis result D42a corresponds to the model input data D101 of the learning model unit 400b. Also, the analysis result D42b corresponds to the model output data D103 of the learning model unit 400b. The learning model unit 400a (particularly, the model control unit 101) may be configured to output, for example, an analysis result D42a corresponding to the input information D41 based on the model information D102 and, if necessary, the model reference information D104 when receiving the input information D41. Also, the learning model unit 400b (particularly, the model control unit 101) may be configured to output, for example, an analysis result D42b corresponding to the analysis result D42a based on the model information D102 and, if necessary, the model reference information D104 when receiving the analysis result D42a.

[0312] In such a case, the model generation unit 107 provided corresponding to the learning model unit 400a may perform machine learning using, for example, the model learning data D105 including candidates for the input information D41 that can be input to the model control unit 101 to generate or update the model information D102, or may perform machine learning using the model learning data D105 including candidates for the input information D41 that can be input to the model control unit 101 and candidates for the corresponding analysis result D42a to generate or update the model information D102. Also, the model generation unit 107 provided corresponding to the learning model unit 400b may perform machine learning using, for example, the model learning data D105 including candidates for the analysis result D42a that can be input to the model control unit 101 to generate or update the model information D102, or may perform machine learning using the model learning data D105 including candidates for the analysis result D42a that can be input to the model control unit 101 and candidates for the corresponding analysis result D42b to generate or update the model information D102.

[0313] Although illustration is omitted, also in the present embodiment, state information D45 and / or feedback information D46 may be acquired from the output destination of the model output data D103 of the learning model unit 400a and the learning model unit 400b and / or information generated based thereon. The control system 4000 may output, for example, the acquired state information D45 and / or feedback information D46 as information indicating a control result to the user 1, the learning model unit 400a, the learning model unit 400b, or another device (not shown). Further, the control system 4000 may be configured to return an inquiry D47 to the user 1 when the input information D41 includes unclear or uncertain information. Further, the control system 4000 may generate supplementary information D48 for the input / output data of the learning model unit 400a and the learning model unit 400b based on the acquired state information D45 and / or feedback information D46, and issue it to the user 1, the learning model unit 400a, the learning model unit 400b, or another device (not shown). The handling of the state information D45, the feedback information D46, the inquiry D47, and the supplementary information D48 may be basically the same as in the first embodiment. Here, the output of information to the user 1 may be performed, for example, via a display 7 or an input / output interface provided in an information processing device 10 (not shown).

[0314] Further, the control system 4000 may further include a state acquisition unit 430 (not shown) that acquires the state information D45 and / or the feedback information D46 and issues supplementary information D48 as necessary. The state acquisition unit 430 is the same as the state acquisition unit 130 of the first embodiment.

[0315] Also in the present embodiment, the target device 2 is not particularly limited. Although it is assumed that the target device 2 is a device that can actually be controlled by receiving the analysis result D42b, this is not the case when the above-described conversion device is included between the target device 2 and the target device 2.

[0316] In the present embodiment, the input information D41 received by the control system 4000 can be information regarding the situation in the working environment (here, the situation in the environment where the monitoring work is performed). Therefore, the input information D41 received by the control system 4000 can be regarded as an example of the first information indicating the situation in the working environment. Further, the analysis results D42a and D42b can be regarded as information used for the work (monitoring work) corresponding to such input information D41. Hereinafter, the analysis results D42a and / or D42b output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D41 is input are sometimes referred to as the second information.

[0317] Next, the operation of the control system 4000 of the present embodiment will be described. FIG. 24 is a flowchart showing an operation example of the control system 4000.

[0318] In the example shown in FIG. 24, first, the control system 4000 receives the input information D41 (step S410). For example, the above-described input unit 102 or input processing unit 201 may receive the input information D41. The received input information D41 is input to the learning model unit 400a as the model input data D101.

[0319] Next, the control system 4000 performs a process of generating the analysis result D42a using the learning model unit 400a (step S411). In step S411, the learning model unit 400a (more specifically, the model control unit 101) outputs the analysis result D42a corresponding to the input information D41 based on the model information D102, the input input information D41, and the model reference information D104 including the device information D43 as necessary. The learning model unit 400a may generate the analysis result D42a of the text data from the input input information D41 using, for example, a learning model capable of generating text data.

[0320] In step S411, the preprocessing unit 105 and / or the postprocessing unit 106 of the learning model unit 400a may further perform the above-described processes.

[0321] The analysis result D42a output from the learning model unit 400a is input to the learning model unit 400b. Also, the analysis result D42a output from the learning model unit 400a is input to the learning model unit 400b and the model interface 6. The analysis result D42a output from the learning model unit 400a may be input to the model interface 6 via the learning model unit 400b. In that case, the learning model unit 400b may output model output data D103 including the analysis result D42a and the analysis result D42b.

[0322] Next, the control system 4000 performs a generation process of the analysis result D42b using the learning model unit 400b (step S412). In step S412, the learning model unit 400b (more specifically, the model control unit 101) outputs an analysis result D42b corresponding to the analysis result D42a based on the model information D102, the input analysis result D42a, and the model reference information D104 including the device information D43 as necessary. The learning model unit 400b may generate an analysis result D42b of binary data from the input analysis result D42a using, for example, a learning model capable of generating text data. Also, the learning model unit 400b may generate analysis results D42b of text data and binary data from the input analysis result D42a using, for example, a learning model capable of generating text data and binary data.

[0323] In step S412, the preprocessing unit 105 and / or the postprocessing unit 106 of the learning model unit 400b may further perform the above-described processing.

[0324] The analysis result D42b output from the learning model unit 400b is input to, for example, the model interface 6.

[0325] The model interface 6 controls the target device 2 and / or causes information to be displayed on the display 7 based on the analysis results by the learning model unit 400a and the learning model unit 400b (step S413). In step S413, for example, the model interface 6 outputs information based on the analysis result D42a and the analysis result D42b to a predetermined output destination. The model interface 6 outputs, for example, result information D44a indicating the situation analysis result and the improvement method to the display 7 based on the analysis result D42a and the analysis result D42b, and outputs result information D44b indicating the improvement method based on the analysis result D42b to the target device 2.

[0326] The result information D44a may indicate, for example, the situation that occurred in the working environment and the improvement method in characters and voice. Also, the result information D44b may indicate the improvement method in characters or a control signal, for example.

[0327] As a result, the display 7 displays the situation analysis result indicated by the analysis result D42a and the improvement method indicated by the analysis result D42b based on the result information D44a, and the target device 2 implements the improvement method indicated by the analysis result D42b based on the result information D44b. The information input to the display 7 and the target device 2 may be directly input from the control system 4000 (more specifically, the model interface 6), or may be indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or a person's hand.

[0328] When there is a change in the state of the target device 2 such that the target device 2 is controlled, etc., and there is feedback from the output destination, the control system 4000 may acquire the state information D45 and the feedback information D46 (step S414). Note that the process of step S414 is not essential and may be omitted as appropriate.

[0329] The control system 4000 may repeat the processes of steps S410 to S414 a plurality of times (for example, until a desired state is reached in the target working environment).

[0330] As described above, according to the present embodiment, since the situation grasping and the acquisition of improvement methods are performed in two steps using different learning models, the accuracy of the final product can be improved, and as a result, the efficiency of the work related to the monitoring of the working situation can be achieved.

[0331] For example, in a scene where the situation is grasped, it is important to widely detect an abnormal state in the working environment, such as "something strange has occurred". On the other hand, in a scene where an improvement method is acquired, specific information such as "stop this machine, move the position of the workpiece to point A, return the state of the machine to state B, and then restart it" is important.

[0332] When the abstraction levels of such information to be extracted, that is, the target information, are different, there is a concern that the accuracy of the output result will decrease if one tries to learn and extract them together using one learning model. In particular, in the acquisition of improvement methods, the presentation of specific methods is required based on the knowledge and information of the working environment. In such a case, by separating the learning models and providing appropriate domain knowledge (environmental information), the output accuracy can be more reliably improved.

[0333] In addition, when trying to obtain solutions for different tasks such as situation grasping and acquisition of improvement methods using one learning model, the problem of hallucination may become prominent. This is because the function of adjusting the solution of the other task (situation grasping) so that the solution of one task (acquisition of improvement methods) seems plausible may work implicitly in the model algorithm. According to the present embodiment, it is also effective against such a hallucination problem. That is, by separating the learning models corresponding to the two tasks of situation grasping and acquisition of improvement methods, the modality input to each learning model can be suppressed, and as a result, the magnitude of hallucination can be suppressed, so that the accuracy of the final product can be improved.

[0334] Furthermore, in the present embodiment, since the analysis results D42a and D42b, which are the output results of the learning model units 400a and 400b, can be verbalized and displayed on the display 7, by a person checking the content, hallucination can be suppressed and a method for improving the situation can be realized more reliably.

[0335] Note that the control system 4000 of the present embodiment can be applied not only to the monitoring of the control system of the devices in the factory described above, but also to the monitoring of the logistics targets in the logistics system, for example.

[0336] Modification Example 4-1. Next, a modification example of the control system 4000 will be described. FIG. 25 is a configuration diagram showing an example of a control system 4000a, which is a modification example of the control system 4000 according to the present embodiment. Note that the same elements as those of the control system 4000 are denoted by the same reference numerals and the description thereof is omitted.

[0337] In the control system 4000a shown in FIG. 25, it is different from the control system 4000 in that it includes two analysis means for analyzing and improving the situation in different ways, and appropriately switches the analysis means to be used according to the generated situation.

[0338] The control system 4000a shown in FIG. 25 includes a first analysis unit 41-1, which is a part for analyzing the situation and obtaining an improvement method using the above-described learning model units 400a and 400b, a second analysis unit 41-2, a switching unit 42, and an output switching switch 43.

[0339] The second analysis unit 41-2 may be any means that analyzes the input information D41 and obtains an improvement method in a different way from the first analysis unit 41-1. As an example, the second analysis unit 41-2 may be a means for analyzing the situation and obtaining an improvement method based on rules. For example, when the input information D41 is input, the second analysis unit 41-2 determines whether the input information D41 matches a predetermined abnormal pattern, and when it matches any abnormal pattern, it may obtain an improvement method corresponding to the abnormal pattern. The second analysis unit 41-2 outputs an analysis result D42c including at least an improvement method for the situation.

[0340] The analysis result D42c may include, for example, information corresponding to the above-described result information D44a and information corresponding to the result information D44b. In this modification example, the analysis result D42c includes at least result information D44b indicating an improvement method obtained by the second analysis unit 41-2.

[0341] In this modification example, the second analysis unit 41-2 may be realized as an internal execution module, for example, by being mounted on a PLC, an information processing device, etc. arranged in the work environment.

[0342] The switching unit 42 is a means for switching the control destination for the input information D41 according to a predetermined condition. In this modification example, the switching unit 42 switches the control destination for the input information D41 between the first analysis unit 41-1 and the second analysis unit 41-2. For example, the switching unit 42 may switch the control destination for the input information D41 according to whether the input information D41 matches an existing rule. At this time, the switching unit 42 may switch the control destination by switching the output destination of the input information D41 to the second analysis unit 41-2 when the input information D41 matches an existing rule, and switching the output destination of the input information D41 to the first analysis unit 41-1 when it does not match.

[0343] The switching unit 42 may switch the control destination for the input information D41, for example, according to an instruction from a supervisor. Further, the switching unit 42 may switch the control destination for the input information D41, for example, according to time, work content, or the presence or absence of a supervisor. Further, the switching unit 42 may switch the control destination for the input information D41, for example, according to whether or not an abnormality has occurred in the working environment. Here, the presence or absence of an abnormality in the working environment may be determined, for example, by whether or not an abnormality signal has occurred. For example, the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1 during an abnormality. Further, the switching unit 42 may switch the control destination for the input information D41, for example, according to the degree or urgency of the abnormality occurring in the working environment.

[0344] Further, the switching unit 42 may control an output switching switch 43 that switches a connection path (such as a circuit or a communication path) connecting the output of the first analysis unit 41-1 or the output of the second analysis unit 41-2 and the target device 2 and the display 7 that are the output destinations of the analysis results, in accordance with the switching of the control destination for the input information D41.

[0345] For example, when the switching unit 42 switches the control destination for the input information D41 to the first analysis unit 41-1, the switching unit 42 may control the output switching switch 43 to turn on the connection path connecting the output of the first analysis unit 41-1 and the target device 2 and the display 7, and turn off the connection path connecting the output of the second analysis unit 41-2 and the target device 2 and the display 7. Similarly, when the switching unit 42 switches the control destination for the input information D41 to the second analysis unit 41-2, the switching unit 42 may control the output switching switch 43 to turn on the connection path connecting the output of the second analysis unit 41-2 and the target device 2 and the display 7, and turn off the connection path connecting the output of the first analysis unit 41-1 and the target device 2 and the display 7.

[0346] FIG. 26 is a flowchart showing an operation example of this modified example. In the example shown in FIG. 26, when the control system 4000a receives the input information D41 at step S410, the switching unit 42 switches the control destination for the input information D41 according to a predetermined condition (step S421). In the example shown in FIG. 26, the switching unit 42 determines whether the input information D41 matches an existing rule, and if it determines that it does not match (No in step S421), it proceeds to the first analysis process (step S422). On the other hand, if it determines that the input information D41 matches an existing rule (Yes in step S421), it proceeds to the second analysis process (step S423).

[0347] In the first analysis process of step S422, the learning model units 400a and 400b as the first analysis unit 41-1 perform situation analysis and acquisition of improvement methods. The learning model unit 400a and the learning model unit 400b output an analysis result D42a including the situation analysis result and an analysis result D42b including the improvement method for the situation as the result of the first analysis process.

[0348] In the second analysis process of step S423, the second analysis unit 41-2 performs situation analysis and acquisition of improvement methods according to the existing rules. The second analysis unit 41-2 outputs, for example, an analysis result D42c including at least the improvement method for the situation as the result of the first analysis process.

[0349] When the result of the analysis process by the first analysis unit 41-1 or the second analysis unit 41-2 is output, according to the state of the output switching switch 43, based on the result of either analysis process, the target device 2 is controlled and / or information is displayed on the display 7 (step S424).

[0350] In this example, when the first analysis unit 41-1 performs analysis processing, the connection path connecting the output of the first analysis unit 41-1, the target device 2, and the display 7 is turned on. In that case, the model interface 6 may output result information D44a indicating the situation analysis result and the improvement method to the display 7 based on, for example, the analysis result D42a and the analysis result D42b, and output result information D44b indicating the improvement method based on the analysis result D42b to the target device 2. On the other hand, when the second analysis unit 41-2 performs analysis processing, the connection path connecting the output of the second analysis unit 41-2, the target device 2, and the display 7 is turned on. In that case, based on the analysis result D42c output from the second analysis unit 41-2, result information D44a indicating the situation analysis result and the improvement method may be output to the display 7, and / or result information D44b indicating the improvement method may be output to the target device 2.

[0351] For example, the result information D44b may be displayed on the display 7 in a manner that can be confirmed by the operator. In that case, the operator may refer to the result information D44b displayed on the display 7, confirm the improvement method indicated by the result information D44b, and perform the work according to the method. Further, the operator may confirm the improvement method indicated by the result information D44b and judge its validity. At this time, if the improvement method indicated by the result information D44b is not appropriate, the operator may prompt the learning model unit 400b to obtain another improvement method (re-acquisition of model output data). For example, when receiving information requesting re-acquisition of model output data, the learning model unit 400b may re-acquire the model output data after changing a part of the input information, a part of the model parameters, or the reference destination of the reference information.

[0352] The subsequent processing may be the same as that of other control systems according to this embodiment.

[0353] As described above, in this modified example, a plurality of analysis units that analyze the situation and obtain improvement methods in different ways are provided, and they are configured to be switched according to the situation. Therefore, it is possible to perform control more suitable for the situation. For example, for a problem with a clear cause, the second analysis unit with a high processing load immediately analyzes the situation and presents and executes an improvement method. For a problem with an unclear cause, the first analysis unit using a learning model analyzes the complex situation and presents and executes a better improvement method.

[0354] Note that in the above example, an example in which the first analysis unit 41-1 analyzes the situation and obtains an improvement method using two learning models is shown, but the configuration of the first analysis unit 41-1 is not limited to the above example. For example, when the analysis of the situation is not required, the learning model unit 400a can be omitted. Also, when the acquisition of the improvement method is not required, the learning model unit 400b can be omitted. It is also possible to perform the analysis of the situation and the acquisition of the improvement method with one learning model unit.

[0355] For example, when the first analysis unit 41-1 includes a learning model unit 400b that obtains an improvement method for the situation based on the input information D41, the switching unit 42 may switch the control destination for the input information D41 to the first analysis unit 41-1 in case of an abnormality. In that case, the learning model unit 400b of the first analysis unit 41-1 only needs to be configured to output information indicating an improvement method corresponding to the occurrence situation of the abnormality indicated by the input information D41 when the input information D41 is input. At this time, the learning model unit 400b may refer to the device information storage unit 410 accessible to the control system and output information indicating an improvement method corresponding to the situation.

[0356] Embodiment 5. Next, Embodiment 5 of the present invention will be described. In this embodiment, an example of assisting a response operation that returns a response to the information transmission from User 1 in a call center, a product site, etc. using a learning model will be described. Here, the information transmission from User 1 may include inquiries and opinions regarding a certain service, information, event, or object.

[0357] FIG. 27 is a configuration diagram showing an example of the control system 5000 according to Embodiment 5. The control system 5000 shown in FIG. 27 includes a learning model unit 500, a reference information storage unit 12, a database search unit 511 (referred to as a DB search unit in the figure), a control generation unit 512, a voice recognition unit 513v, and a voice synthesis unit 514v. Here, the reference information storage unit 12, the database search unit 511, and the control generation unit 512 may be provided as a part of the learning model unit 500.

[0358] When the input information D51 is input, the learning model unit 500 outputs response information D52 indicating the response content. For example, when the input information D51 is input, the learning model unit 500 outputs the response information D52 based on the model information D102. The configuration of the learning model unit 500 may be basically the same as that of the learning model unit 100 in Embodiment 1.

[0359] In the present embodiment, the learning model unit 500 is a model and its operating environment configured to output response information D52 corresponding to the input information D51 when the input information D51 is input. Further, the learning model unit 500 may be a model and its operating environment configured to generate and output the response information D52 based on the input information D51 and other information that can be referred to in the learning model unit 500 when the input information D51 is input.

[0360] In this embodiment, the input information D51 includes information indicating the content transmitted from the user 1 or the like. The input information D51 may include information indicating the content for which a response is required in the working environment. The input information D51 may be, for example, text, an image, voice, or a combination thereof indicating an inquiry or opinion regarding a certain service, information, event, or object. The input information D51 may be, for example, text, an image, voice, or a combination thereof indicating a plurality of inquiries or opinions regarding a certain service, information, event, or object. Further, the input information D51 may include information indicating the transmitted content that is temporally continuous. In that case, it may be time-series data having a predetermined data structure including text, an image, voice, or a combination thereof indicating the transmitted content as described above. It is assumed that the way of indicating the transmitted content conforms to the input format of the model used by the learning model unit 500, but this is not the case when error processing, correction processing, or conversion processing is provided in the stage before the learning model unit 500.

[0361] The response information D52 includes information indicating a response to the transmitted content included in the input information D51. The response information D52 may be, for example, information indicating a response to an inquiry or opinion regarding a service, information, event, or object indicated by the transmitted content included in the input information D51.

[0362] The reference information storage unit 12 stores model reference information D104 that the model control unit 101 of the learning model unit 500 refers to in order to output the response information D52. The model reference information D104 includes, for example, information related to services, information, events, or things that may be included in the input information D51. Here, the reference information storage unit 12 may specifically store information related to a specific service, information, event, or thing as the model reference information D104. The model reference information D104 may include, for example, a response manual in digital form. Also, the model reference information D104 may include, for example, the input information D51 input in the past or the history of the transmission content included therein. At this time, the reference information storage unit 12 may store, together with the information of the user 1 as the source (for example, user identifier, attribute information of the user 1, etc.), the input information D51 input in the past or the history information indicating the transmission content included therein as the model reference information D104. Hereinafter, in the present embodiment, information indicating the state of the user 1 as the source may sometimes be referred to as state information D55.

[0363] The database search unit 511 is a search engine for the reference information storage unit 12 and other databases. The database search unit 511 searches the database to which the database search unit 511 is accessibly connected in response to a request from the model control unit 101 of the learning model unit 500 and outputs the search result. At this time, the database search unit 511 may have a restricted database as the access destination.

[0364] The control generation unit 512 is an interface for setting the preconditions when the learning model unit 500 (particularly, the model control unit 101) generates model output data. The control generation unit 512 may be, for example, an interface used to recognize the information to be controlled by the learning model unit 500 and / or set the tendency of the output. Here, the information to be controlled is information indicating the target to which the focus of control in the model control unit 101 is applied. The model control unit 101 may be configured to generate the model output data D103 from the model input data D101 based on, for example, the information to be controlled indicated by the control generation unit 512. The control generation unit 512 may, for example, cause the model control unit 101 to recognize a part of the model input data input by the user 1 as the information to be controlled, cause the model control unit 101 to recognize the information generated by the model control unit 101 as the information to be controlled, or cause the model control unit 101 to recognize the information generated by the model control unit 101 and modified by another control unit as the information to be controlled. The setting of the information to be controlled and / or the output tendency may be specified by the user 1, may be specified by an external processing unit, or may be specified by the control generation unit 512 according to a predetermined algorithm.

[0365] When the input information D51v in audio format is included in the input from the user 1, the speech recognition unit 513v recognizes the speech indicated by the input information D51v, converts it into a format that matches the data format of the learning model unit 500, and outputs it. The speech recognition unit 513v may, for example, convert the input information D51v in audio format into the input information D51 in text format.

[0366] The voice synthesis unit 514v converts the content indicated by the response information D52 into a voice format and outputs it. For example, when the response information D52, which is the output from the learning model unit 500, contains a data format other than voice, the voice synthesis unit 514v converts the content of that part indicated by the response information D52 into a voice format and outputs it. For example, when the response information D52 is a data structure including a specification of a data format, the voice synthesis unit 514v may convert a data element specified as a voice format in the specification into a voice format and output it. For example, the voice synthesis unit 514v may convert the response information D52 in text format into the response information D52v in voice format.

[0367] Note that in the above example, an example in which voice format data is used for the input / output with the user 1 is shown, but the data format used for the input / output with the user 1 is not limited to the voice format. In that case, instead of the voice recognition unit 513v and the voice synthesis unit 514v, a processing unit that converts the data format used for the input from the user 1 into the data format used for the input of the learning model unit 500 and a processing unit that converts the data format used for the output from the learning model unit 500 into the data format used for the input of the user 1 may be provided.

[0368] Also, when the learning model unit 500 can receive the data format used for the input from the user 1, the voice recognition unit 513v can be omitted. Also, when the user 1 can receive the data format used for the output from the learning model unit 500, the voice synthesis unit 514v can be omitted.

[0369] In the present embodiment, the input information D51 corresponds to the model input data D101. Also, the response information D52 corresponds to the model output data D103. The learning model unit 500 (particularly, the model control unit 101) may be configured to output the response information D52 corresponding to the input information D51 based on the model information D102 and, if necessary, the model reference information D104 when receiving the input information D51, for example.

[0370] Also, in such a case, the model generation unit 107 provided corresponding to the learning model unit 500 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D51 that can be input to the model control unit 101 to generate or update the model information D102. Further, the model generation unit 107 may perform machine learning using, for example, the model learning data D105 including candidates for the input information D51 that can be input to the model control unit 101 and candidates for the corresponding response information D52 to generate or update the model information D102.

[0371] Although not shown in the figure, also in the present embodiment, the state information D55 and / or the feedback information D56 may be acquired from the model output data D103 of the learning model unit 500 and / or the output destination of the information generated based on it. The control system 5000 may output, for example, the acquired state information D55 and / or the feedback information D56 as information indicating the response result to a predetermined supervisor, the learning model unit 500, or another device (not shown). Further, the control system 5000 may be configured to return an inquiry D57 to the user 1 when the input information D51 includes unclear or uncertain information. Also, the control system 5000 can generate supplementary information D58 for the input / output data of the learning model unit 500 based on the acquired state information D55 and / or the feedback information D56 and issue it to the user 1, a predetermined supervisor, the learning model unit 500, or another device (not shown). The handling of the state information D55, the feedback information D56, the inquiry D57, and the supplementary information D58 may be basically the same as in the first embodiment.

[0372] Also, the control system 5000 may further include a state acquisition unit 530 (not shown) that acquires the state information D55 and / or the feedback information D56 and issues the supplementary information D58 as necessary. The state acquisition unit 530 is the same as the state acquisition unit 130 in the first embodiment.

[0373] In the present embodiment, the input information D51 received by the control system 5000 can be information regarding a request in the working environment (here, the content of a transmission that requests a response in an environment where a response operation is performed in response to an inquiry). Therefore, the input information D51 received by the control system 5000 can be regarded as an example of first information indicating a request in the working environment. Further, the response information D52 can be information used for the operation (response operation) corresponding to such input information D51. Hereinafter, the response information D52 output to a predetermined output destination from the operating environment of the learning model to which the model input data based on the input information D51 is input will be referred to as second information.

[0374] Next, the operation of the control system 5000 of the present embodiment will be described. FIG. 28 is a flowchart showing an operation example of the control system 5000.

[0375] In the example shown in FIG. 28, first, the control system 5000 receives input information D51v (step S510). For example, the above-described input unit 102 or input processing unit 201 may receive the input information D51v. The received input information D51v is input to the speech recognition unit 513v.

[0376] When the speech recognition unit 513v receives the input information D51v, it recognizes the speech included in the input information D51v and converts it into input information D51 that matches the data format of the input to the learning model unit 500 (step S511). The converted input information D51 is input to the learning model unit 500 as model input data D101.

[0377] When the speech recognition unit 513v is omitted, the received input information D51v may be input to the learning model unit 500 as model input data D101.

[0378] Next, the control system 5000 performs a process of generating response information D52 using the learning model unit 500 (step S512). In step S512, the learning model unit 500 (more specifically, the model control unit 101) outputs response information D52 corresponding to the input information D51 based on the model information D102, the input input information D51, and, if necessary, the model reference information D104.

[0379] In step S512, the preprocessing unit 105 and / or the postprocessing unit 106 of the learning model unit 500 may further perform the above-described processes.

[0380] The response information D52 output from the learning model unit 500 is input to, for example, the voice synthesis unit 514v (step S513). The input of the response information D52 to the voice synthesis unit 514v may be directly input from the control system 5000 (more specifically, the learning model unit 500 or the information processing apparatus 10 as its operating environment, etc.), or may be indirectly input via a communication network, other devices (servers, various conversion devices, etc.), or a person's hand.

[0381] Next, the voice synthesis unit 514v converts the input response information D52 into voice-formatted response information D52v and outputs it (step S514). The voice synthesis unit 514v may generate the response information D52v, for example, by synthesizing a voice for uttering the response content indicated by the response information D52 in a data format other than voice. The response information D52v is output toward the user 1 who is the source of the input information D51v (step S515).

[0382] Note that when the voice synthesis unit 514v is omitted, the response information D52 output from the learning model unit 500 may be output toward the user 1 who is the source of the input information D51v.

[0383] As described above, in the present embodiment, even without preparing an operator or a site or the like in which the content to be pre-responded is embedded for the transmission information from the user 1, the response information D52 can be dynamically generated using the learning model unit 500 and returned to the user 1 who is the transmission source. Therefore, it is possible to improve the efficiency and performance of the response operation.

[0384] Modification Example 5-1. Next, a modification example of the control system 5000 will be described. FIG. 29 is a configuration diagram showing an example of a control system 5000a which is a modification example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control system 5000 are denoted by the same reference numerals and the description thereof is omitted.

[0385] In the control system 5000a shown in FIG. 29, the difference from the control system 5000 is that it includes a correctness determination unit 515.

[0386] The correctness determination unit 515 determines whether the content indicated by the response information D52 which is the output from the learning model unit 500 is correct. For example, the correctness determination unit 515 may output the response information D52 to the user 1 or update the content of the reference information storage unit 12 only when it determines that the content indicated by the response information D52 is correct.

[0387] Further, for example, when the correctness determination unit 515 determines that the content indicated by the response information D52 is not correct, it may prompt the learning model unit 500 to acquire another response information D52 (re-acquire the model output data). The correctness determination unit 515 may be provided, for example, as an example of the post-processing unit 106 described above.

[0388] Regarding other points, it may be the same as other control systems according to the present embodiment.

[0389] As described above, according to this modification example, it is determined whether the content indicated by the response information, which is the output from the learning model unit 500, is correct. Based on the result, the presence or absence of output to the user 1, the re-acquisition of the response information, and the update of the reference information are performed, so that the performance of the response operation can be further improved.

[0390] Modification Example 5-2. Next, a second modification example of the control system 5000 will be described. FIG. 30 is a configuration diagram showing an example of a control system 5000b, which is a modification example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control system 5000 and the control system 5000a are denoted by the same reference numerals, and the description thereof is omitted.

[0391] As shown in FIG. 30, the control system 5000b may further include an emotion determination unit 516.

[0392] The emotion determination unit 516 determines the emotion of the user 1 as the source using the input information D51 and other information. Further, the emotion determination unit 516 may determine the emotion of the user 1 after outputting the response information D52 from the learning model unit 500 to the user 1.

[0393] The emotion of the user 1 determined by the emotion determination unit 516 may be input to the learning model unit 500 as the state information D55 included in the model reference information D104, or may be recorded as a history together with the input / output data of the model in the reference information storage unit 12.

[0394] As a method of recording in the reference information storage unit 12, for example, the control system 5000b may further include a registration determination unit 518, and the registration determination unit 518 may determine whether to record in the reference information storage unit 12 based on the determination result of the emotion of the user 1 by the emotion determination unit 516.

[0395] For example, if the emotion of the determined User 1 is positive, the registration determination unit 518 may record the input / output data of the model as history information in the reference information storage unit 12 as a good example. At this time, if there is a determination result of the emotion of User 1 before the output of the response information D52 from the learning model unit 500, the registration determination unit 518 may record the input / output data of the model including the emotion information before and after the response as history information in the reference information storage unit 12.

[0396] Also, for example, if the emotion of the determined User 1 is negative, the registration determination unit 518 may record the input / output data of the model as history information in the reference information storage unit 12 as a bad example. At this time, if there is a determination result of the emotion of User 1 before the output of the response information D52 from the learning model unit 500, the registration determination unit 518 may record the input / output data of the model including the emotion information before and after the response as history information in the reference information storage unit 12.

[0397] In addition, the control system 5000b further includes an additional learning unit 519, and when updating the content of the reference information storage unit 12, based on the update information, it may reconstruct (additional learning) the model reference information D104 stored in the reference information storage unit 12 and the information referred to by the other model control unit 101.

[0398] In addition, the control system 5000b may include an evaluation acquisition unit 517 instead of or in addition to the emotion determination unit 516.

[0399] The evaluation acquisition unit 517 inquires the evaluation of the response information D52 from User 1 and acquires evaluation information D59 as the answer. The evaluation information D59 can be used, for example, for updating the information referred to by the model, additional learning, etc., similar to the emotion of User 1 described above.

[0400] In addition, the control system 5000b may further include a control decision unit 520.

[0401] The control determination unit 520 designates the information to be controlled and / or designates the tendency setting of the output to the control generation unit 512 based on the speech recognition result, the emotion determination result, and / or the evaluation result of the response information D52 for the input information from the user 1, instructions from an operator (not shown), etc. Here, the speech recognition result for the input information from the user 1 may include information such as the attributes, emotions, regions, languages, presence or absence of past usage, usage frequency, etc. of the user 1. Further, the control determination unit 520 may set the synthesized speech for the speech synthesis unit 514v based on the speech recognition result, the emotion determination result, and / or the evaluation result of the response information D52 for the input information from the user 1, instructions from an operator (not shown), etc.

[0402] For example, as an example of the tendency setting of the output, the control determination unit 520 can specify the difficulty level of the explanation in the response, the way of speaking (tone and intonation), language, grammar level, politeness, the position of the speaker, the destination of the conversation, etc. The gender, tone, intonation, etc. of the synthesized speech can be specified. Further, for example, as an example of the setting of the synthesized speech, the control determination unit 520 can specify the gender, way of speaking, language, grammar level, politeness, etc. of the synthesized speech. The control determination unit 520 may perform these settings based on, for example, predetermined setting rules.

[0403] Note that the elements of the control system 5000b shown in FIG. 30 can be appropriately selected according to the desired functions.

[0404] Regarding other points, it may be the same as other control systems according to this embodiment.

[0405] As described above, according to this modification example, since the control determination unit 520 designates the information to be controlled and / or designates the tendency setting of the output based on the information that can be obtained from the control system 5000b, etc., it is possible to generate response information that easily matches the request of the source. For this reason, it is possible to further improve the performance of the response work to the user 1.

[0406] Modification Example 5-3. Next, a third modification example of the control system 5000 will be described. FIG. 31 is a configuration diagram showing an example of a control system 5000c, which is a modification example of the control system 5000 according to the present embodiment. Note that the same elements as those in the control system 5000, the control system 5000a, and the control system 5000b are denoted by the same reference numerals and the description thereof will be omitted.

[0407] As shown in FIG. 31, the control system 5000c may further include an image analysis unit 513i, an image generation unit 514i, and a program generation unit 514p.

[0408] When the input information D51i in the image format is included in the input from the user 1, the image analysis unit 513i analyzes the image indicated by the input information D51i and converts it into a format that matches the data format of the learning model unit 500 and outputs the converted image. The image analysis unit 513i may, for example, convert the input information D51i in the image format into the input information D51 in the text format.

[0409] For example, when the input from the user 1 includes an image capturing the operation screen of the product held by the user 1, the image analysis unit 513i analyzes the image to identify which product's operation screen it is and what operation state it is in, and may convert it into text explaining the same and output the text. Further, for example, when the input from the user 1 includes an image capturing a certain purchase site that the user 1 is browsing, the image analysis unit 513i analyzes the image to identify which site's operation screen it is and what operation state it is in, and may convert it into text explaining the same and output the text.

[0410] The image generation unit 514i generates and outputs an image based on the response information D52. For example, when the response information D52, which is the output from the learning model unit 500, contains a data format other than an image, the image generation unit 514i may generate and output an image showing the content of the relevant part indicated by the response information D52. For example, when the response information D52 is a data structure including a data format specification, the image generation unit 514i may convert the data element specified as an image format in the specification into an image format and output it. For example, the image generation unit 514i may generate response information D52v in an image format based on the response information D52 in a text format. For example, the image generation unit 514i may perform a synthesis process of adding, as an annotation, the content indicated by the text-format response information D52 to the image included in the input information D51. Also, based on the text-format response information D52, the image generation unit 514i may perform a process of highlighting a part of the image included in the input information D51. The image generation unit 514i may generate an image from input information (response information D52 and, if necessary, input information D51) using a learning model.

[0411] The program generation unit 514p converts the content indicated by the response information D52 into a data format of a predetermined program and outputs it. For example, when the response information D52, which is the output from the learning model unit 500, contains a data format other than the data format of a predetermined program, the program generation unit 514p converts the content of the relevant part indicated by the response information D52 into the data format of a predetermined program and outputs it. For example, when the response information D52 is a data structure including a data format specification, the program generation unit 514p may convert the data element specified as the data format of a predetermined program in the specification into the data format of a predetermined program and output it. For example, the program generation unit 514p may convert the text-format response information D52 into response information D52p in the data format of a predetermined program. The program generation unit 514p may generate a predetermined program from input information using a learning model.

[0412] The image analysis process by the image analysis unit 513i is performed, for example, in step S511 described above. Further, the image generation process by the image generation unit 514i and the program generation process by the program generation unit 514p are performed, for example, in step S514 described above.

[0413] Regarding other points, it may be the same as other control systems according to this embodiment.

[0414] As described above, according to this modification example, inquiries and responses can be made not only by voice but also by voice and images. Therefore, for example, responses can be made more effectively to inquiries such as inquiries about the operation screen. Further, according to this modification example, in addition to voice and images, a program can also be provided to the source as response information, so that responses can be made more effectively to inquiries such as defect handling.

[0415] Modification Example 5-4. Next, a fourth modification example of the control system 5000 will be described. FIG. 32 is a configuration diagram showing an example of a control system 5000d which is a modification example of the control system 5000 according to this embodiment. Note that the same elements as those of the control systems 5000 to 5000c are denoted by the same reference numerals and the description thereof is omitted.

[0416] In this modification example, it has a function of switching to a response by the operator 8 or a response by another learning model based on the inquiry content from the user 1 and / or the output result from the learning model.

[0417] As shown in FIG. 32, the control system 5000d can further include a call confirmation unit 531 and an output selection unit 532.

[0418] Here, it is assumed that the control system 5000d includes a learning model unit 500a as a first response function and a communication channel with the operator 8 and the operator 8 as a second response function. Further, the control system 5000d may further include another learning model unit 500b whose algorithm or data used is different from that of the learning model unit 500a as a third response function. Note that as the second response function, another learning model unit 500b whose algorithm or data used is different from that of the learning model unit 500a may be provided. In that case, as the third response function, a communication channel with the operator 8 and the operator 8 may be further provided. Note that the types and numbers of response functions are not particularly limited. For example, the switching destination response function may be a response system that does not use a learning model.

[0419] In this example, a case will be described in which the learning model unit 500a regarded as the first response function is the above-described learning model unit 500, the second response function is a communication channel with the operator 8 and the operator 8, and the third response function is another learning model unit 500b whose algorithm or data used is different from that of the learning model unit 500a.

[0420] Here, the learning model unit 500a may be a local learning model that obtains an output result based on local information such as the reference destination database being restricted, and the learning model unit 500b may be a global learning model that obtains an output result based on global information such as being able to freely access an external network.

[0421] The call confirmation unit 531 switches the processing destination for performing response processing based on the inquiry content from the user 1 and / or the output result from the learning model.

[0422] The call confirmation unit 531 may call the operator 8 as a second response function, for example, when it is determined that the accuracy of the output by the first response function cannot be expected based on the inquiry content from the user 1 and / or the output result from the learning model. The call confirmation unit 531 may call the operator 8 using, for example, the communication channel with the operator 8 and input the input information D51 into the operating device of the operator 8. Further, the call confirmation unit 531 may call the operator 8 using the communication channel with the operator 8 and input the input information D51 into the operation terminal (not shown) of the operator 8.

[0423] Further, when the call confirmation unit 531 determines that the call to the second response function is impossible or the accuracy of the output cannot be expected, the call confirmation unit 531 may call the learning model unit 500b as a third response function. The call confirmation unit 531 may call the learning model unit 500b, for example, by inputting the input information D51 into the learning model unit 500b using the interface with the learning model unit 500b.

[0424] Here, the determination of the output accuracy may be made, for example, using the evaluation value or likelihood output by the response function itself, or using the reliability evaluation described above. Further, when the response function itself outputs a message indicating that it does not know or requesting the call of another function, it is also possible to make a determination based on the presence or absence of such a message.

[0425] The output selection unit 532 selects the response information D52 to be output to the user 1 based on the switching result of the response process by the call confirmation unit 531. When the switching result of the response process by the call confirmation unit 531 and the execution entity of the response process is the first response function, the output selection unit 532 outputs the response information D52a output from the first response function to the user 1. Further, when the switching result of the response process by the call confirmation unit 531 and the execution entity of the response process is the second response function, the output selection unit 532 outputs the response information D52b output from the second response function to the user 1. Further, when the switching result of the response process by the call confirmation unit 531 and the execution entity of the response process is the third response function, the output selection unit 532 outputs the response information D52c output from the third response function to the user 1.

[0426] The output selection unit 532 may output the output from the selected response function to the user 1 by controlling an output switching switch (not shown) that switches a connection path (such as a circuit or a communication path) connecting the response function that is the execution entity and the user 1 who is the output destination.

[0427] Here, the connection path between the response function and the user 1 may include various conversion devices such as the above-described speech synthesis unit, image generation unit, program generation unit, etc. and a predetermined interface as necessary.

[0428] For example, when what the operator 8 inputs in characters using the operation terminal is output as the response information D52b, the connection path between the response function and the user 1 may include a speech synthesis unit that converts text into speech. Further, the output selection unit 532 can also receive what is obtained by modifying the response information D52a output by the first response function as the output of the second response function or the like. In this case, the operation terminal of the operator 8 includes a text display unit and a text input unit, and the control system 5000d may receive the response information D52b obtained by modifying a part of the response information D52b output from the operation terminal of the operator 8, for example.

[0429] Regarding other points, it may be the same as other control systems according to this embodiment.

[0430] As described above, according to this modification example, in addition to generating a response using the learning model unit 500 described above, for example, response generation by an operator or generation of a response using another learning model (for example, including a tandem structure model in which a plurality of models are connected, a multimodal model, or a model learned specifically for a predetermined device or service) can be performed. Therefore, it is possible to further improve the performance of the response operation to the user 1.

[0431] In each of the above-described embodiments, an example of the system configuration according to the operation of interest has been illustrated and described. However, the control system according to the present disclosure is not limited to the above-described examples. For example, the control system according to the present disclosure can also appropriately combine one or more of the above-described embodiments.

[0432] As an example, the control system according to the present disclosure can combine the configuration of Embodiment 1 and the configuration of Embodiment 4, input information indicating a solution method obtained from sensor data using the function of Embodiment 4 into the control system of Embodiment 1, convert it into a program, and directly control the target device 2.

[0433] In addition, each embodiment and modification example are not limited to the above-described examples and can be appropriately changed within the scope of the disclosure.

Industrial Applicability

[0434] The control system according to the present disclosure can be suitably applied as a part of a work support system that supports work by a person or an object. In addition, the control system according to the present disclosure can be suitably applied as a control system that controls a device when performing some control or work using the device. Here, the control system can also be suitably applied as a control system that controls FA devices, a control system in a home or a building, or a control system that controls an information processing device such as a server device that performs information processing on a network.

Description of Symbols

[0435] 1000, 1000a, 1000b, 1000c, 2000, 3000, 3000a, 3000b, 3000c, 3000d, 3000e, 4000, 4000a, 5000, 5000a, 5000b, 5000d Control System 100, 200, 300, 300a, 300b, 400, 400a, 400b, 500, 500a, 500b Learning Model Unit 10, 20 Information Processing Device 11 Model Information Storage Unit 12 Reference Information Storage Unit 101 Model Control Unit 102 Input Unit 103 Output Unit 105 Preprocessing Unit 106 Postprocessing Unit 107 Model Generation Unit 104, 104a Control Unit 201 Input Processing Unit 202 Output Confirmation Unit 203 Correction Confirmation Unit 1 User 1a Input Source 2 Target Device 2a Output Destination 3 Operator Screen User Interface 4 Controller 41-1, 41-2 Analysis Unit 42 Switching Unit 43 Output Switch 5 Sensor 6 Model Interface 7 Display 8 Operator 110, 210, 310, 410 Device Information Storage Unit 120 Execution Code Generation Unit 230 State Acquisition Unit 311 Input Interface 312 Output Interface 313 Environment Information Storage Unit 511 Database Search Unit 512 Control Generation Unit 513v Voice Recognition Unit 513i Image Analysis Unit 514v Image Synthesis Unit 514p Program Generation Unit 515 Correct / Error Judgment Unit 516 Emotion Judgment Unit 517 Evaluation Acquisition Unit 518 Registration Judgment Unit 519 Additional Learning Unit 531 Call Confirmation Unit 532 Output Selection Unit 533 Output Switching Unit D101 Model Input Data D102 Model Information D103 Model Output Data D104 Model Reference Information D11, D21, D31, D41, D51, D51v, D51i Input Information D12 Control Description D22, D34 Control Commands D42a, D42b Analysis Results D52, D52v, D52i, D52p, D52a, D52b, D52c Response Information D32, D32a, D32b Operation Commands D320 Operation Information D13, D23, D33, D43 Device Information D33a Environment Information D14 Execution Code D44a, D44b Result Information D15, D25, D35, D45 Status Information D16, D26, D36, D46 Feedback Information D17, D27, D37, D47, D57 Inquiries D18, D28, D38, D48 Supplementary Information D59 Evaluation Information

Claims

1. A user terminal that generates input information including information indicating control content required for a device in response to an operation by a user, A control description generated by a learning model unit that generates a control description based on the input information generated by the user terminal, or a state acquisition unit that acquires information regarding at least any one of the operations of the device and feeds back the information to at least one of the user terminal or the learning model unit, The user terminal receives, from the learning model unit, an inquiry that re-asks the input information as an inquiry for confirmation of an instruction included in the input information, or outputs an inquiry that requests re-input to a changed state or expression to the user, determines the user's emotion using the user's input information input to the learning model unit, or further includes an emotion determination unit that determines the user's emotion after outputting information regarding at least any one of the control description generated by the learning model unit or the operation of the device to the user, The user's emotion determined by the emotion determination unit is input to the learning model unit or recorded together with the input / output data of the learning model unit, A control system characterized by the above.

2. An execution code generation unit that generates an execution code for the device based on the control description generated by the learning model unit, The learning model unit outputs the generated control description to the execution code generation unit, The state acquisition unit acquires information regarding at least any one of the control description generated by the learning model unit, the execution code generated by the execution code generation unit, or the operation of the device, and feeds back the information to at least one of the user terminal or the learning model unit, The control system according to claim 1, characterized by the above.

3. An execution code generation unit that generates execution code for the device based on the control description generated by the learning model unit. The learning model unit outputs the generated control description to the user terminal. The user terminal outputs the control description output by the learning model unit to the execution code generation unit according to an operation by the user, or causes the learning model unit to output the control description to the execution code generation unit. The state acquisition unit acquires information related to at least one of the execution code generated by the execution code generation unit or the operation of the device, and feeds back the information to at least one of the user terminal or the learning model unit. The control system according to claim 1, characterized in that.

4. The learning model unit uses the information stored in the device information storage unit that stores information related to the device as additional information when generating the control description. The control system according to any one of claims 1 to 3, characterized in that.

5. The input information generated by the user terminal is at least one of a document string explaining the specification, a specification document, or a specification document, design document, operation instruction, control code, or source code applied to another device. The control system according to any one of claims 1 to 3, characterized in that.

6. When the user terminal determines that the generated input information does not match the rule, the user terminal outputs the generated input information to the learning model unit. When the user terminal determines that the generated input information matches the rule, the user terminal outputs the generated input information or the corresponding control description to the execution code generation unit. The control system according to claim 2 or claim 3, characterized in that.

7. A control system for assisting work by a person or an object using a device, An input interface that receives an input of first information indicating a situation or a request in a work environment, which is an environment where the work is performed; A model processing unit that is provided to be accessible to a predetermined learning model; An output interface that outputs second information for assisting the work based on an output from the learning model; When the first information includes an instruction word, an inquiry unit that makes an inquiry to re-enter the first information as an inquiry for confirmation of the instruction word to the input source, or makes an inquiry to request re-entry to something with a changed state or expression; An emotion determination unit that determines the emotion of a user using the input information of the user input to the learning model, or determines the emotion of the user after outputting information regarding at least any one of the control description generated by the learning model or the operation of the device to the user; The model processing unit inputs model input data based on the first information to the learning model and receives model output data corresponding to the model input data from the learning model; The model output data includes information used for the work; The output interface outputs the second information based on the model output data; The emotion of the user determined by the emotion determination unit is input to the learning model or recorded together with the input / output data of the learning model; A control system characterized by the above.

8. The first information includes information indicating control content or operation content required for the device; The model input data is data indicating the control content or operation content indicated by the first information in a format that matches the input of the learning model; The model output data includes information used for controlling or operating the device corresponding to the control content or operation content indicated by the model input data; The second information includes information in which information used for controlling or operating the device included in the model output data is described in a predetermined format distinguishable at the output destination of the output interface. The control system according to claim 7.

9. The output destination of the output interface is the device or an interface that requests control of the device, As a result of the second information being output to the device or an interface that requests control of the device, the device is controlled. The control system according to claim 8.

10. The control system further includes an execution code generation unit that generates and outputs an execution code that is executable code of the device, The output destination of the output interface is the execution code generation unit, As a result of the second information being output to the execution code generation unit, the device is controlled by the generated execution code. The control system according to claim 8.

11. The output destination of the output interface is a user terminal operated by a user, As a result of the second information being output to the user terminal, the device is controlled. The control system according to claim 8.

12. The first information includes information indicating the situation in the working environment, The model input data is data in which the situation of the working environment indicated by the first information is shown in a format that matches the input of the learning model, The model output data includes information regarding the analysis result of the situation of the working environment indicated by the model input data and / or a method for improving the situation, The second information includes information in which information regarding the analysis result of the situation in the working environment and / or a method for improving the situation is described in a predetermined format distinguishable at the output destination of the output interface. The control system according to claim 7.

13. The model processing unit is provided to be accessible to a first learning model and a second learning model, the model processing unit inputs first model input data based on the first information into the first learning model, and receives first model output data corresponding to the first model input data from the first learning model, the model processing unit inputs second model input data based on the first model output data into the second learning model, and receives second model output data corresponding to the second model input data from the second learning model, the output interface outputs the second information based on the second model output data The control system according to claim 7.

14. The first information includes information indicating control content or operation content required for the device, the first model input data is data represented in a format in which the control content or operation content indicated by the first information matches the input of the first learning model, the first model output data includes information in which the control content or operation content indicated by the first model input data is represented in a more generalized or specific form, the second model input data is data represented in a format in which the control content or operation content indicated by the first model output data matches the input of the second learning model, the second model output data includes information used for control or operation of the device corresponding to the control content or operation content indicated by the second model input data, the second information includes information in which the information used for control or operation of the device included in the second model output data is described in a predetermined format distinguishable at the output destination of the output interface The control system according to claim 13.

15. The first information includes information indicating the situation in the working environment. The first model input data is data represented in a format in which the situation of the working environment indicated by the first information matches the input of the first learning model. The first model output data includes the analysis result of the situation of the working environment indicated by the model input data. The second model input data is data represented in a format in which the analysis result of the situation of the working environment indicated by the first model output data matches the input of the second learning model. The second model output data includes information regarding a method for improving the situation in the working environment corresponding to the analysis result of the work indicated by the second model input data. The second information includes information in which at least the information regarding the method for improving the situation in the working environment included in the second model output data is described in a predetermined format that can be discriminated at the output destination of the output interface. The control system according to claim 13.

16. A control system for assisting a response operation by a person or an object using equipment, The first information includes information indicating a reaction requirement content which is the content for which a reaction is required in the working environment. The model input data is data represented in a format in which the reaction requirement content indicated by the first information matches the input of the learning model. The model output data includes information used for the response operation corresponding to the reaction requirement content indicated by the model input data. The second information includes information in which the information used for the response operation included in the model output data is described in a predetermined format that can be discriminated at the output destination of the output interface. The control system according to claim 7.

17. The output destination of the output interface is a screen operation interface that requests control of the equipment via an operation screen. The model output data is an operation screen for actually performing an operation corresponding to the control content or operation content indicated by the model input data on the device, and includes information on the operation screen described in a predetermined format that can be determined at the output destination of the output interface. The control system according to claim 9.

18. The input interface receives input of first information indicating requests in the work environment from a plurality of users. The model processing unit inputs model input data including the first information input from the plurality of users to the learning model, and receives model output data corresponding to the model input data from the learning model. The control system according to any one of claims 7 to 17.

19. The learning model is a language learning model that inputs natural language to obtain an output result, an image learning model that inputs an image to obtain an output result, and a multimodal model that inputs natural language and an image to obtain an output result. The control system according to any one of claims 7 to 17.

20. The model processing unit is provided so as to be accessible to a first learning model and a second learning model. One of the first learning model and the second learning model is a local learning model in which the reference destination database is restricted to internal information. The other of the first learning model and the second learning model is a global learning model in which the reference destination database is not restricted to internal information. The control system according to any one of claims 7 to 17.

21. The model processing unit is provided so as to be accessible to a first learning model and a second learning model. One of the first learning model and the second learning model is a learning model that can refer to information specifically defined in the work environment. The other of the first learning model and the second learning model is a learning model that cannot refer to information specifically defined in the working environment. The control system according to any one of claims 7 to 17.

22. An output confirmation unit that performs a simulation that simulates the control and state of the device based on the model output data output from the learning model is provided. The control system according to any one of claims 7 to 17.

23. Based on the information collected from the output destination of the output interface, additional learning of the learning model, or determination of the correctness of the output information, and flow control of the output information are performed. The control system according to any one of claims 7 to 17.

24. When there are a plurality of options with the highest possibility for the unclear content included in the first information, the input processing unit updates the first information with the content of each option or determines the interpretation of the first information. The control system according to any one of claims 7 to 17.

25. The inquiry includes intermediate control descriptions and information indicating corrections, additions, and cancellations to the input / output data of the learning model. The control system according to any one of claims 7 to 17.

26. The first information is time-series data indicating the situation or request in the working environment, which is the environment where the work is performed, together with time information. The control system according to any one of claims 7 to 17.

27. As an execution environment of the learning model, a model information storage unit that stores model information, and a model control unit that receives the model input data and outputs the model output data based on the model input data and the information stored in the model information storage unit are provided. The control system according to any one of claims 7 to 17.

28. A user terminal generates input information including information indicating control content required for a device in response to an operation by a user, A state acquisition unit acquires control description generated by a learning model unit that generates a control description based on the input information generated by the user terminal, or information regarding at least any one of the operations of the device, and feeds back the information to at least one of the user terminal or the learning model unit, The user terminal receives, from the learning model unit, an inquiry for re-asking the input information as an inquiry for confirmation of an instruction included in the input information, or an inquiry for requesting re-input to a state or expression that has been changed, or outputs the inquiry to the user, An emotion determination unit determines the emotion of a user using the input information of the user input to the learning model unit, or determines the emotion of the user after outputting information regarding at least any one of the control description generated by the learning model unit or the operation of the device to the user, The emotion of the user determined by the emotion determination unit is input to the learning model unit or recorded together with the input / output data of the learning model unit. A control method characterized by the above.

29. A control method for assisting work by a person or an object using a device, An input interface receives input of first information indicating a situation or a desire in a work environment which is an environment in which the work is performed, A model processing unit provided to be accessible to a predetermined learning model inputs model input data based on the first information to the learning model, and receives model output data corresponding to the model input data from the learning model, the model output data including information used for the work, The output interface outputs second information based on the output from the learning model, the second information being for assisting the operation and being based on the model output data. When the first information includes an instruction word, the input processing unit makes an inquiry to re-ask the first information as an inquiry for confirmation of the instruction word to the input source, or makes an inquiry to request re-input to something with a changed state or expression. The emotion determination unit determines the emotion of the user using the input information of the user input to the learning model, or determines the emotion of the user after outputting information regarding at least any one of the control description generated by the learning model or the operation of the device to the user. The emotion of the user determined by the emotion determination unit is input to the learning model or recorded together with the input / output data of the learning model. A control method characterized by the above.

Images