Operation support devices, operation support methods and procedures

Abstract

Model (31) is configured to be trained using information via machine learning to output intent information indicating the intent of the question based on the question and screen recognition information. Inference unit (21) obtains intent information from the model based on the question and screen recognition information used to identify the description screen displayed when the input unit (11) has obtained the question. Inference unit (21) obtains a response corresponding to the obtained intent information, a control command corresponding to the intent information and screen recognition information, and destination screen recognition information used to identify the description screen corresponding to the destination change of the intent information and screen recognition information. The application outputs the response obtained by inference unit (21), causing control command execution unit (41) to execute the control command obtained by inference unit (21) and display the description screen corresponding to the destination screen recognition information obtained by inference unit (21) on display unit (43).

Description

Technical Field

[0001] This invention relates to an operation support device, operation support method, and procedure. Background Technology

[0002] Every industry needs a simple and appropriate way to respond to questions from users about how to operate the device.

[0003] Patent Document 1 discloses a device operation guidance device. In this device operation guidance device, a query from a user regarding a device operation method is input through a query input mechanism. This device operation input mechanism is used by the user to operate the device. A receiving mechanism receives operation guidance information read from a database. Then, the operation guidance mechanism presents operation guidance based on the operation guidance information, allowing the user to operate the device operation input mechanism to perform the desired operation.

[0004] Citation List

[0005] Patent documents

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2005-107583 Summary of the Invention

[0007] Technical issues

[0008] However, in every industry, products are becoming increasingly smaller and thinner, and the processes for manufacturing or testing these products are becoming increasingly complex. Consequently, the equipment used in each of these processes is also becoming more complex. In this context, a problem arises where the use of this equipment becomes increasingly difficult and prone to operator error. This problem affects companies in every industry, ultimately leading to increased losses.

[0009] Given the aforementioned complexity, a problem arises: when questions are raised by users operating the target device, the intent of the question may vary depending on the circumstances. Furthermore, simply returning a response to a user's question does not guarantee that the user has reliably understood all the correct operating procedures. In other words, the operation support device must provide information consistent with the intent of the question. Figure 1 The intent of the question depends on the circumstances. Furthermore, in addition to simply returning a response to the question, it is expected that the operational support device will be able to properly control the target device when necessary.

[0010] The present invention was made in consideration of the above circumstances. The present invention is to provide an operation support device, operation support method and program that can not only respond to the surface layer of the problem content, but also, depending on the situation, perform at least a portion of the control of the target device on behalf of the user in some cases.

[0011] Technical solution

[0012] [1] To address the above problems, according to one aspect of the present invention, an operation support device is provided, comprising an input unit, a display unit, a control command execution unit, an inference unit, an application running unit, a response storage unit, and a control command generation unit. The input unit is used to acquire a question. The display unit displays a description screen for a target device. The control command execution unit executes control commands for controlling the target device. The inference unit acquires intent information from a model based on a model, a question, and screen recognition information for identifying the description screen displayed when the input unit has acquired the question. The inference unit obtains a response corresponding to the acquired intent information, control commands corresponding to the intent information and screen recognition information, and change-destination screen recognition information for identifying a description screen for a change-destination, the change-destination corresponding to the intent information and screen recognition information. The model is configured to be trained using information via machine learning to output intent information indicating the intent of the question based on the question and screen recognition information. The application running unit executes an application to output the response obtained by the inference unit, causes the control command execution unit to execute the control commands obtained by the inference unit, and displays a description screen corresponding to the change-destination screen recognition information obtained by the inference unit on the display unit. The response storage unit stores the correspondence between intent information and responses. The control command generation unit stores information about control commands corresponding to intent information and screen recognition information, and changes the destination screen recognition information. It responds to the inference unit's query based on the intent information and screen recognition information, using the control commands and changes the destination screen recognition information corresponding to the query. Furthermore, the inference unit obtains a response corresponding to the intent information by referring to the response storage unit based on the acquired intent information. Additionally, the inference unit obtains the control commands and changes the destination screen recognition information by querying the control command generation unit based on the acquired intent information and screen recognition information.

[0013] [2] In addition, the operation support device according to one aspect of the present invention further includes a machine learning processing unit, which provides training data including questions, screen recognition information and intent information to train a model through machine learning.

[0014] [3] Furthermore, in the operation support device according to one aspect of the present invention, the display unit displays the response output by the application operation unit.

[0015] [4] Furthermore, in an operation support device according to another aspect of the invention, the display unit has an area for displaying an image captured by a camera included in the target device.

[0016] [5] According to one aspect of the present invention, an operation support method is provided, the operation support method comprising: obtaining a question by an input unit; displaying a description screen for a target device by a display unit; executing a control command execution unit for controlling the target device by a control command execution unit; providing a model trained by machine learning using information to output intent information indicating the intent of the question based on the question and screen recognition information; obtaining intent information from the model by an inference unit based on the question and screen recognition information for identifying the description screen displayed when the input unit has obtained the question; obtaining a response corresponding to the obtained intent information, control information corresponding to the intent information and screen recognition information, and change destination screen recognition information for identifying the description screen for a change destination, the change destination corresponding to the intent information and screen recognition information; executing an application by an application execution unit for outputting the response obtained by the inference unit; causing the control command execution unit to execute the control command obtained by the inference unit; and displaying a description screen corresponding to the change destination screen recognition information obtained by the inference unit on the display unit.

[0017] [6] According to one aspect of the invention, a program is provided that enables a computer to be used as an operation support device according to any one of [1] to [4] above.

[0018] Beneficial effects

[0019] According to the present invention, the operation support device can respond to problems and control external hardware based on the intent of the situation. Attached Figure Description

[0020] Figure 1 This is a block diagram illustrating a schematic functional configuration of an operation support device according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram (1 / 2) illustrating an example of the information (signal) exchange process in the operation support device of an embodiment.

[0022] Figure 3 This is a schematic diagram (2 / 2) illustrating an example of the flow of information (signal) exchange in the operation support device of an embodiment.

[0023] Figure 4 This is a schematic diagram illustrating the relationship between data processed by the operation support device according to an embodiment, showing the relationship between problem content and screen ID and interface.

[0024] Figure 5 This is a schematic diagram illustrating the relationship between data processed by the operation support device according to an embodiment, showing the relationship between intent and response content.

[0025] Figure 6 This is a schematic diagram illustrating the relationships between data processed by the operation support device according to an embodiment, showing the relationships between intent, screen ID, control command, and change destination screen ID.

[0026] Figure 7 It is a table listing the types of control commands to be used by the operation support device according to the embodiment.

[0027] Figure 8 This is a schematic diagram illustrating an example configuration of a screen displayed by a display unit of an operation support device according to an embodiment.

[0028] Figure 9 This is a schematic diagram illustrating a configuration example of a management table used to manage information about a description screen displayed by an operation support device according to an embodiment.

[0029] Figure 10 This is a schematic diagram illustrating an example of training data according to an embodiment, which is used to train a learning model through machine learning.

[0030] Figure 11 This is a schematic diagram illustrating the processing of an operation support device based on a specific problem example according to an embodiment. Detailed Implementation

[0031] Next, embodiments of the present invention will be described with reference to the accompanying drawings. In this embodiment, the operation support device estimates intent information based on the question and performs processing based on the intent information, which indicates the intent of the question input by the user. Hereinafter, the intent information is also referred to as "intent". The intent is estimated based on the text of the question (surface layer) and the current situation.

[0032] In this embodiment, a "problem" is a sentence or the like that input by the user using the input unit 11. However, a "problem" here is not limited to a sentence in the form of an interrogative sentence. For example, in this embodiment, any language expression related to device operation (such as "I don't know ○○", "○○ is not working", "○○ is not appearing", "I want to start ○○", "I want to stop ○○", "I want to change the settings of ○○") is considered a "problem".

[0033] The "screen ID" described in this embodiment is information used to uniquely identify the type of description screen displayed on the operation support device. The "screen ID" is also referred to as "screen identification information." Furthermore, the description screen for the next destination to be changed to is called a "destination change screen." The "destination change screen identification (ID)" is information used to uniquely identify the type of destination change screen. The "destination change screen ID" is also referred to as "destination change screen identification information." Additionally, the description screen is a screen used to describe the device (target device 3) to be operated on. The description screen may include text, images (still images), and video (moving images). Furthermore, when the operation support device displays the description screen, the description may also be output as a predetermined voice message. Furthermore, a control command is a command that the operation support device can execute to control external devices, etc. The "control command ID" is information used to uniquely identify the control command. The "control command ID" is also referred to as "control command identification information."

[0034] Figure 1 This is a block diagram illustrating a schematic functional configuration of the operation support device according to this embodiment. As shown, the operation support device 1 may include an input unit 11, an application running unit 12, an inference unit 21, a response storage unit 22, a control command generation unit 23, a learning model 31, a machine learning processing unit 32, a control command execution unit 41, a hardware control unit 42, and a display unit 43. Each of the functional units may be implemented by, for example, a computer and a program. Furthermore, each functional unit has a storage mechanism if necessary. The storage mechanism is, for example, memory allocated by variables on the program or by the execution of the program. Moreover, for the storage mechanism, a non-volatile storage mechanism such as a magnetic hard disk device or a solid-state drive (SSD) may be used as needed. In addition, at least some functions of each functional unit may be implemented by dedicated electronic circuitry instead of a program.

[0035] Specifically, an operation support device 1 is constructed using, for example, a personal computer (PC) or various portable terminal devices (e.g., tablet-type terminal devices, smartphones, etc.). The user who wants to run the target device 3 operates the operation support device 1.

[0036] The target device 3 is a device to be operated, etc. The target device 3 is, for example, a device used at a manufacturing site or service implementation site. The target device 3 is, for example, a manufacturing device, a processing device, a testing / inspection device, or a conveying device, etc. (however, the target device 3 is not limited to these). The target device 3 may include a camera. For example, when the target device 3 is an inspection device for inspecting semiconductor products, the camera captures images of the semiconductor products to be inspected. The image captured by the camera is initially intended to be processed in the target device 3, but the image signal can be sent to the operation support device 1. In this case, the image captured by the camera can be displayed on the display screen of the operation support device 1. The operation support device 1 and the target device 3 are connected via a wired or wireless communication mechanism. The operation support device 1 and the target device 3 can be interconnected via a communication line such as the Internet.

[0037] The function of each unit that constitutes the operation support device 1 will be described later.

[0038] The input unit 11 has the function of acquiring input from the user. For example, the input unit 11 acquires character input signals from a keyboard. Alternatively, the input unit 11 acquires character input signals via a touchpad. Alternatively, the input unit 11 may have a voice input function, which acquires the user's voice through a microphone or the like and performs voice recognition processing.

[0039] The application execution unit 12 is a mechanism for running application programs. Specifically, the application execution unit 12 includes a processing unit (CPU, etc.) for executing the application program and a memory for storing data, etc. The application execution unit 12 executes a dedicated application program for controlling the target device 3.

[0040] By executing the application, the application execution unit 12 obtains the text of the question from the input unit 11. Additionally, the application execution unit 12 transmits the text of the question to the inference unit 21. Furthermore, the application execution unit 12 holds a screen ID used to identify the currently displayed description screen. When the question is input, the application execution unit 12 transmits the screen ID to the inference unit 21. The application execution unit 12 receives and outputs the response obtained by the inference unit 21 based on the question and the screen ID. Furthermore, the application execution unit 12 causes the control command execution unit 41 to execute the control command obtained by the inference unit 21. Additionally, the application execution unit 12 causes the display unit 43 to display a description screen corresponding to the change destination screen ID obtained by the inference unit 21. In this way, in response to an input question, the application can output a response, execute control commands, or display a specific description screen (a description of the target device 3).

[0041] The inference unit 21 performs inference processing based on the input question. Specifically, the inference unit 21 receives the question text and screen ID (screen identification information) from the application execution unit 12. The screen ID is identification information used to identify the description screen displayed when the input unit 11 has obtained the question. The inference unit 21 sends the question and screen ID to the learning model 31 and receives the intent (intent information) as its response from the learning model 31. In addition, the inference unit 21 obtains the response corresponding to the intent based on the intent reference response storage unit 22. Furthermore, the inference unit 21 queries the control command generation unit 23 based on the intent and screen ID to obtain the corresponding destination screen ID (destination screen identification information). The destination screen ID is identification information used to identify the description screen of the next destination to be changed. The inference unit 21 returns the response, control command, and destination screen ID obtained as described above to the application execution unit 12.

[0042] The response storage unit 22 stores responses associated with the intent (intent information). In other words, the response associated with the intent can be read by referring to the response storage unit 22. The response is, for example, presented as text data.

[0043] The control command generation unit 23 stores information associated with a combination of intent (intent information) and screen ID (screen identification information) for control commands and a change destination screen ID (change destination screen identification information). Based on the stored information, the control command generation unit 23 responds to queries from the inference unit 21 with control commands and change destination screen IDs corresponding to the intent and screen ID.

[0044] Learning model 31 has information for outputting intent (intent information), which indicates the intent of the question based on the question and screen ID (screen recognition information). In learning model 31, machine learning is performed using the relationship between the question and the screen ID as input information and the intent as output information. Learning model 31 is constructed using, for example, a neural network.

[0045] The machine learning processing unit 32 has the function of providing training data to the learning model 31, enabling the learning model 31 to perform machine learning processing. The training data is a set of data including screen ID (screen recognition information), question, and intent (intent information). Furthermore, when the machine learning of the learning model 31 is complete, the operation support device 1 can be configured to omit the machine learning processing unit 32.

[0046] The control command execution unit 41 executes commands for controlling external devices, etc. For example, the control command execution unit 41 executes commands for controlling the target device 3. Control of the external device is performed, for example, by outputting signals from an input / output port. Additionally, the control command execution unit 41 can execute commands for displaying characters or images on the display unit 43. Specifically, the control command execution unit 41 can display question text, response text, or a description screen, etc., on the display unit 43.

[0047] The hardware control unit 42 controls external hardware (target device 3, etc.) according to control commands executed by the control command execution unit 41. Specifically, the hardware control unit 42 outputs control signals from I / O ports, etc.

[0048] Display unit 43 displays images or text, etc. Specifically, display unit 43 displays a description screen for describing the target device 3, displays text of questions entered by the user, displays response text output by the operation support device 1, or displays images based on image signals from an external source. External image signals are, for example, signals of images (videos) captured by a camera included in the target device 3. An example configuration of the screen displayed by display unit 43 will be described later with reference to the accompanying drawings.

[0049] Figure 2 and Figure 3 This is a schematic diagram illustrating an example of the information (signal) exchange process in the operation support device 1. Below, an example of the operation sequence of the operation support device 1 will be described based on the schematic diagram.

[0050] First of all, Figure 2 In step S1, the input unit 11 retrieves a question from the user and passes it to the application execution unit 12. The question is retrieved using a mechanism such as key presses on a keyboard or voice recognition. The application execution unit 12 retrieves the question in text data form. Even at the time the question is retrieved, the application execution unit 12 displays a predetermined description screen on the display unit 43. The application execution unit 12 holds the screen ID of the currently displayed description screen.

[0051] Next, in step S2, the application execution unit 12 transmits the screen ID of the current description screen and the text of the question obtained in step S1 to the inference unit 21. Thus, the inference unit 21 begins the inference process.

[0052] Next, in step S3, the inference unit 21 sends the screen ID and question obtained in step S2 to the learning model 31. The learning model 31 has completed machine learning. Based on the results of machine learning, the learning model 31 obtains an intent suitable for the screen ID and question.

[0053] Next, in step S4, the learning model 31 sends the intent obtained in step S3 to the inference unit 21. The inference unit 21 receives the intent.

[0054] Next, in step S5, the inference unit 21 performs an access based on the intent reference response storage unit 22 received in step S4. The response storage unit 22 specifies the response corresponding to the intent based on the stored information.

[0055] Next, in step S6, the response storage unit 22 sends the response corresponding to the intent to the inference unit 21. In other words, the inference unit 21 reads the information of the response corresponding to the intent from the response storage unit 22. Furthermore, the inference unit 21 obtains, for example, the text data of the response.

[0056] Next, in step S7, the inference unit 21 sends the screen ID received in step S2 and the intent received in step S4 to the control command generation unit 23. The control command generation unit 23 receives information about the above items. The control command generation unit 23 pre-stores information about control commands and change destination screen IDs, where the control command corresponds to the input combination of the screen ID and the intent. In other words, the control command generation unit 23 generates information about control commands and change destination screen IDs corresponding to the received combination of screen ID and intent.

[0057] Next, in step S8, the control command generation unit 23 sends the control command generated in step S7 and the change destination screen ID to the inference unit 21. In other words, the inference unit 21 obtains the control command and the change destination screen ID based on the screen ID and the intent.

[0058] Next, in step S9, the inference unit 21 transmits the response obtained in step S6, along with the control command and the change destination screen ID obtained in step S8, to the application running on the application execution unit 12. The application execution unit 12 receives the response, control command, and change destination screen ID.

[0059] Next, refer to Figure 3 In step S10, the application running in the application execution unit 12 sends the text of the response received in step S9 to the control command execution unit 41. Additionally, the application sends the control command received in step S9 to the control command execution unit 41. Furthermore, the application sends the screen identified by the changed destination screen ID received in step S9 to the control command execution unit 41.

[0060] Next, in step S11, the control command execution unit 41 sends the control command received in step S10 to the hardware control unit 42. Thus, the hardware control unit 42 can execute the control command. The control command is, for example, a command used to control the target device 3. In other words, the control command execution unit 41 can control the behavior of the target device 3, for example, by executing the control command.

[0061] Next, in step S12, the control command execution unit 41 transmits the text of the response received in step S10 or the screen (signal indicating the screen) identified by the destination screen ID to the display unit 43. Thus, the display unit 43 can display the destination screen (the screen describing the target device 3). Additionally, the display unit 43 can display the response output by the inference unit 21.

[0062] Figure 4 , Figure 5 and Figure 6 The relationship between the data processed by the operation support device 1 is shown.

[0063] Figure 4 This is a schematic diagram illustrating the relationship between the question content, screen ID, and intent during processing in the operation support device 1. Here, the question content is the text of a question obtained from the user by the input unit 11. The screen ID is identification information used to identify the screen displayed when the input unit 11 queries a question from the user. In other words, the screen ID is information indicating that a question has been asked. The intent is intent information indicating the intention of the question content in this situation. (See reference...) Figure 2 As described, learning model 31 estimates and outputs intent based on the input question content and screen ID. Since learning model 31 has undergone machine learning, it possesses intent information corresponding to the input question content and screen ID. Learning model 31 does not necessarily possess definitive information about the relationship between the question content, screen ID, and intent. However, learning model 31 at least possesses information about the possible intents corresponding to the question content and screen ID. Learning model 31 can estimate the intent based on this information.

[0064] Figure 5This is a schematic diagram illustrating the relationship between intent and response content during processing in the operation support device 1. Here, as described above, the intent is information output from the learning model 31. The response content is the text of the response to be output. As already described, the response storage unit 22 stores the intent associated with the response content. In other words, by searching the response storage unit 22 while using the intent as a key, the response content corresponding to the intent can be obtained. In other words, the response storage unit 22 stores data corresponding to the conversion table from intent to response content. The inference unit 21 can perform the conversion from intent to response content by referring to the response storage unit 22.

[0065] Figure 6 This is a schematic diagram illustrating the relationship between the intent and screen ID, and the control command and destination screen ID during processing in the operation support device 1. As described above, the control command generation unit 23 stores information about the correspondence between the intent and screen ID and the control command and destination screen ID. In other words, the control command generation unit 23 stores data corresponding to a conversion table used to convert the combination of intent and screen ID into a control command and destination screen ID. When the intent and screen ID are received from the inference unit 21, the control command generation unit 23 can determine the control command and destination screen ID to be output by referring to the data corresponding to the conversion table.

[0066] As referenced above Figure 4 , Figure 5 and Figure 6 The described operation support device 1 has information and processing logic for obtaining response content (text), control commands, and changing destination screen IDs based on the question content (text) and screen ID.

[0067] Figure 7This is a table listing the types of control commands to be used by the operation support device 1. As shown, the control target of the control commands includes the target device 3. The operation support device 1 can use several control commands for each target device. Here, for convenience, control command IDs are provided to identify the control commands. For example, the control command ID for the target device 3, which is the control target, begins with "CA-" and includes a serial number. For example, the control command ID CA-01 is a command for "powering on". The control command ID CA-02 is a command for "powering off". The control command ID CA-03 is a command for "initializing the motor system". The control command ID CA-04 is a command for "starting transmission". The control command ID CA-05 is a command for powering on the camera of the target device 3. The control command ID CA-06 is a command for powering off the camera of the target device 3. Other control commands may exist.

[0068] Figure 7 The control command symbol string shown is code that is actually executed by the control command execution unit 41. The control command symbol string can be represented as, for example, a string, a hexadecimal string, or a bit string.

[0069] Operation support device 1 can perform Figure 7 The control commands shown or other control commands. Specifically, as described above, the control command generation unit 23 appropriately selects and generates control commands according to specific circumstances. The control command execution unit 41 can execute the generated control commands. Additionally, although in Figure 7 While a single command is enumerated, the control command generation unit 23 can generate control commands for executing multiple commands sequentially or for executing multiple commands in parallel. Additionally, the control command execution unit 41 also executes such commands.

[0070] Figure 8 This is a schematic diagram illustrating an example configuration of a screen displayed by the display unit 43 of the operation support device 1. The screen is displayed using, for example, a color liquid crystal display. As shown, screen 1001 includes a description screen display area 1002, a captured image display area 1003, and a question and answer display area 1004. The arrangement of the description screen display area 1002, the captured image display area 1003, and the question and answer display area 1004 shown is exemplary, and other arrangements are possible. Each of the description screen display area 1002, the captured image display area 1003, and the question and answer display area 1004 may be scrollable or non-scrollable.

[0071] The description screen display area 1002 is a screen display area used to display a description of the operation method of the target device 3, etc. In the example shown, the description screen display area 1002 displays the screen ID (G00101) of the description screen and the screen title "Operation Method".

[0072] The captured image display area 1003 is an area used to display images (still images or moving images) captured by the camera of the target device 3. In other words, the captured image display area 1003 can display images output by the target device 3.

[0073] The question and answer display area 1004 is used to display the text of the question obtained by the operation support device 1 and the text of the answer output by the operation support device 1 in response to the question. In the example shown, the question is displayed in a language balloon box on the right side of the area. Additionally, the answer is displayed in a language balloon box in the same area. The questions and answers are displayed in chronological order. Furthermore, the method of displaying questions and answers is exemplary. Other display methods can be used to display questions or answers.

[0074] Figure 9 This is a schematic diagram illustrating a management table describing screens, managed by an application operated by application execution unit 12. As shown, the management table contains relational data including screen IDs, screen titles, and each item in the screen definition. The application can access the management table stored in or outside the operation support device 1. The screen ID is identification information used to uniquely identify the type describing the screen. The screen title is a string representing the title of the screen. (Example...) Figure 8 As shown, for example, a screen ID or title can be displayed in the screen display area 1002. A screen definition is information defining the configuration of the screen. Applications can configure the screen by referring to the screen definition and send image signals used to render the screen to the display unit 43. Information used for the screen definition can include images, video (moving images), text, etc. In the example shown, the management table can hold screen information with titles such as "Operation Method," "Error Message," "Initialization Method," and "Charging Warning Display," each corresponding to a unique screen ID.

[0075] Figure 10 This is a schematic diagram illustrating an example of training data used to train learning model 31 via machine learning. As shown, the training data is a set of screen IDs (screen recognition information), questions, and intents (intent information). The first row of the example data shown is as follows: screen ID is "G02231", question is "No image can be seen from the camera", and intent is "No image can be seen from the camera". The machine learning processing unit 32 provides the learning model 31 with... Figure 10The large amount of data shown in the diagram serves as training data, enabling the learning model 31 to be trained using machine learning. Through machine learning, when a question is given, the learning model 31 can estimate the screen ID and the corresponding intent.

[0076] Figure 11 This is a schematic diagram illustrating the operational state of the operation support device 1 based on a specific problem example. In the example shown, the input unit 11 transmits the problem "The camera screen is black" to the application execution unit 12. The application execution unit 12 transmits the text of the problem and the screen ID (G02232) of the currently displayed description screen to the inference unit 21. The inference unit 21 transmits the text of the problem and the screen ID to the learning model 31. The learning model 31 estimates the intent based on the text of the problem and the screen ID received from the inference unit 21 through the model's actions. In this example, the estimated intent is "The camera screen is not visible." Furthermore, the intent can be in the form of text as described above, or it can be in the form of any symbol. The learning model 31 returns the intent to the inference unit 21. Based on this intent, the inference unit 21 obtains the response "You can check the camera's power on the screen" from the response storage unit 22. Additionally, based on the intent and the screen ID (G02232), the inference unit 21 obtains the destination screen ID (G02501) and the control command for performing the "Power on the camera" control from the control command generation unit 23. Therefore, the application running in the application running unit 12 can perform the following processing: output the response "You can check the camera's power on the screen", display the description screen identified by the change destination screen ID (G02501), and execute the control command corresponding to "Power on the camera".

[0077] Figure 11 The processing shown is an example; the operation support device 1 can output various responses based on the information stored in the response storage unit 22. Additionally, the operation support device 1 can execute various control commands or display various description screens based on the information stored in the control command generation unit 23.

[0078] For example, in response to the intention "I want to initialize the motor system", the operation support device 1 can display a description screen about initializing the motor system to the target device 3 or execute control commands for initializing the motor system to the target device 3. Additionally, the operation support device 1 can perform various operations such as starting or stopping the target device 3 or changing the settings of the target device 3.

[0079] Furthermore, at least some functions of the operation support device, etc., in the above embodiments can be implemented by a computer. In this case, this implementation can be achieved by recording a program for implementing the functions on a computer-readable recording medium and reading the program recorded on the recording medium into a computer system and executing the program. Furthermore, the term "computer system" as used herein includes an operating system and hardware such as peripheral devices. Additionally, "computer-readable recording medium" refers to portable media such as floppy disks, magneto-optical disks, ROMs, CD-ROMs, DVD-ROMs, and USB storage devices, as well as storage devices such as hard disks embedded in a computer system. Furthermore, "computer-readable recording medium" can also include media that temporarily and dynamically store programs (e.g., communication lines when sending programs via a network such as the Internet or communication lines such as telephone lines), and media that store programs for a specific period of time (e.g., volatile memory within a computer system acting as a server or client in the above cases). Furthermore, the program can be designed to implement some of the functions described above, or it can be designed to implement the functions in conjunction with programs already recorded in the computer system.

[0080] Although the embodiments have been described above, the present invention can also be implemented in the following modified examples.

[0081] [Modification Example]

[0082] In the above embodiments, the case of displaying response text on the screen has been described; alternatively, recorded messages or synthesized speech can be used to output response speech.

[0083] Embodiments of the present invention have been described above with reference to the accompanying drawings, but specific configurations are not limited to the embodiments and also include designs within the scope of the present invention.

[0084] Industrial Applications

[0085] This invention can be used for purposes such as supporting the operation of devices in all industries. However, the scope of use of this invention is not limited to those examples illustrated herein.

[0086] Symbol Explanation

[0087] 1. Operation support device

[0088] 3. Target device

[0089] 11 Input Unit

[0090] 12 Application Runtime Unit

[0091] 21 Reasoning Units

[0092] 22 Response storage unit

[0093] 23 Control Command Generation Unit

[0094] 31 Learning Model

[0095] 32 Machine Learning Processing Units

[0096] 41 Control Command Execution Unit

[0097] 42 Hardware Control Unit

[0098] 43 Display Units

[0099] 1001 screen

[0100] 1002 Describes the screen display area

[0101] 1003 Capture image display area

[0102] 1004 Question and Response Display Area

Claims

1. An operation support device, comprising: The input unit is used to obtain the question; The display unit displays a description screen for the target device; A control command execution unit, which executes control commands for controlling the target device; The model is configured to be trained using information via machine learning to output intent information indicating the intent of the question based on the question and screen recognition information; The reasoning unit obtains the intent information from the model based on the question and screen recognition information, and obtains the response corresponding to the obtained intent information, the control command corresponding to the intent information and the screen recognition information, and the destination change screen recognition information. The screen recognition information is used to identify the description screen displayed when the input unit has obtained the question, and the destination change screen recognition information is used to identify the description screen of the destination change corresponding to the intent information and the screen recognition information. An application execution unit executes an application program that outputs the response obtained by the inference unit, causes the control command execution unit to execute the control command obtained by the inference unit, and displays a description screen on the display unit that corresponds to the change destination screen identification information obtained by the inference unit. A response storage unit stores the correspondence between the intent information and the response; as well as A control command generation unit stores information about the control commands corresponding to the intent information and the screen recognition information, and the change destination screen recognition information, and responds to a query from the inference unit based on the intent information and the screen recognition information with the control commands corresponding to the query and the change destination screen recognition information. The reasoning unit obtains the response corresponding to the intent information by referring to the response storage unit based on the acquired intent information, and The reasoning unit obtains the control command and the destination change screen recognition information by querying the control command generation unit based on the acquired intent information and the screen recognition information.

2. The operation support device according to claim 1 further includes a machine learning processing unit, the machine learning processing unit providing training data including the question, the screen recognition information and the intent information, to train the model through machine learning.

3. The operation support device according to claim 1 or 2, wherein, The display unit displays the response output by the application running unit.

4. The operation support device according to any one of claims 1 to 3, wherein, The display unit has an area for displaying images captured by a camera included in the target device.

5. An operation support method, comprising: The problem is obtained from the input unit; The display unit displays a description screen for the target device; The control command execution unit executes the control commands for controlling the target device; A model is provided, which is configured to be trained using information via machine learning to output intent information indicating the intent of the question based on the question and screen recognition information; The inference unit obtains the intent information from the model based on the question and screen recognition information, and obtains the response corresponding to the obtained intent information, the control command corresponding to the intent information and the screen recognition information, and the destination change screen recognition information. The screen recognition information is used to identify the description screen displayed when the input unit has obtained the question, and the destination change screen recognition information is used to identify the description screen of the destination change corresponding to the intent information and the screen recognition information. The application execution unit executes the application, which outputs the response obtained by the inference unit, causing the control command execution unit to execute the control command obtained by the inference unit, and displays a description screen on the display unit that corresponds to the change destination screen identification information obtained by the inference unit; The correspondence between the intent information and the response is stored in the response storage unit; as well as The control command generation unit stores information about the control commands and the changed destination screen recognition information corresponding to the intent information and the screen recognition information, and responds to the query from the inference unit based on the intent information and the screen recognition information with the control commands and the changed destination screen recognition information corresponding to the query. The reasoning unit obtains the response corresponding to the intent information by referring to the response storage unit based on the acquired intent information, and The reasoning unit obtains the control command and the destination change screen recognition information by querying the control command generation unit based on the acquired intent information and the screen recognition information.

6. A program that enables a computer to be used as an operating support device according to any one of claims 1 to 4.

Images