Operation support apparatus, program, and operation support method

Abstract

To provide an operation support apparatus, a program, and an operation support method for allowing a user of software to learn the operation more easily.SOLUTION: An operation support apparatus includes an answer acquisition unit. The answer acquisition unit acquires an answer to a question about the operation of software, the question being output from a language model, by inputting a prompt including the question and a structure of a GUI of the software to the trained language model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an operation assistance device, a program, and an operation assistance method. [Background technology]

[0002] There are various software programs on the market. Demand for software with a wide range of functions is also increasing. As a result, some software programs have complex menu structures. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-71690 Summary of the Invention [Problem to be solved by the invention]

[0004] Therefore, it is an important issue for software users to quickly learn how to operate the software.

[0005] However, operation assistance using conventional retrieval-augmented generation (RAG)-based approaches requires the collection of vectorized information and relies on web search results, which may result in insufficient information when using different versions of software or when using less popular software.

[0006] The problem to be solved by the embodiments of the present invention is to provide an operation assistance device, a program, and an operation assistance method that allow software users to learn how to operate the software more easily than ever before. [Means for solving the problem]

[0007] According to an embodiment, an operation assistance device includes an answer acquisition unit that inputs a prompt including a question about software operation and a GUI structure of the software into a trained language model, and acquires an answer to the question output from the language model. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of a configuration of a main part of an operation assistance device according to an embodiment. [Figure 2] 2 is a flowchart showing an example of processing by a processor in FIG. 1; [Figure 3] 2 is a flowchart showing an example of processing by a processor in FIG. 1; DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an operation assistance device according to an embodiment will be described with reference to the drawings. Note that the scale of each part in each drawing used in the following description of the embodiment may be changed as appropriate. Also, for the sake of explanation, each drawing used in the following description of the embodiment may omit configurations. Also, in each drawing and in this specification, the same reference numerals indicate similar elements.

[0010] 1 is a block diagram showing an example of the main configuration of an operation assistance device 100 according to an embodiment. Note that each component may be built-in or external.

[0011] The operation assistance device 100 is a device with an operation assistance function. The operation assistance device 100 functions as a bot that assists operations. The operation assistance function is a function that supports software operations using an LLM (large language model). The operation assistance device 100 includes, as an example, a processor 110, a ROM (read-only memory) 120, a RAM (random-access memory) 130, an auxiliary storage device 140, an input device 150, and a display device 160. A bus 170 and the like connect these components.

[0012] The processor 110 is a central part of a computer that performs various calculations and processes, such as calculations and controls, necessary for the operation of the operation assistance device 100. The processor 110 may be, for example, a central processing unit (CPU), a microprocessing unit (MPU), a system on a chip (SoC), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a field-programmable gate array (FPGA). Alternatively, the processor 110 may be a combination of these. The processor 110 may also be a combination of these with a hardware accelerator or the like. The processor 110 controls each component to realize various functions of the operation assistance device 100 based on programs such as firmware, system software, and application software stored in the ROM 120 or the auxiliary storage device 140. The processor 110 also executes the processes described below based on the programs. Note that some or all of the programs may be incorporated into the circuitry of the processor 110.

[0013] The ROM 120 and RAM 130 are the main storage devices of the computer centered around the processor 110. The ROM 120 is a non-volatile memory used exclusively for reading data. The ROM 120 stores, for example, firmware among the above programs. The ROM 120 also stores data used by the processor 110 when it performs various processes.

[0014] The RAM 130 is a memory used for reading and writing data. The RAM 130 is used as a work area for storing data that is temporarily used when the processor 110 performs various processes. The RAM 130 is typically a volatile memory.

[0015] The auxiliary storage device 140 is an auxiliary storage device of a computer centered around the processor 110. The auxiliary storage device 140 is, for example, an EEPROM (electric erasable programmable read-only memory), an HDD (hard disk drive), or a flash memory. The auxiliary storage device 140 stores, for example, system software and application software among the above programs. The auxiliary storage device 140 also stores data used by the processor 110 when performing various processes, data generated by the processes in the processor 110, various setting values, and the like.

[0016] The programs stored in the auxiliary storage device 140 include software with operational support functions (hereinafter referred to as "support software"). The support software may be system software or application software. The type of support software is not important. Examples of the types of software include various types such as operating systems, word processing software, graphics software, programming tools, web browsers, and game software.

[0017] The data stored in the auxiliary storage device 140 includes an LLM 141 and a GUI (graphical user interface) structure 142. The LLM 141 is a trained AI model. The LLM 141 is a language model capable of natural language processing. The LLM 141 may be similar to a well-known LLM. Therefore, a detailed description of the LLM 141 will be omitted. The assistance software may include the LLM 141.

[0018] The GUI structure 142 is data indicating the structure of the GUI of the support software. GUI elements targeted by the GUI structure 142 include, for example, menus, buttons, toolbars, taskbars, ribbons, and dialog boxes. GUI elements targeted by the GUI structure 142 may also include other controls. There are various types of menus, for example, menu bars, context menus, and pop-up menus. The GUI structure 142 includes, for example, UI component information, items, and operation instructions. The GUI structure 142 is preferably data in a format that can be input as a prompt to the LLM 141. Therefore, the GUI structure 142 is, for example, text data.

[0019] The GUI structure 142 may be created by a human or may be automatically created by a computer, or may be a mixture of parts that are automatically generated by a computer and parts that are created by a human.

[0020] A method by which the processor 110 automatically generates the GUI structure 142 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of processing by the processor 110 of the operation assistance device 100. The processor 110 executes the processing of Fig. 2 based on a program stored in, for example, the ROM 120 or the auxiliary storage device 140.

[0021] In step ST11 of FIG. 2, the processor 110 of the operation assistance device 100 acquires the source code of the assistance software.

[0022] In step ST12, the processor 110 analyzes the source code acquired in step ST11 to identify GUI elements from the source code.

[0023] In step ST13, processor 110 extracts the hierarchical relationships of the GUI elements identified in step ST12. To indicate the hierarchical relationships, processor 110 uses a structure that clearly shows the hierarchical relationships, such as a tree structure. By extracting the hierarchical relationships, processor 110 can organize the GUI elements into a format that is easy for LLM to analyze.

[0024] In step ST14, processor 110 extracts attributes for each GUI element identified in step ST12. The attributes of a GUI element include, for example, a label name, a caption, a shortcut key, a position, information related to appearance, and reference information.

[0025] In step ST15, the processor 110 formats the contents extracted in steps ST13 and ST14 into a data format that is easy for the LLM to analyze, and outputs the format.

[0026] As described above, processor 110 performs the processes of steps ST12 to ST15, thereby functioning as an example of an analysis section that analyzes software and generates data indicating the structure of a GUI.

[0027] In step ST16, the processor 110 stores the data output in step ST15 in the auxiliary storage device 140 or the like as the GUI structure 142. After the processing of step ST16, the processor 110 ends the processing shown in FIG.

[0028] Although the operation assistance device 100 has been described as generating the GUI structure 142, the operation assistance device 100 may use a GUI structure 142 generated by a device other than the operation assistance device 100.

[0029] The processor 110 may change the method of analyzing the source code of the assistance software depending on the software type of the assistance software. The processor 110 may automatically determine the software type of the assistance software. Alternatively, the operator may input the software type of the assistance software into the operation assistance device 100. This allows the processor 110 to generate a more accurate GUI structure 142 depending on the software type of the assistance software.

[0030] An example of the GUI structure 142 is shown in the following quotation marks. Note that the method of describing the GUI structure 142 is not limited to the following example.

[0031] " # menu Label name `Shortcut key` (reference information) - Main Menu -File - New `Ctrl+N` - Open File `Ctrl+O` ...(Omitted) # Settings dialog: dialog - Tab - Settings Group - Option 1: text - Option 2: checkbox - Option 3: number - Option 4: dropdown - Option 5: radio button ...(Omitted) "

[0032] In the above example, "#" indicates the item name. GUI structure 142 includes two items: a menu and a settings dialog. The settings dialog is a dialog box for settings. Note that all items in the above example are abbreviated by notating them as "... (omitted)".

[0033] In the above example, "Label Name `Shortcut Key` (Reference Information)" indicates a legend. In other words, the part enclosed in "``" following the label name indicates that it is a shortcut key. For example, the shortcut key for the label name "Create New" is "Ctrl+N". The label name may be a caption. The shortcut key may be a shortcut using an input device 150 other than a keyboard. Note that the GUI structure 142 may include attributes other than the label name, caption, shortcut key, and reference information.

[0034] In addition, in the above example, the hierarchical structure is indicated by the number of indentations. The number of indentations indicates how many specified whitespace characters, such as spaces or tabs, are at the beginning. In the above example, the whitespace characters used for indentation are full-width spaces. "Main Menu" and "Tab" have an indentation of 0. "File" and "Settings Group" have an indentation of 1. "New", "Open File", and "Options" have an indentation of 2. In the above example, there is a "-" as a delimiter between the whitespace character and the label name.

[0035] An item with an indent number of n indicates that it is directly below the nearest item with an indent number of (n-1) that precedes it. Note that n is an integer greater than or equal to 1. For example, "File" has an indent number of 1, so it is directly below the nearest item with an indent number of 0 that precedes "File". Such an item with an indent number of 0 is "Main Menu". "New" has an indent number of 2, so it is directly below the nearest item with an indent number of 1 that precedes "New". Such an item with an indent number of 1 is "File". Also, "Open File" has an indent number of 2, so it is directly below the nearest item with an indent number of 1 that precedes "Open File". Such an item with an indent number of 1 is "File". In other words, both "New" and "Open File" are directly below "File".

[0036] The part to the right of each label name, separated by a colon (:), indicates the UI (user interface) component used for the GUI element indicated by the label name. For example, "Settings Dialog" uses a dialog component. "Option 1" uses a text box. "Option 2" uses a check box. "Option 3" uses a control for entering numbers. "Option 4" uses a drop-down list. "Option 5" uses a radio button.

[0037] The data stored in the auxiliary storage device 140 may also include a glossary 143 and a vectorized glossary 144. The glossary 143 is data including terms used in the assistance software and explanations of those terms. The terms may include terms exclusive to the assistance software. The terms may include commonly used terms. The glossary 143 is, for example, a database. The vectorized glossary 144 is a vectorized glossary 143. The vectorized glossary 144 is, for example, a vector database. The processor 110 generates the vectorized glossary 144 by vectorizing the glossary 143. Note that the glossary 143 and the vectorized glossary 144 are each an example of data indicating software terms.

[0038] The input device 150 accepts operations by an operator of the operation assistance apparatus 100 (hereinafter simply referred to as "operator"). The input device 150 is, for example, a keyboard, a keypad, a touchpad, a mouse, or a controller. The input device 150 may also be a device for voice input.

[0039] The display device 160 displays a screen for notifying an operator or the like of various pieces of information. The display device 160 is, for example, a display such as a liquid crystal display or an organic EL (electro-luminescence) display. A touch panel can also be used as the input device 150 and the display device 160. That is, a display panel provided in a touch panel can be used as the display device 160, and a pointing device provided in the touch panel that allows touch input can be used as the input device 150.

[0040] The bus 170 includes a control bus, an address bus, a data bus, etc., and transmits signals exchanged among the various components of the operation assistance device 100 .

[0041] The operation of the operation assistance device 100 according to the embodiment will be described below with reference to FIG. 3 and other figures. Note that the content of the processing in the following operation description is an example, and various processing that can obtain similar results can be used as appropriate. FIG. 3 is a flowchart showing an example of processing by the processor 110 of the operation assistance device 100. The processor 110 executes the processing of FIG. 3 based on a program stored in, for example, the ROM 120 or the auxiliary storage device 140. The processing of the assistance software includes, for example, the processing of FIG. 3. The processor 110 starts the processing of FIG. 3 upon activation of the assistance software, for example.

[0042] In step ST21 of FIG. 3, the processor 110 of the operation assistance device 100 determines whether to start accepting input of a question. When the operator wants to ask the operation assistance device 100 a question about how to use the assistance software, the operator performs an operation using the input device 150 indicating that the operator will ask the question. The processor 110 determines to start accepting input of a question in response to the operation. If the processor 110 does not determine to start accepting input of a question, the processor 110 determines No in step ST21 and repeats the processing of step ST21. On the other hand, if the processor 110 determines to start accepting input of a question, the processor 110 determines Yes in step ST21 and proceeds to step ST22.

[0043] In step ST22, processor 110 generates an image corresponding to the input screen. Then, processor 110 instructs display device 160 to display the generated image. Upon receiving the display instruction, display device 160 displays the input screen.

[0044] The input screen is a screen that accepts input of a question. The input screen includes, as an example, an input field for inputting the question. The operator inputs the question into the input field using, for example, the input device 150. The operator inputs the question in, for example, a natural language. The question is, for example, a content related to the assistance software. The content related to the assistance software is, for example, a question related to the operation of the assistance software, such as how to operate the assistance software.

[0045] The input screen also includes a confirm button, which the operator operates to confirm the content of the question. The operator operates the confirm button after entering the content of the question in the input field.

[0046] Processor 110 may accept input of the question content by voice. In this case, processor 110 does not need to display an input screen.

[0047] As described above, processor 110 cooperates with input device 150 to accept input of question content, thereby functioning as an example of an input unit that accepts input of a question.

[0048] In step ST23, processor 110 waits for the question to be determined. That is, processor 110 waits for a predetermined operation, such as operating the enter button, to be performed after the question has been input. If the question is input by voice, processor 110 considers the question to have been determined when it is determined that the question has been spoken. If the question has been determined, processor 110 determines Yes in step ST23 and proceeds to step ST24.

[0049] In step ST24, the processor 110 inputs a prompt and a vectorization glossary 144 to the LLM 141. The prompt includes the question entered by the operator, a sentence instructing the operator to answer the question, and a GUI structure 142. The processor 110 executes the LLM 141 to which the prompt and vectorization glossary 144 have been input. The LLM 141 executed by the processor 110 generates an output for the input. That is, the LLM 141 generates and outputs an answer to the question. The processor 110 may input a glossary 143 to the LLM 141 instead of the vectorization glossary 144. The processor 110 may also input the prompt multiple times. When the prompt is input multiple times, the LLM 141 generates an output for each input. An example of a prompt input to the LLM 141 is shown in the following quotation marks. In this example, the processor 110 inputs the prompt in one go without dividing it.

[0050] " You are the assistant that guides you through the app. Understand the request from the user's instructions and answer any questions about the operation of the target app if the information is reliable. Please refer to the UI information of the app below. {UI information} # User question {User question} "

[0051] In the above example, {UI information} contains the GUI structure 142. In the above example, {user question} contains the question input by the operator.

[0052] With the above prompts, the LLM 141 acts as an assistant, analyzing the operator's question and generating an answer.

[0053] The prompt may include a character string that indicates what type of software the support software is. By including such a character string in the prompt, the LLM 141 can output a more appropriate answer depending on the type of software the support software is.

[0054] In step ST25, the processor 110 waits for an answer to be output by the LLM 141. When the LLM 141 outputs an answer, the processor 110 acquires the answer.

[0055] As described above, by performing the processing of steps ST24 and ST25, processor 110 functions as an example of an answer acquisition unit that inputs a prompt including a question regarding software operation and the GUI structure of the software into a trained language model, and thereby acquires an answer to the question output from the language model.

[0056] In step ST26, processor 110 generates an image corresponding to the output screen. Furthermore, processor 110 instructs display device 160 to display the generated image. In response to the display instruction, display device 160 displays the output screen. After processing step ST26, processor 110 returns to step ST22.

[0057] The output screen is a screen for notifying the operator of the answer content acquired in step ST24. The output screen includes the answer content.

[0058] Processor 110 may notify the operator of the answer content by a method other than displaying it on the screen, such as by reading it out loud.

[0059] As described above, processor 110 cooperates with display device 160 to notify the content of the answer, thereby functioning as an example of a notifying unit that notifies the answer.

[0060] As an example, in response to a question such as "I want to hide the grid," and then "How do I hide the grid?", the operation assistance device 100 will announce the answer "Uncheck Grid from the View menu."

[0061] The operation support device 100 of the embodiment inputs a prompt including a question about the operation of the support software and a GUI structure 142 to the LLM 141. The operation support device 100 of the embodiment then obtains an answer to the question as an output of the LLM 141. Therefore, by using the support device 100 of the embodiment, an operator can obtain an answer to a question about the operation of the support software without reading the support software manual. Furthermore, by using the support device 100 of the embodiment, answers to questions about the operation of the support software can be obtained even if they are not described in the support software manual. Therefore, part or all of the support software manual becomes unnecessary. In other words, the creator of the support software can reduce the effort required to create the manual. Furthermore, when the support software is upgraded, the creator of the support software can reduce the effort required to update the manual. Therefore, the creator of the support software can quickly release upgraded versions of the support software.

[0062] Furthermore, the operation assistance device 100 of the embodiment functions as a bot that assists with operations. The bot provides a practical means for an operator, who is a software user, to easily understand and utilize all or many of the software's functions. This is expected to improve the operator's ability to solve problems and increase product satisfaction. Furthermore, by using the bot, the operator can obtain appropriate answers to questions about software operation, making it possible to operate and learn the software more quickly and easily than before. In particular, operation and learning of assistance software with a complex GUI configuration is easier than before. Furthermore, by using the operation assistance device 100, the operator can make the most of the software's functions.

[0063] Furthermore, the operation support device 100 of the embodiment uses a prepared prompt as the GUI structure 142. Therefore, the operation support device 100 of the embodiment does not need to vectorize and collect the GUI structure. Furthermore, the operation support device 100 of the embodiment does not need to rely on a web search to collect the GUI structure, and there is less concern about a lack of information about the GUI structure than in the past.

[0064] Furthermore, the operation support device 100 of the embodiment inputs data indicating terms used in the support software into the LLM 141. This allows the operation support device 100 of the embodiment to obtain more accurate answers to questions. Furthermore, the creator of the support software only needs to prepare the GUI structure 142 and a glossary, reducing the effort required to create a manual.

[0065] Furthermore, the operation support device 100 of the embodiment uses vectorized data as data indicating terms in the support software. In this way, the operation support device 100 of the embodiment can reduce the number of prompts by not inputting terms in the support software as prompts. Furthermore, the operation support device 100 of the embodiment uses vectorized data, which makes it easier for the LLM 141 to handle the data.

[0066] Furthermore, the operation assistance device 100 of the embodiment automatically generates the GUI structure 142 of the assistance software. Therefore, the creator of the assistance software can reduce the effort of generating the GUI structure 142. Furthermore, when upgrading the assistance software, the creator of the assistance software can reduce the effort of creating the GUI structure 142. This allows the creator of the assistance software to quickly release upgraded versions of the assistance software.

[0067] The operation support device 100 of the embodiment also receives a question input by the operator. The operation support device 100 of the embodiment also inputs the question to the LLM 141 and notifies the operator of the answer obtained. This allows the operator to ask a question about the operation of the support software and to receive the answer.

[0068] The above embodiment can be modified as follows.

[0069] In the above embodiment, the operation assistance device 100 uses the glossary 143 or the vectorized glossary 144 as external data available to the LLM 141. However, the operation assistance device 100 may use an LLM that has been trained with the glossary 143 instead of the LLM 141. In this case, the processor 110 of the operation assistance device 100, for example, causes the LLM 141 to additionally train the glossary 143.

[0070] In the above embodiment, the operation support device 100 uses the glossary 143 or the vectorized glossary 144 as external data available to the LLM 141. However, the processor 110 of the operation support device 100 may input the glossary 143 as a prompt to the LLM 141. For this purpose, the processor 110 converts the glossary 143 into data that can be input as a prompt and stores it in the auxiliary storage device 140.

[0071] The operation support device 100 does not need to use the glossary 143 and the vectorization glossary 144 .

[0072] The processor 110 of the operation assistance device 100 may generate the GUI structure 142 by analyzing compiled assistance software instead of the source code of the assistance software. In this case, the processor 110 may reverse engineer the assistance software.

[0073] The support software may be built-in software.

[0074] The operation assistance device according to the embodiment may be composed of multiple devices. The operation assistance device according to the embodiment may be realized using cloud computing. As an example, the operation assistance device includes a terminal device and a server device. The server device is a server in cloud computing. The server device executes the LLM 141. The terminal device transmits a prompt and a vectorized glossary 144 to the server device via a communication network such as the Internet or a local area network (LAN). The processor of the server device executes the processes of steps ST24 and ST25 using the prompt and the vectorized glossary 144. The server device then transmits the acquired answer to the terminal device via the communication network. The terminal device executes the process of step ST26 using the answer.

[0075] The processor 110 may implement some or all of the processes implemented by the programs in the above embodiments by using a hardware circuit configuration.

[0076] A program for implementing the processes of the embodiments may be transferred in a state where it is stored in a non-transitory computer-readable storage medium within the device. However, the device may also be transferred without the program stored therein. The program may then be transferred separately and written to the device. In this case, the program may be transferred by, for example, recording it on a removable non-transitory computer-readable storage medium or by downloading it via a network such as the Internet or a LAN.

[0077] Although the embodiments of the present invention have been described above, they are merely examples and are not intended to limit the scope of the present invention. The embodiments of the present invention can be implemented in various forms without departing from the spirit of the present invention. [Explanation of symbols]

[0078] 100 Operation support equipment 110 processors 120 ROM 130 RAM 140 Auxiliary storage 141 LLM 142 GUI structure 143 Glossary 144 Vectorization Glossary 150 input devices 160 Display Devices 170 Bus

Claims

1. An operation assistance device comprising an answer acquisition unit that inputs a prompt including a question regarding software operation and the GUI structure of the software into a trained language model, and acquires an answer to the question output from the language model.

2. The operation assistance device according to claim 1 , wherein the response acquisition unit inputs the prompt and data indicating a term of the software into the language model.

3. The operation assistance device according to claim 2 , wherein the data representing the software term is vectorized data.

4. The operation assistance device according to claim 1 , further comprising an analysis unit that analyzes the software and generates data indicating the structure of the GUI.

5. an input unit for receiving the question; The operation assistance device according to claim 1 , further comprising: a notification unit that notifies the user of the answer.

6. The processor of the operation support device is A program that functions as an answer acquisition unit that inputs a prompt containing a question regarding software operation and the GUI structure of the software into a trained language model, and acquires an answer to the question output from the language model.

7. An operation assistance method that inputs a prompt including a question about software operation and the GUI structure of the software into a trained language model, and obtains an answer to the question output from the language model.

Images