Computer program, information processing device, and information processing method

Abstract

To provide a novel mechanism for assisting operation of a device, for example, in a dialogue format using natural language.SOLUTION: The computer program causes a computer to function so as to accept a request in a natural language from a user, the request specifying a device and relating to an operation of the device. The computer program also causes the computer to function so as to input a prompt and device operation information based on the request to a language model server. The computer program further causes the computer to function so as to provide a user, via a display unit, with an answer generated by the language model server on the basis of the prompt and the device operation information.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a computer program, an information processing device, and an information processing method that utilize a natural language model. [Background technology]

[0002] One method for operating a device from a PC or the like via a network is to operate the device from a remote UI. Patent Document 1 proposes a mechanism for improving the operability of software used in device remote management technology. Also known is a service that provides information to users in a natural language dialogue format using a generation AI such as ChatGPT (registered trademark). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5738388 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology has the following problems: While the above-mentioned conventional technology allows device settings to be made from a web browser, depending on the device, there are many setting items, requiring cumbersome user operations. Also, users need to refer to manuals, etc. to find out which items should be set, which takes time for the user to look up the information.

[0005] The present invention has been made in consideration of at least one of the above problems, and provides a novel mechanism for supporting device operation in natural language. [Means for solving the problem]

[0006] The present invention is characterized in that, for example, a computer functions as: a receiving means for receiving a request from a user in natural language, the request relating to the operation of a specified device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; and a providing means for providing a response generated by the language model server based on the prompt and the operation information of the device to the user via a display unit. [Effects of the Invention]

[0007] According to the present invention, it is possible to support device operation in natural language. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing an example of a system for operating an MFP 100 in an interactive format using natural language according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the hardware configuration of an MFP 100 according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the hardware configuration of a PC 400 according to an embodiment. [Figure 4] FIG. 2 is a diagram showing an example of the hardware configuration of a language model server 300 according to an embodiment. [Figure 5] FIG. 4 is a sequence diagram for changing the settings of the MFP 100 according to an embodiment. [Figure 6] 10 is a flowchart showing a processing procedure of a PC 400 according to an embodiment. [Figure 7] 1 is a flowchart showing a processing procedure of the MFP 100 according to an embodiment. [Figure 8] FIG. 4 is a diagram showing an example of a screen displayed on a display unit 410 by an agent program 413 of a PC 400 according to an embodiment. [Figure 9] FIG. 4 is a diagram showing an example of a portion of data in a data repository 414 that is expanded in a RAM 404 of a PC 400 according to an embodiment. [Figure 10]FIG. 4 is a diagram showing an example of a prompt generated by an agent program 413 of a PC 400 and transmitted to a language model server 300 according to an embodiment. [Figure 11] FIG. 4 is a diagram showing an example of a response from the language model server 300 to a prompt sent by the PC 400 according to an embodiment. [Figure 12] 4 is a diagram showing an example of an API generated by an API generation / execution program 415 for the MFP 100 according to an embodiment. [Figure 13] 10 is a flowchart showing a processing procedure of a PC 400 according to an embodiment. [Figure 14] FIG. 4 is a diagram showing an example of a screen displayed on a display unit 410 by an agent program 413 of a PC 400 according to an embodiment. [Figure 15] FIG. 4 is a diagram showing an example of a response from the language model server 300 to a prompt sent by the PC 400 according to an embodiment. [Figure 16] FIG. 4 is a sequence diagram for investigating the cause of an error or warning in the MFP 100 according to an embodiment. [Figure 17] FIG. 4 is a diagram showing an example of a screen displayed on a display unit 410 by an agent program 413 of a PC 400 according to an embodiment. [Figure 18] 2A and 2B are diagrams showing a method for switching the display language from the display unit 210 of the MFP 100 according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] First Embodiment <System configuration> Hereinafter, a first embodiment of the present invention will be described. Referring to FIG. 1, an example of a system for operating the MFP 100 in a natural language dialogue format according to this embodiment will be described.

[0011] In this system, the MFP 100, which is an image processing apparatus, and the PC 400, which is an information processing apparatus, are connected via the network 500, and further, the language model server 300, which is a server apparatus, is connected via the Internet 200. The present invention supports the setting operation of the MFP 100 in a natural language dialogue format when remotely operating the MFP 100 from the PC 400 or the like in such a system configuration. However, the present invention is not intended to be limited to this system, and as another example, some functions of the PC may be provided via the Internet 200. Also, the configuration of this system is not intended to be limited to the apparatus configuration shown in FIG. 1, and other apparatuses may be included. In this embodiment, the MFP 100 is described as an example of an image processing apparatus, but the present invention can be applied to any apparatus having at least one of an image forming function and an image reading function.

[0012] <Configuration of MFP> Referring to FIG. 2, a hardware configuration example of the MFP 100 according to this embodiment will be described. The MFP 100 includes a control unit 110, a communication connector 206, an operation unit 208, a display unit  210, a printing unit 213, and a reading unit 215. The control unit 110 includes a CPU 201, a ROM 202, a RAM 203, a storage 204, a communication unit I / F 205, an operation unit I / F 207, a UI control unit 209, a printer control I / F 212, and a scanner controller I / F 214.

[0013] The CPU 201 controls the overall operation of the MFP 100. The CPU 201 reads control programs stored in the ROM 202 or the storage 204 and performs various types of control, such as scanning control and printing control. The ROM 202 stores a boot program for loading control programs executable by the CPU 201 stored in the storage 204 into the RAM 203. The RAM 203 is the main memory of the CPU 201 and is used as a work area and a temporary storage area for loading the various control programs stored in the ROM 202 and the storage 204. The storage 204 stores print data, image data, various programs, and various setting information in addition to the executable control programs. Note that in the MFP 100 of this embodiment, one CPU 201 uses one memory (RAM 203) to execute each process shown in the flowcharts described below, but other configurations are also possible. For example, multiple CPUs, RAMs, ROMs, and storages may cooperate to execute each process shown in the flowcharts described below. Furthermore, some of the processing may be performed using hardware circuits such as an application specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0014] The operation unit I / F 207 is connected to an operation unit 208. The operation unit 208 can be configured to include a touch panel input device, push buttons, and the like. The scanner controller I / F 214 is connected to a reading unit 215. The reading unit 215 reads an image on a sheet and generates image data. The image data generated by the reading unit 215 is sent to an external device or printed on a sheet. The reading unit 215 can also scan sheets placed on a document feeder (not shown) to read multiple documents.

[0015] The printing unit 213 is connected to the printer controller I / F 212. The image data to be printed is transferred to the printing unit 213 via the printer controller I / F 212. The printing unit 213 receives the control command and the image data to be printed, and prints an image based on the image data on a sheet. The printing method of the printing unit 213 may be an electrophotographic method or an inkjet method. In the case of the electrophotographic method, an electrostatic latent image is formed on a photoreceptor, then developed with toner, the toner image is transferred to a sheet, and the transferred toner image is fixed to form an image. On the other hand, in the case of the inkjet method, ink is ejected to print an image on a sheet.

[0016] Also, the MFP 100 is connected to the network 500 via a communication connector 206 connected to the communication unit I / F 205. The communication unit I / F 205 is connected to a PC 400 on the network 500 and the Internet 200. The UI control unit 209 is connected to a display unit 210 such as an LCD to control the display unit. The image processing unit 211 performs rotation, compression, resolution conversion, etc. of the image data output to the printing unit 213 via the printer controller I / F 212. These modules are connected so as to exchange data with each other via the system bus 216.

[0017] Programs developed in the RAM 203 and executed by the CPU 201 include a status collection program 217 and an API execution program 218, etc. The status collection program 217 is a program that collects the status information of each module connected via the system bus 216, or receives and executes a setting command from a PC described later via the communication connector 206 and the communication unit I / F 205. The API execution program 218 executes a command received via the communication connector 206 and the communication unit I / F 205.

[0018] <Configuration of the PC> The hardware configuration of the PC 400 according to this embodiment will be described with reference to Fig. 3. The PC 400 includes a CPU 401, a ROM 402, a storage 403, a RAM 404, a communication I / F 405, a communication connector 406, a display unit 410, and an operation unit 411.

[0019] The CPU 401 controls the entire PC 400. The CPU 401 directly reads and executes programs stored in the ROM 402 or storage 403, or loads and executes the programs in the RAM 404. Based on the control of the CPU 401, the communication unit I / F 405 transmits data generated by the CPU 401 to the MFP 100 or language model server 300 via the network 500 or the Internet 200 connected to the communication connector 406. The communication unit I / F 405 also receives data from the MFP 100 or the language model server 300 via the network 500 or the Internet 200. The display unit 410 controls an LCD or the like to display an operation screen or the like to the user. The operation unit 411 accepts input operations, such as a request to change settings or questions for the MFP 100, from the user. These modules are connected to each other via a system bus 420.

[0020] A printer driver list 407 that lists a plurality of pre-installed printer drivers that operate the MFP 100, etc., is stored in the storage 403. The printer driver list 407 is read and used by the CPU 401, and the user can specify, by an agent program 413 (described later) for which MFP a setting change request or question is being sent.

[0021] The RAM 404 stores an MFP status request program 412, an agent program 413, a data repository 414, and an API generation / execution program 415, which are extracted from the storage 403 by the CPU 401. The data repository 414 stores a manual (described later) for each MFP model. The manual may be stored in text format or in a format searchable by known cosine similarity. The agent program 413 accepts questions about the MFP 100 and setting change requests from the user in natural language via the operation unit 411. The agent program 413 also requests the MFP status request program 412 to inquire about error information, current setting information, and the like for the MFP previously set using the printer driver list 407. The agent program 413 also searches for related information from the information stored in the data repository 414 in response to the user's question or setting change request, and transmits the search results, along with the user's question and setting change request information, in natural language to the language model server 300 (described later). The API generation / execution program 415 is a program that generates a command that can be executed by the MFP 100 from a response from the language model server 300 and issues an execution request to the MFP 100 .

[0022] <Language model server configuration> The hardware configuration of the language model server 300 according to this embodiment will be described with reference to Fig. 4. The language model server 300 includes a CPU 301, a ROM 302, a RAM 303, a storage 304, a communication I / F 305, and a communication connector 306.

[0023] CPU 301 controls the entire language model server 300. CPU 301 directly reads and executes programs stored in ROM 302 or storage 304, or loads and executes the programs in RAM 303. Under the control of CPU 301, communication unit I / F 305 transmits data generated by CPU 301 to MFP 100 or PC 400 connected to network 500 via communication connector 306. Communication unit I / F 305 also receives data from MFP 100 or PC 400 connected to network 500 via communication connector 306. These modules are connected via system bus 307 so that they can send and receive data to and from each other. Language model program 308 loaded in RAM 303 is a known arbitrary program that returns an appropriate answer in natural language from information in natural language received from PC 400 via communication connector 306 and communication unit 305 via Internet 200.

[0024] <Settings change sequence> An example of a setting change sequence for the MFP 100 according to this embodiment will be described with reference to Fig. 5. In the figure, "S" indicates a step, and the number following it indicates the step number. This also applies to the subsequent figures. Note that, for ease of explanation, the following description will be given as if various programs are the main actors in executing the processes, but these processes are actually realized by the CPU reading and executing the programs.

[0025] When the agent program 413 is started in the PC 400, in S501 the agent program 413 accesses the printer driver list 407 to check the MFP driver. Furthermore, in S502 the agent program 413 creates a list of drivers that the user can operate based on the accessed printer driver list 407.

[0026] In S503, the PC 400 accepts from the user the selection of an MFP whose settings are to be changed from the driver list created in S502. Here, selecting an MFP means selecting an MFP whose settings are to be changed from among multiple MFPs connected via the Internet 200 or the network 500. Furthermore, in S504, the PC 400 accepts from the user a request to change the settings of the MFP specified in S503 using natural language. In S505, the MFP status request program 412 of the PC 400 generates a command to request the status of the MFP requested by the user in S504. In S506, the MFP status request program 412 sends a status request command to the MFP 100 specified by the user in S503. Upon accepting the status request command, the MFP 100 collects its own status information and setting state (state information) in S507 and returns the information to the PC 400 in S508.

[0027] In S509, the agent program 413 selects data for the specified MFP type from a list of MFP models in the data repository 414 based on the setting request made in S504 and the MFP specified in S503, and searches for similar information from that data. For example, if the setting request is to "change the display language to English," the search here searches for items that match "display," "language," "English," and "change." Next, in S510, the agent program 413 generates a prompt for making an inquiry to the language model server 300. In S511, the agent program 413 transmits the prompt generated in S510 to the language model server 300.

[0028] In S512, the language model program 308 of the language model server 300 receives the prompt generated in S510 via the communication connector 306 and the communication unit I / F 305, and generates an answer. In S513, the language model program 308 returns the answer generated in S512 to the PC 400. In S514, the agent program 413 creates command candidates that can be operated by the user according to the received answer. Further, the agent program 413 generates a screen and causes the display unit 410 to display the screen in S515.

[0029] The user specifies a command using the operation unit 411 from among the command candidates displayed on the display unit 410. In S516, the agent program 413 accepts the user's command specification. On the other hand, if there is no command assumed by the user, it answers to that effect and ends. In S517, the API generation / execution program 415 of the PC 400 creates a command such as a setting change for the MFP 100 from the command specified in S516, and transmits the command to the MFP 100 in S518. In S519, the MFP 100 executes the received command by the API execution program 218 and makes a setting change.

[0030] <Processing Flow of PC400> Referring to FIG. 6, the processing procedure of the PC 400 according to the present embodiment will be described. The processing described below is realized, for example, by the CPU 401 reading out programs stored in the ROM 402 and the storage 403 into the RAM 404 and executing them. In the following, for the sake of easy explanation, there may be cases where various programs executed by the CPU 401 are described with the operating entity.

[0031] In S601, the agent program 413 accesses the printer driver list 407 and creates a list of drivers that the user can operate. In S602, the agent program 413 determines whether the MFP to be operated by the user has been specified by the user from the driver list. If it has been specified from the driver list, the process proceeds to S603, where the agent program 413 sets the specified MFP and proceeds to S605. On the other hand, if it has not been selected, the process proceeds to S604, where the agent program 413 specifies a default MFP and proceeds to S605.

[0032] In S605, the agent program 413 determines whether the user has input a setting request or a question (input in natural language). If there has been an input, the process proceeds to S606, where the agent program 413 checks via the MFP status request program 412 executed by the CPU 401 whether the MFP set in S603 or S604 is connected. If it is connected, the process proceeds to S607, where the MFP status request program 412 sends a status request to the MFP 100 via the network 500. Upon receiving the status request, the MFP 100 executes the status collection program 217 via the CPU 201 to collect the status and return it to the PC 400. As a result, the status collection program 217 acquires the status, and the process proceeds to S608. On the other hand, if the MFP specified by the user is not connected in S606, the agent program 413 proceeds to S608.

[0033] In S608, the agent program 413 searches for data related to the setting change instruction or question entered by the user from among the data corresponding to the MFP specified in S603 or S604, which is stored in the data repository 414. Any search method may be used. Subsequently, in S609, the agent program 413 waits until the search results executed in S608 are obtained. Upon obtaining the search results, in S610, the agent program 413 creates a prompt (text) to be entered into the language model server 300 and transmits the prompt and the data obtained by the search to the language model server 300 via the Internet 200. Here, the transmitted prompt is a prompt generated based on the setting request or question entered by the user. For example, if the user inputs a setting request to change the display language of the MFP 100 to English, a description for changing the display language setting is searched for from among the manual data, which is operation information for the MFP 100, stored in the data repository 408. Then, the manual data (including API identification information) containing the corresponding description is transmitted to the language model server 300 along with the generated prompt. At this time, the generated prompt will be something like, "Manual data regarding the MFP's display language has been entered. Please refer to that manual data and output the method for changing the MFP's display language to English. Also, please identify the API corresponding to that method from the manual data and output it."

[0034] The identification information of the API included in the manual data may be an ID that corresponds one-to-one with a plurality of APIs, or may be a command or URL for executing the API.

[0035] It is also possible to skip the search process of S608 and send all of the manual data corresponding to the MFP specified in S603 or S604 to the language model server 300. In this case, the prompt generated in S610 is something like, "MFP manual data has been entered. Please output a method for changing the display language of the MFP to English after referring to the description regarding changing the display language in the manual data. Also, please identify and output the API corresponding to that method from the manual data."

[0036] The language model server 300 outputs the optimal answer based on the received prompt and manual data, and sends it to the PC 400.

[0037] In S611, the agent program 413 waits until it receives a response from the language model server 300, and if it receives a response, in S612 it determines whether the response contains an API that can be executed by the MFP 100. If there is a corresponding API, the process proceeds to S613; if there is no corresponding API, the process proceeds to S615, where the agent program 413 displays on the display unit 410 that there is no corresponding API, presents the response to the user, and ends the processing of this flowchart.

[0038] Note that the process of S615 may always be executed if the result of S611 is YES, without executing the determination processes of S612 and S613. Also, the answer displayed in the process of S615 does not have to be on display unit 410. For example, PC 400 may transmit the answer to MFP 100 so that the answer is displayed on display unit 210 of MFP 100, and the answer may be provided to the user via the display unit.

[0039] On the one hand, in S613, the agent program 413 checks whether the MFP set in S603 or S604 is connected via the MFP status request program 412 executed by the CPU 401. If it is not connected, the process proceeds to S616, where the agent program 413 displays on the display unit 410 that the API cannot be executed, presents only the answer from the language model server to the user, and ends the processing of this flowchart.

[0040] If it is determined in S613 that the connection exists, the process proceeds to S614. In S614, the agent program 413 generates, via the API generation / execution program 415 from the answer from the language model server 300, an API that can be executed by the API execution program 218 of the MFP 100 via the network 500. Further, the agent program 413 displays the generated API on the display unit 410 as a candidate for execution. In S617, the agent program 413 determines whether there is an API execution instruction from the user. If there is an execution instruction, the process proceeds to S618, where the API generation / execution program 415 transmits the API to the MFP 100 via the network 500 and ends the processing of this flowchart. On the other hand, if there is no execution instruction from the user, the processing of this flowchart ends as it is.

[0041] <Processing Flowchart of MFP 100> Referring to FIG. 7, the processing procedure of the MFP 100 according to this embodiment will be described. The processing described below is realized, for example, when the CPU 201 reads out the programs stored in the ROM 202 and the storage 20 four and executes them in the RAM 203. In the following, for the sake of easy explanation, there may be cases where various programs executed by the CPU 201 are described with the operating entity.

[0042] 7A shows the processing steps when the MFP 100 receives a status request from the MFP status request program 412 of the PC 400 via the network 500. In S701, the CPU 201 waits until it receives a status request from the PC 400. Upon receiving the status request, in S702, the status collection program 217 collects the setting status of the MFP 100 and status information, including errors and warnings, and transmits the collected status information to the PC 400 via the network 500, thereby terminating the processing of this flowchart. In this way, by using the collected status information when providing a response to a user request, it is possible to provide support appropriate to the current situation. For example, it is possible to check the current setting status, or to provide information on device errors and warnings and corresponding solutions.

[0043] 7(b) shows the processing procedure when MFP 100 receives an API from API generation / execution program 415 of PC 400 via network 500. In S703, CPU 201 waits until an API request is received from API generation / execution program 415 of PC 400 via network 500. Upon receiving the API request, in S704 API execution program 218 changes the operation settings according to the content of the requested API, and ends the processing of this flowchart.

[0044] <Screen example> 8, an example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 according to this embodiment will be described. The screen 800 includes an MFP selection item 801, a prompt input item 802, a prompt send button 803, an answer field 804, a slide bar 805, an API candidate field 806, and an API execution button 807. The screen 800 is an interactive screen for interacting with the user in natural language.

[0045] The MFP selection item 801 is a selection item in a driver list based on the printer driver list 407, and the MFP to be operated is specified from the options in accordance with a user operation. The prompt input item 802 is an item in which the user inputs setting changes, questions, etc. for the MFP 100 in natural language. In this way, the MFP selection item 801 and prompt input item 802 are examples of a receiving means. When the prompt send button (object) 803 is operated, the agent program 413 selects data for the MFP set in the MFP selection item 801 from the data storage 414 and searches for related items from within that data. Any search method may be used. Furthermore, the agent program 413 creates a request to generate an API along with the search results and the setting changes or questions input by the user in the prompt input item 802, and transmits the request to the language model server 300.

[0046] The answer field 804 displays answers from the language model server 300, and the display area can be changed (scrolled) using a slide bar 805. The API candidate field 806 displays commands that the API generation / execution program 415 generates from answers from the language model server 300 and that the MFP 100 can execute via the network 500 in an easy-to-understand manner for the user as candidates to be executed. When the user operates the API execution button (object) 807, the API generation / execution program 415 executes the selected command on the MFP 100. As described above, according to this embodiment, not only is an operation method provided, but an API that executes the operation method can also be provided. This allows the user to achieve a desired operation simply by selecting and executing the API, without having to execute the provided operation method. Multiple candidates can also be displayed in the API candidate field 806. As described above, it is possible to provide multiple options related to a user's request, and to provide an API that is more suited to the user's request.

[0047] <Data Storage> Referring to FIG. 9, a portion of the data stored in the data repository 414 in the RAM 404 of the PC 400 according to this embodiment will be described. The data shown in FIG. 9 is provided for illustrative purposes and does not necessarily represent an actual data structure. In the present invention, the data content described below may be managed using any data structure. The data repository 414 is an example of a storage unit and stores operation information for multiple devices communicably connected to the network 500. The operation information includes an operation method for each device, its description (manual), and information about the API for executing the operation method, stored for each operation item. Details of the operation information will be described below. Here, the operation items correspond to the setting items described below with reference to FIG. 18, and include various settings such as settings for the screen displayed after startup / restart, button display settings on the home screen, display language switching settings, and document scanning unit cleaning display settings. In the present invention, by storing a manual for an image processing device such as an MFP in the data repository 414 of the PC 400, which is an information processing device, operation assistance can be easily provided in an interactive format. The operation information may also be stored in an external server device.

[0048] Display language switching explanation item 901 indicates an explanation item for changing the display language setting of MFP 100. 902 indicates an explanation of how to switch the display language, describing the operation method for switching and its explanation. 903 indicates a template for an API for switching the display language, which is an example of a template that language model server 300 uses to generate a command that API generation / execution program 415 of PC 400 can execute on MFP 100.

[0049] A remote UI display language switching explanation item 904 indicates an explanation item for changing the remote UI display language of the MFP 100. Item 905 indicates an explanation of how to switch the remote UI display language, describing the operation method for switching the remote UI display language and its explanation. Item 906 indicates a template for an API for switching the remote UI display language, which, like template 903, is an example of a template that the language model server 300 uses to generate a command that the API generation / execution program 415 of the PC 400 can execute on the MFP 100.

[0050] An auto-sleep time setting change explanation item 907 indicates an explanation item for changing the auto-sleep time setting of the MFP 100. 908 indicates an explanation of how to change the auto-sleep time setting, and describes the method for changing the auto-sleep time setting. 909 indicates a template for an API for changing the auto-sleep time setting, and is an example of a template that the language model server 300 uses to generate a command that the API generation / execution program 415 of the PC 400 can execute on the MFP 100.

[0051] Although an example of data according to this embodiment has been described above, the present invention is not limited to the above-described form. For example, the data may be in a vectorized form so that a search based on cosine similarity can be performed.

[0052] <prompt> An example of a prompt generated by the agent program 413 of the PC 400 according to this embodiment and transmitted to the language model server 300 will be described with reference to FIG.

[0053] Input document 1 1001 and input document 2 1002 are the results of the search by agent program 413 in S509, and although this embodiment shows an example in which two documents are searched, it is also possible that one document is searched, or three or more documents are searched. It is also possible that the search result is 0. If the search result is 0, agent program 413 displays a message to that effect to the user on display unit 410 and ends the process.

[0054] An instruction statement 1003 is an example of an instruction statement generated by the agent program 413 of the PC 400 and sent to the language model server 300. The content of the instruction statement is a document requesting the creation of an API that changes the settings of the MFP 100 entered by the user in the prompt input item 802 from input document 1 1001 and input document 2 1002.

[0055] <Answer example> An example of a response by the language model server 300 to a prompt sent by the PC 400 according to this embodiment will be described with reference to FIG.

[0056] Explanation 1101 of the response to input document #1 1001 explains how to change the display language switching settings. Explanation 1101 is received by agent program 413 of PC 400 via communication connector 406 and communication unit I / F 405, and is displayed in answer column 804 by CPU 401 via display unit 410. API explanation 1102 explains how the settings will be changed when PC 400 executes API candidate 1103. Agent program 413 of PC 400 receives API explanation 1102 via communication connector 406 and communication unit I / F 405, and CPU 401 operates display unit 410 to display in API candidate column 806. API candidate 1103 is generated by API generation / execution program 415 of PC 400 in accordance with this description, generating an operation API that can be executed by MFP 100.

[0057] Similarly, response explanation 1104 for input document 2 1004 explains how to change the settings for switching the remote UI display language. API explanation 1105 explains how the settings will be changed when the PC 400 executes API candidate 1106. The API candidate 1106 is used by the API generation / execution program 415 of the PC 400 to generate an operation API that can be executed by the MFP 100 in accordance with this description.

[0058] 12 shows an example of an API generated by the API generation / execution program 415 for the MFP 100 in S614. API 1201 is an API generated by the API generation / execution program 415 in accordance with the API candidate 1103 received from the language model server 300 so that the MFP 100 can execute it. Similarly, API 1202 is an example of an operation API generated by the API generation / execution program 415 in accordance with the API candidate 1106.

[0059] <Display language switching> An example of an operation for switching the display language from the display unit 210 of the MFP 100 according to this embodiment will be described with reference to Fig. 18. A home screen 1801 is displayed on the display unit 210 of the MFP 100. The display unit 210 is covered by the operation unit 208, such as a touch panel. The home screen 1801 displays various selectable function icons, and includes a menu icon 1802.

[0060] When the user operates a menu icon 1802 on screen 1801, operation unit 208 transmits the coordinates of the press to CPU 201 via operation unit I / F 207. Upon receiving the coordinates, CPU 201 runs a display program (not shown) and displays menu screen 1803 on display unit 210 via UI control unit 209. Various setting items are displayed in a selectable manner on menu screen 1803. When environment setting button 1804 on menu screen 1803 is operated, environment setting screen 1805 is displayed.

[0061] Various setting items related to the environment settings are displayed in a selectable manner on the environment settings screen 1805. When a display settings button 1806 on the environment settings screen 1805 is operated, the screen transitions to a display settings screen 1807. Various setting items related to the display settings are displayed in a selectable manner on the display settings screen 1807. In particular, the display settings screen 1807 includes a display language switching button 1808 and a remote UI display language switching button 1809. When the display language switching button 1808 is operated, the screen transitions to a display language selection screen 1810, and when the remote UI display language switching button 1809 is operated, the screen transitions to a remote UI display language selection screen 1811.

[0062] On the language selection screen 1810, various selectable languages are displayed. When the language to be displayed on the display unit 210 is selected, the language of the display screen of the display unit 210 is switched to the selected language. Similarly, on the remote UI display language selection screen 1811, various selectable languages are displayed. When the language to be displayed on the remote UI is selected, the display language of the remote UI (not shown) is displayed in the selected language.

[0063] As described above, the computer program according to the present embodiment causes the computer to function so as to receive a request in a natural language from a user, which is a request regarding the operation of a device specifying the device. Further, this computer program causes the computer to function so as to input a prompt based on the request and operation information of the device to a language model server. Further, this computer program causes the computer to function so as to provide the user with an answer generated by the language model server based on the prompt and the operation information of the device via the display unit. Thus, according to the present embodiment, for example, a new mechanism for assisting the operation of a device in a natural language dialogue format can be provided. Further, according to the present invention, when remotely operating the MFP, which is a device, from an information processing device, which is an external device, the user operation can be supported in a dialogue format. As a result, the user can save the trouble of reading a manual or the like, and a more user-friendly operation system can be provided.

[0064] <Second Embodiment> <Processing Flow of PC400> A second embodiment of the present invention will be described below. This embodiment is an example that includes processing for checking the setting status of the MFP 100. The processing flow of the PC 400 of this embodiment will be described with reference to FIG. 13. The processing described below is realized, for example, by the CPU 401 reading a program stored in the ROM 402 or storage 403 into the RAM 404 and executing it. To simplify the description below, various programs executed by the CPU 401 may be described as the subject of operations. Note that the same control steps as those in the flowchart of FIG. 6 are assigned the same step numbers, and descriptions thereof will be omitted.

[0065] If it is determined in S612 that there is no API (in this embodiment, a settings confirmation API), the agent program 413 displays on the display unit 410 in S1301 that there is no corresponding API, presents a response to the user, and terminates the processing of this flowchart. Also, if it is determined in S613 that no MFP is connected, the process proceeds to S1302. In S1302, the agent program 413 displays on the display unit 410 that the API cannot be executed because no MFP is connected, presents only the response from the language model server 300 to the user, and terminates the processing of this flowchart. On the other hand, if it is determined that there is a connection, the process proceeds to S614. In S614, based on the response from the language model server 300, the API generation / execution program 415 generates a settings confirmation API that executes the API execution program 218 of the MFP 100, and displays it on the display unit 410 as a candidate for execution.

[0066] In S616, the agent program 413 determines whether or not the user has instructed the MFP 100 to execute the settings confirmation API. If an execution instruction has been received, the process proceeds to S1303, where the API generation / execution program 415 transmits the settings confirmation API to the MFP 100 via the network 500. On the other hand, if an execution instruction has not been received from the user, the process of this flowchart ends.

[0067] In S1303, the agent program 413 waits for a response from the MFP 100 to the setting confirmation API sent in S617. When a response arrives, the process proceeds to S1304, where the agent program 413 accepts the response from the MFP 100, the CPU 401 displays the response on the display unit 410, and the processing of this flowchart ends.

[0068] <Screen example> An example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 in this embodiment will be described with reference to Fig. 14. The same components as those in the screen 800 shown in Fig. 8 are assigned the same numbers and will not be described again.

[0069] 14 includes, in addition to the configuration of screen 800, a setting confirmation item field 1401 for MFP 100, a setting confirmation item execution button 1402, and a setting confirmation field 1403. The setting confirmation item field 1401 displays the items currently set by MFP 100. The setting confirmation item field 1401 is generated by API generation / execution program 415 from a response from language model server 300, and displays to the user in an easy-to-understand manner items for which MFP 100 can check the settings of MFP 100 via network 500 as candidates. When the user operates setting confirmation item execution button 1402, API generation / execution program 415 executes a confirmation command to MFP 100, and displays the confirmation response results in setting confirmation field 1403.

[0070] <Answer example> An example of a response by the language model server 300 to a prompt sent by the PC 400 according to this embodiment will be described with reference to Fig. 15. Components similar to those in the example of the response shown in Fig. 11 are assigned the same numbers and descriptions thereof will be omitted. In this embodiment, in addition to the example of the response shown in Fig. 11 (description 1101), a confirmation API description 1501, a confirmation API 1502, and a return value description 1503 are added.

[0071] Confirmation API description 1501 is a description of confirmation API 1502 for confirming the settings of the MFP, and is displayed in setting confirmation item column 1401. Confirmation API 1502 is used by API generation / execution program 415 of PC 400 to generate an API executable on MFP 100 in accordance with this description. Return value description 1503 indicates a setting confirmation value received as a result of execution of API generation / execution program 415 on MFP 100 in accordance with confirmation API 1502. Agent program 413 refers to return value description 1503 and causes display unit 410 to display the same as setting confirmation column 1403.

[0072] Confirmation API description 1504 is a description of confirmation API 1505 for confirming the settings of the MFP. Confirmation API 1505 is used by API generation / execution program 415 of PC 400 to generate an API executable on MFP 100 in accordance with this description. Return value description 1506 indicates a setting confirmation value received as a result of execution of API generation / execution program 415 on MFP 100 in accordance with confirmation API 1502. Agent program 413 refers to return value description 1506 and causes display unit 410 to display the value as in setting confirmation field 1403.

[0073] As described above, the computer program according to this embodiment causes the computer of the information processing device to further function to acquire status information of a specified device from the specified device. Furthermore, the computer program causes the computer to function to provide the device status information to a user via a display unit. This allows the user to easily check the current status of the specified MFP.

[0074] <Third embodiment> <sequence> A third embodiment of the present invention will be described below. In the above embodiment, a form of supporting the setting operation of an MFP has been described, but the use of the present invention is not limited to setting and can be applied to various operations. Therefore, in this embodiment, operation support when investigating the cause of an error or warning of the MFP 100 in response to dialogue with the user will be described. With reference to FIG. 16, the sequence for investigating the cause of an error or warning of the MFP 100 in this embodiment will be described. The processes of S1601 to S1606 are the same as S501 to S506 in FIG. 5, and therefore description thereof will be omitted.

[0075] In S1607, the MFP 100 acquires the status information, including the current error and setting status, in response to the status request command received in S1606, and returns this information to the PC 400 in S1608. In S1609, the agent program 413 selects data for the specified MFP type from the list of MFP models in the data storage 414 based on the error or warning investigation request made in S1604, and searches for similar information from within that data. For example, if the setting request is "Cannot send fax," the search here searches for items that match "FAX" and "Cannot send."

[0076] Next, in S1610, the agent program 413 generates a prompt to make an inquiry to the language model server 300, and in S1611, sends the generated prompt to the language model server 300. In S1612, the language model program 308 of the language model server 300 receives the prompt generated in S1610 via the communication connector 306 and the communication unit I / F 305, and generates a response. In S1613, the agent program 413 receives the response generated by the language model server 300, and in S1614 generates a screen for the user to see the cause of the error or warning and how to deal with it, based on the received response, and in S1615 displays this on the display unit 410.

[0077] <Screen example> 17, an example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 according to this embodiment will be described. A screen 1700 includes an MFP selection item 1701, a prompt input item 1702, a prompt send button 1703, and an answer field 1704.

[0078] The MFP selection item 1701 is a selection item that allows the user to select a driver list to be operated by the user. The user selects the MFP where the problem is occurring from the selection. When the MFP is selected, the agent program 413 acquires information such as errors and warnings from the selected MFP.

[0079] The prompt input field 1702 is a field in which the user enters problems occurring with the MFP 100 using natural language. When the prompt send button 1703 is operated, the agent program 413 selects data for the MFP set in the MFP selection field 1701 from the data storage 414 and searches for items related to MFP 100 error information and the problem entered by the user from among the data. Any known search method may be used. The agent program 413 creates a request to generate an API along with the search results and the problem with the MFP 100 entered by the user in the prompt input field 1702, and sends this request to the language model server 300. The answer field 1704 is an area in which an answer from the language model server 300 is displayed.

[0080] As described above, in the computer program according to this embodiment, the request relates to operations for investigating the cause of a device error or warning, and the response to the request includes a method of dealing with the error or warning. In this way, according to this embodiment, the user can be effectively supported not only with regard to operation settings, but also with regard to operation methods or methods of dealing with errors, etc.

[0081] The disclosure of this specification includes the following computer program, information processing device, and information processing method. (Item 1) Computer, a receiving means for receiving a request from a user in natural language, the request specifying a device and relating to an operation of the device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; providing means for providing a response generated by the language model server based on the prompt and the operation information of the device to the user via a display unit; A computer program characterized by causing the computer to function as follows. (Item 2) The computer further comprises: 2. The computer program according to item 1, characterized in that the computer program functions as a search means for searching operation information relating to the request from a storage means that holds operation information of the specified device. (Item 3) The computer program according to item 2, characterized in that the storage means stores, as the operation information, an operation method for the device, a description thereof, and identification information of an API that executes the operation method, for each operation item. (Item 4) The computer program according to item 2 or 3, characterized in that the response includes information indicating an operation method related to the request and identification information of one or more APIs that execute a specified process related to the request on the device. (Item 5) 5. The computer program according to item 4, wherein the providing means provides the one or more APIs in a selectable manner on the display unit. (Item 6) the providing means provides information to the user by displaying an interactive screen on the display unit; The interactive screen may include: an area for inputting the request in natural language from the user and an object for executing the request; an area for displaying the answer; an area for displaying the one or more APIs and an object for selecting the one or more APIs; 6. The computer program according to item 5, comprising: (Item 7) The computer further comprises: 7. The computer program according to any one of items 4 to 6, characterized in that the computer program functions as an execution means for executing an API selected from the one or more provided APIs in accordance with a user operation. (Item 8) The computer further comprises: functioning as an acquisition means for acquiring status information of a designated device from the designated device; 8. The computer program according to any one of items 1 to 7, wherein the providing means further provides the status information of the device to the user via the display unit. (Item 9) The request relates to an operation in investigating the cause of an error or warning on the device; 9. The computer program according to item 8, wherein the response includes a method for dealing with the error or warning. (Item 10) When the request is a setting operation for the device, the response further includes identification information of one or more APIs for confirming the setting status of the setting item corresponding to the request in the device; 8. The computer program according to any one of items 1 to 7, wherein the providing means further provides the one or more APIs on the display unit in a selectable manner. (Item 11) a receiving means for receiving a request from a user in natural language, the request specifying a device and relating to an operation of the device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; providing means for providing a response generated by the language model server based on the prompt and the operation information of the device to the user via a display unit; An information processing device comprising: (Item 12) An information processing method, comprising: a receiving step of receiving a request from a user in natural language, the request specifying a device and relating to an operation of the device; an input step of inputting a prompt based on the request and operation information of the device to a language model server; a providing step of providing a response generated by the language model server based on the prompt and operation information of the device to the user via a display unit; An information processing method comprising:

[0082] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0083] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0084] 100: MFP, 200: Internet, 300: Language model server, 400: PC, 500: Network

Claims

1. Computer, a receiving means for receiving a request from a user in natural language, the request specifying the device and relating to an operation of the device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; providing means for providing a user with an answer generated by the language model server based on the prompt and operation information of the device via a display unit; A computer program characterized by causing the computer to function as follows.

2. The computer further comprises:

2. The computer program according to claim 1, wherein the computer program functions as a search unit that searches a storage unit that holds operation information of the specified device for operation information relating to the request.

3. 3. The computer program according to claim 2, wherein the storage means stores, as the operation information, an operation method for the device, a description thereof, and identification information of an API that executes the operation method, for each operation item.

4. 4. The computer program according to claim 3, wherein the response includes information indicating an operation method related to the request and identification information of one or more APIs that execute predetermined processing related to the request on a device.

5. 5. The computer program according to claim 4, wherein the providing means provides the one or more APIs in a selectable manner on the display unit.

6. the providing means provides information to the user by displaying an interactive screen on the display unit; The interactive screen may include: an area for inputting the request in natural language from the user and an object for executing the request; an area for displaying the answer; an area for displaying the one or more APIs and an object for selecting the one or more APIs; 6. The computer program of claim 5, further comprising:

7. The computer further comprises:

7. The computer program according to claim 6, wherein the computer program functions as an execution unit that executes an API selected from the one or more provided APIs in accordance with a user operation.

8. The computer further comprises: functioning as an acquisition means for acquiring status information of a designated device from the designated device; 8. The computer program according to claim 1, wherein the providing means further provides the status information of the device to the user via the display unit.

9. The request relates to an operation in investigating the cause of an error or warning on the device; 9. The computer program according to claim 8, wherein the response includes a method for dealing with the error or warning.

10. When the request is a setting operation for the device, the response further includes identification information of one or more APIs for confirming the setting state of the setting item corresponding to the request in the device; 8. The computer program according to claim 1, wherein the providing means further provides the one or more APIs on the display unit in a selectable manner.

11. a receiving means for receiving a request from a user in natural language, the request specifying the device and relating to an operation of the device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; providing means for providing a user with an answer generated by the language model server based on the prompt and operation information of the device via a display unit; An information processing device comprising:

12. An information processing method, comprising: a receiving step of receiving a request from a user in natural language, the request specifying the device and relating to an operation of the device; an input step of inputting a prompt based on the request and operation information of the device into a language model server; a providing step of providing a response generated by the language model server based on the prompt and operation information of the device to the user via a display unit; An information processing method comprising:

Images