Computer program, information processing device, and information processing method

Abstract

A computer program according to the present invention causes a computer to function so as to accept a request in a natural language from a user, the request specifying a device and being related to an operation of the device. The computer program also causes the computer to function so as to input a prompt and device operation information based on the request to a language model server. The computer program also causes the computer to function so as to provide a user, via a display unit, with an answer generated by the language model server on the basis of the prompt and the device operation information.

Description

Computer program, information processing device, and information processing method

[0001] The present invention relates to a computer program, an information processing device, and an information processing method that utilize a natural language model.

[0002] Among methods for operating devices from a PC or the like via a network, there is a method for operating a device from a remote UI. Patent Document 1 proposes a mechanism for improving the operability of software used in device remote management technology. Also known is a service that provides information to users in a natural language interactive format using a generation AI such as ChatGPT (registered trademark).

[0003] Patent No. 5738388

[0004] However, the above-mentioned conventional technology has the following problems: While the above-mentioned conventional technology allows device settings to be made from a web browser, depending on the device, there are many setting items, which requires cumbersome user operations. Also, users need to refer to a manual or the like to find out which items should be set, which takes time for the user to look up the information.

[0005] The present invention has been made in consideration of at least one of the above problems, and provides a novel mechanism for supporting device operation in natural language.

[0006] The present invention is characterized in that, for example, a computer functions as: a receiving means for receiving a request from a user in natural language, the request relating to the operation of a specified device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; and a providing means for providing a response generated by the language model server based on the prompt and the operation information of the device to the user via a display unit.

[0007] According to the present invention, it is possible to support device operation in natural language.

[0008] FIG. 1 is a diagram showing an example of a system for operating an MFP 100 in an interactive format using natural language according to an embodiment. FIG. 1 is a diagram showing an example of the hardware configuration of an MFP 100 according to an embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of a PC 400 according to an embodiment. FIG. 3 is a diagram showing an example of the hardware configuration of a language model server 300 according to an embodiment. FIG. 4 is a sequence diagram for changing settings of an MFP 100 according to an embodiment. FIG. 5 is a flowchart showing a processing procedure of a PC 400 according to an embodiment. FIG. 6 is a flowchart showing a processing procedure of an MFP 100 according to an embodiment. FIG. 7 is a flowchart showing a processing procedure of an MFP 100 according to an embodiment. FIG. 8 is a flowchart showing a processing procedure of an MFP 100 according to an embodiment. FIG. 9 is a diagram showing an example of a screen displayed on a display unit 410 by an agent program 413 of a PC 400 according to an embodiment. FIG. 10 is a diagram showing an example of part of data in a data repository 414 deployed in a RAM 404 of a PC 400 according to an embodiment. FIG. 11 is a diagram showing an example of data in a data repository 414 deployed in a RAM 404 of a PC 400 according to an embodiment. FIG. 12 is a diagram showing an example of a prompt generated by an agent program 413 of a PC 400 according to an embodiment and transmitted to a language model server 300. 1 is a diagram showing an example of a prompt generated by an agent program 413 of the PC 400 and transmitted to the language model server 300 according to an embodiment. FIG. 2 is a diagram showing an example of a response from the language model server 300 to a prompt transmitted by the PC 400 according to an embodiment. FIG. 3 is a diagram showing an example of an API generated by an API generation / execution program 415 for the MFP 100 according to an embodiment. A flowchart showing a processing procedure of the PC 400 according to an embodiment. A flowchart showing a processing procedure of the PC 400 according to an embodiment. A flowchart showing a processing procedure of the PC 400 according to an embodiment. A diagram showing an example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 according to an embodiment. A diagram showing an example of a response from the language model server 300 to a prompt transmitted by the PC 400 according to an embodiment. A diagram showing an example of a response from the language model server 300 to a prompt transmitted by the PC 400 according to an embodiment. A sequence diagram for investigating the cause of an error or warning in the MFP 100 according to an embodiment. A diagram showing an example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 according to an embodiment.1A and 1B are diagrams showing a method for switching the display language from the display unit 210 of the MFP 100 according to an embodiment, and a method for switching the display language from the display unit 210 of the MFP 100 according to an embodiment.

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] First Embodiment System Configuration A first embodiment of the present invention will be described below with reference to Fig. 1. An example of a system for operating an MFP 100 in an interactive manner using natural language according to this embodiment will be described.

[0011] In this system, an MFP 100, which is an image processing device, and a PC 400, which is an information processing device, are connected via a network 500, and a language model server 300, which is a server device, is further connected via the Internet 200. In this system configuration, the present invention provides support for setting up the MFP 100 in a natural language interactive format when remotely controlling the MFP 100 from the PC 400 or the like. However, the present invention is not intended to be limited to this system, and as another example, some functions of the PC may be provided via the Internet 200. Furthermore, the configuration of this system is not intended to be limited to the device configuration shown in FIG. 1, and other devices may be included. In this embodiment, the MFP 100 is used as an example of an image processing device, but the present invention can be applied to any device that has at least one of an image forming function and an image reading function.

[0012] <Configuration of MFP> An example of the hardware configuration of the MFP 100 according to this embodiment will be described with reference to Fig. 2. The MFP 100 includes a control unit 110, a communication connector 206, an operation unit 208, a display unit 210, a printing unit 213, and a reading unit 215. The control unit 110 includes a CPU 201, a ROM 202, a RAM 203, a storage 204, a communication unit I / F 205, an operation unit I / F 207, a UI control unit 209, a printer control I / F 212, and a scanner controller I / F 214.

[0013] The CPU 201 controls the overall operation of the MFP 100. The CPU 201 reads control programs stored in the ROM 202 or the storage 204 and performs various control operations, such as scanning control and printing control. The ROM 202 mainly stores a boot program for loading control programs executable by the CPU 201 stored in the storage 204 into the RAM 203. The RAM 203 is the main memory of the CPU 201 and is used as a work area and a temporary storage area for loading the various control programs stored in the ROM 202 and the storage 204. The storage 204 stores print data, image data, various programs, and various setting information in addition to the executable control programs. Note that in the MFP 100 of this embodiment, one CPU 201 uses one memory (RAM 203) to execute each process shown in the flowcharts described below, but other configurations are also possible. For example, multiple CPUs, RAMs, ROMs, and storages may cooperate to execute each process shown in the flowcharts described below. Furthermore, some of the processing may be performed using hardware circuits such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array).

[0014] The operation unit I / F 207 is connected to an operation unit 208. The operation unit 208 can be configured to include a touch panel input device, push buttons, and the like. The scanner controller I / F 214 is connected to a reading unit 215. The reading unit 215 reads an image on a sheet and generates image data. The image data generated by the reading unit 215 is transmitted to an external device or printed on a sheet. The reading unit 215 can also scan sheets placed on a document feeder (not shown) to read multiple documents.

[0015] The printing unit 213 is connected to the printer controller I / F 212. Image data to be printed is transferred to the printing unit 213 via the printer controller I / F 212. The printing unit 213 receives control commands and image data to be printed, and prints an image based on the image data on a sheet. The printing method of the printing unit 213 may be an electrophotographic method or an inkjet method. In the case of the electrophotographic method, an electrostatic latent image is formed on a photosensitive member, then developed with toner, and the toner image is transferred to a sheet, and the transferred toner image is fixed to form an image. On the other hand, in the case of the inkjet method, ink is ejected to print an image on a sheet.

[0016] The MFP 100 is also connected to a network 500 via a communication connector 206 connected to a communication unit I / F 205. The communication unit I / F 205 is connected to a PC 400 on the network 500 and the Internet 200. A UI control unit 209 is connected to a display unit 210 such as an LCD and controls the display unit. An image processing unit 211 performs rotation, compression, resolution conversion, etc. of image data to be output to a printing unit 213 via a printer controller I / F 212. These modules are connected to each other via a system bus 216 so as to exchange data with each other.

[0017] Programs that are loaded into RAM 203 and executed by CPU 201 include a status collection program 217 and an API execution program 218. The status collection program 217 is a program that collects status information of each module connected via a system bus 216, and receives and executes setting commands from a PC (described later) via a communication connector 206 and a communication unit I / F 205. The API execution program 218 executes commands received via the communication connector 206 and a communication unit I / F 205.

[0018] <PC Configuration> The hardware configuration of the PC 400 according to this embodiment will be described with reference to Fig. 3. The PC 400 includes a CPU 401, a ROM 402, a storage 403, a RAM 404, a communication interface 405, a communication connector 406, a display unit 410, and an operation unit 411.

[0019] The CPU 401 controls the entire PC 400. The CPU 401 directly reads and executes programs stored in the ROM 402 or storage 403, or loads and executes the programs in the RAM 404. Based on the control of the CPU 401, the communication unit I / F 405 transmits data generated by the CPU 401 to the MFP 100 or the language model server 300 via the network 500 or the Internet 200 connected to the communication connector 406. The communication unit I / F 405 also receives data from the MFP 100 or the language model server 300 via the network 500 or the Internet 200. The display unit 410 controls an LCD or the like to display an operation screen or the like to the user. The operation unit 411 accepts input operations, such as a request to change settings or questions, for the MFP 100 from the user. These modules are connected to each other via a system bus 420.

[0020] A printer driver list 407 that lists a plurality of pre-installed printer drivers that operate the MFP 100, etc., is stored in the storage 403. The printer driver list 407 is read and used by the CPU 401, and the user can specify, by an agent program 413 (described later) for which MFP a setting change request or question is being sent.

[0021] The RAM 404 stores an MFP status request program 412, an agent program 413, a data repository 414, and an API generation / execution program 415, which are extracted from the storage 403 by the CPU 401. The data repository 414 stores manuals (described below) for each MFP model. The stored manuals may be in text format or a format searchable by known cosine similarity. The agent program 413 accepts questions about the MFP 100 and setting change requests from the user in natural language via the operation unit 411. The agent program 413 also requests the MFP status request program 412 to inquire about error information, current setting information, and the like from the MFP previously set in the printer driver list 407. The agent program 413 also searches for related information from the information stored in the data repository 414 in response to the user's question or setting change request, and transmits the search results, along with the user's question and setting change request information, in natural language to the language model server 300 (described below). The API generation / execution program 415 is a program that generates a command that can be executed by the MFP 100 from a response from the language model server 300 and issues an execution request to the MFP 100 .

[0022] <Configuration of Language Model Server> The hardware configuration of the language model server 300 according to this embodiment will be described with reference to Fig. 4. The language model server 300 includes a CPU 301, a ROM 302, a RAM 303, a storage 304, a communication I / F 305, and a communication connector 306.

[0023] The CPU 301 controls the entire language model server 300. The CPU 301 directly reads and executes programs stored in the ROM 302 or storage 304, or loads and executes the programs in the RAM 303. The communication unit I / F 305 transmits data generated by the CPU 301 to the MFP 100 or the PC 400 connected to the network 500 via the communication connector 306 under the control of the CPU 301. The communication unit I / F 305 also receives data from the MFP 100 or the PC 400 connected to the network 500 via the communication connector 306. These modules are connected via a system bus 307 so that they can send and receive data to and from each other. The language model program 308 loaded in the RAM 303 is a known arbitrary program that returns an appropriate response in natural language from information in natural language received from the PC 400 via the communication connector 306 and the communication unit 305 via the Internet 200.

[0024] <Settings Change Sequence> An example of a settings change sequence for the MFP 100 according to this embodiment will be described with reference to Figure 5. In the figure, "S" indicates a step, and the number following it indicates the step number. This also applies to the subsequent figures. Note that, for ease of explanation, the following description will be given as if various programs are the main actors in executing the processes, but these processes are actually realized by the CPU reading and executing the programs.

[0025] When the agent program 413 is started in the PC 400, in S501 the agent program 413 accesses the printer driver list 407 to check the MFP driver. Furthermore, in S502 the agent program 413 creates a list of drivers that the user can operate based on the accessed printer driver list 407.

[0026] In S503, the PC 400 accepts from the user a selection of an MFP whose settings are to be changed from the driver list created in S502. Here, selecting an MFP refers to selecting an MFP whose settings are to be changed from among multiple MFPs connected via the Internet 200 or the network 500. Furthermore, in S504, the PC 400 accepts from the user a request for changing the settings of the MFP specified in S503 using natural language. In S505, the MFP status request program 412 of the PC 400 generates a command for requesting the status of the MFP requested by the user in S504. In S506, the MFP status request program 412 transmits a status request command to the MFP 100 specified by the user in S503. Upon accepting the status request command, the MFP 100 collects its own status information and setting state (state information) in S507 and returns the information to the PC 400 in S508.

[0027] In S509, the agent program 413 selects data for the specified MFP type from the list of MFP models in the data repository 414 based on the setting request made in S504 and the MFP specified in S503, and searches for similar information from that data. For example, if the setting request is to "change the display language to English," the search involves searching for items that match "display," "language," "English," and "change." Next, in S510, the agent program 413 generates a prompt for making an inquiry to the language model server 300. In S511, the agent program 413 transmits the prompt generated in S510 to the language model server 300.

[0028] In S512, the language model program 308 of the language model server 300 receives the prompt generated in S510 via the communication connector 306 and the communication unit I / F 305 and generates a response. In S513, the language model program 308 returns the response generated in S512 to the PC 400. In S514, the agent program 413 creates command candidates that the user can operate based on the received response. The agent program 413 also generates a screen and displays the screen on the display unit 410 in S515.

[0029] The user uses the operation unit 411 to specify a command from among the command candidates displayed on the display unit 410. In S516, the agent program 413 accepts the user's command specification. On the other hand, if the command the user expects is not available, the agent program 413 responds to that effect and terminates. In S517, the API generation / execution program 415 of the PC 400 creates a command for the MFP 100, such as a setting change command, from the command specified in S516, and sends the command to the MFP 100 in S518. In S519, the MFP 100 executes the received command using the API execution program 218 and changes the settings.

[0030] <Processing Flow of PC 400> The processing procedure of the PC 400 according to this embodiment will be described with reference to Figures 6A and 6B. The processing described below is realized, for example, by the CPU 401 reading a program stored in the ROM 402 or storage 403 into the RAM 404 and executing it. Note that, for ease of explanation, the various programs executed by the CPU 401 may be described as the operating subjects below.

[0031] In S601, the agent program 413 accesses the printer driver list 407 and creates a list of drivers that the user can operate. In S602, the agent program 413 determines whether the MFP that the user will operate has been specified by the user from the driver list. If it has been specified from the driver list, the process proceeds to S603, where the agent program 413 sets the specified MFP and proceeds to S605. On the other hand, if it has not been selected, the process proceeds to S604, where the agent program 413 specifies a default MFP and proceeds to S605.

[0032] In S605, the agent program 413 determines whether the user has input a setting request or question (input in natural language). If there has been input, the process proceeds to S606, where the agent program 413 checks via the MFP status request program 412 executed by the CPU 401 whether the MFP set in S603 or S604 is connected. If it is connected, the process proceeds to S607, where the MFP status request program 412 sends a status request to the MFP 100 via the network 500. Upon receiving the status request, the MFP 100 executes the status collection program 217 via the CPU 201 to collect the status and return it to the PC 400. As a result, the status collection program 217 acquires the status, and the process proceeds to S608. On the other hand, if the MFP specified by the user is not connected in S606, the agent program 413 proceeds to S608.

[0033] In S608, the agent program 413 searches for data related to the setting change instruction or question entered by the user from among the data corresponding to the MFP specified in S603 or S604, which is stored in the data repository 414. Any search method may be used. Subsequently, in S609, the agent program 413 waits until the search results executed in S608 are obtained. Upon obtaining the search results, in S610, the agent program 413 creates a prompt (text) to be input to the language model server 300 and transmits the prompt and the searched data to the language model server 300 via the Internet 200. Here, the transmitted prompt is a prompt generated based on the setting request or question entered by the user. For example, if the user inputs a setting request to change the display language of the MFP 100 to English, a description for changing the display language setting is searched for from among the manual data, which is operation information for the MFP 100, stored in the data repository 408. Then, the manual data containing the corresponding description (including API identification information) is transmitted to the language model server 300 along with the generated prompt. At this time, the generated prompt is something like, "Manual data regarding the MFP's display language has been entered. Please refer to that manual data and output the method for changing the MFP's display language to English. Also, please identify the API corresponding to that method from the manual data and output it."

[0034] The identification information of the API included in the manual data may be an ID that corresponds one-to-one with a plurality of APIs, or may be a command or URL for executing the API.

[0035] It is also possible to skip the search process of S608 and send all of the manual data corresponding to the MFP specified in S603 or S604 to the language model server 300. In this case, the prompt generated in S610 is something like, "MFP manual data has been input. Please output a method for changing the MFP's display language to English after referring to the description regarding changing the display language in the manual data. Also, please identify and output the API corresponding to that method from the manual data."

[0036] The language model server 300 outputs an optimal answer based on the received prompt and manual data, and transmits it to the PC 400.

[0037] In S611, the agent program 413 waits until it receives a response from the language model server 300, and if it receives a response, in S612 it determines whether the response contains an API that can be executed by the MFP 100. If there is a corresponding API, the process proceeds to S613; if there is no corresponding API, the process proceeds to S615, where the agent program 413 displays on the display unit 410 that there is no corresponding API, presents the response to the user, and ends the processing of this flowchart.

[0038] Note that the process of S615 may always be executed if the answer is YES in S611 without executing the determination processes of S612 and S613. Also, the answer displayed in the process of S615 does not have to be on display unit 410. For example, PC 400 may transmit the answer to MFP 100 so that the answer is displayed on display unit 210 of MFP 100, and the answer may be provided to the user via the display unit.

[0039] Meanwhile, in S613, the agent program 413 checks whether the MFP set in S603 or S604 is connected via the MFP status request program 412 executed by the CPU 401. If the MFP is not connected, the process proceeds to S616, where the agent program 413 displays on the display unit 410 that the API cannot be executed, presents only the response from the language model server to the user, and ends the processing of this flowchart.

[0040] If it is determined in S613 that a connection has been established, the process proceeds to S614. In S614, the agent program 413 generates an API that can be executed by the API execution program 218 of the MFP 100 via the network 500 using the API generation / execution program 415 based on the response from the language model server 300. Furthermore, the agent program 413 displays the generated API on the display unit 410 as a candidate for execution. In S617, the agent program 413 determines whether or not the user has issued an instruction to execute the API. If an execution instruction has been issued, the process proceeds to S618, in which the API generation / execution program 415 transmits the API to the MFP 100 via the network 500, and the process of this flowchart ends. On the other hand, if an execution instruction has not been issued from the user, the process of this flowchart ends as is.

[0041] <Processing Flow of MFP 100> The processing procedure of the MFP 100 according to this embodiment will be described with reference to Figures 7A and 7B. The processing described below is realized, for example, by the CPU 201 reading a program stored in the ROM 202 or storage 204 into the RAM 203 and executing it. Note that, for ease of explanation, the various programs executed by the CPU 201 may be described as the operating subjects below.

[0042] 7A shows the processing procedure when the MFP 100 receives a status request from the MFP status request program 412 of the PC 400 via the network 500. In S701, the CPU 201 waits until it receives a status request from the PC 400. Upon receiving the status request, in S702, the status collection program 217 collects the MFP 100's status, including its settings and errors and warnings, and transmits the collected status information to the PC 400 via the network 500, thereby terminating the processing of this flowchart. In this way, by using the collected status information when providing a response to a user's request, support tailored to the current situation can be provided. For example, it is possible to check the current settings, or to provide information on device errors and warnings, as well as corresponding solutions.

[0043] 7B shows the processing procedure when MFP 100 receives an API transmission from API generation / execution program 415 of PC 400 via network 500. In S703, CPU 201 waits until an API request is received from API generation / execution program 415 of PC 400 via network 500. Upon receiving the API request, in S704, API execution program 218 changes the operation settings according to the content of the requested API, and ends the processing of this flowchart.

[0044] 8, an example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 according to this embodiment will be described. The screen 800 includes an MFP selection item 801, a prompt input item 802, a prompt send button 803, an answer field 804, a slide bar 805, an API candidate field 806, and an API execution button 807. The screen 800 is an interactive screen for interacting with the user in natural language.

[0045] The MFP selection item 801 is a selection item of a driver list based on the printer driver list 407, and the MFP to be operated is specified from the options in accordance with a user operation. The prompt input item 802 is an item in which the user inputs setting changes, questions, etc. for the MFP 100 in natural language. In this way, the MFP selection item 801 and the prompt input item 802 are examples of a receiving means. When the prompt send button (object) 803 is operated, the agent program 413 selects data for the MFP set in the MFP selection item 801 from the data storage 414 and searches for related items from within that data. Any search method may be used. Furthermore, the agent program 413 creates a request to generate an API along with the search results and the setting changes or questions input by the user in the prompt input item 802, and transmits the request to the language model server 300.

[0046] The answer field 804 displays the answer from the language model server 300, and the display area can be changed (scrolled) using a slide bar 805. The API candidate field 806 displays commands that the API generation / execution program 415 generates from the answer from the language model server 300 and that the MFP 100 can execute via the network 500 in an easy-to-understand manner for the user as candidates to be executed. When the user operates the API execution button (object) 807, the API generation / execution program 415 executes the selected command on the MFP 100. As described above, according to this embodiment, not only is an operation method provided, but an API that executes the operation method can also be provided. This allows the user to achieve the desired operation simply by selecting and executing the API, without having to execute the provided operation method. Multiple candidates can also be displayed in the API candidate field 806. In this way, multiple options related to the user's request can be provided, making it possible to provide an API that better meets the user's request.

[0047] <Data Repository> Referring to Figures 9A and 9B, a portion of the data in the data repository 414 stored in the RAM 404 of the PC 400 according to this embodiment will be described. The data shown in Figures 9A and 9B is provided for illustrative purposes and does not necessarily represent the actual data structure. In the present invention, the data content described below may be managed using any data structure. The data repository 414 is an example of a storage unit and stores operation information for multiple devices communicably connected to the network 500. The operation information includes, for each operation item, an operation method for each device, its description (manual), and information about the API that executes the operation method. Details of the operation information are described below. Here, the operation items correspond to the setting items described below with reference to Figure 18, and include various settings such as settings for the screen displayed after startup / restart, button display settings on the home screen, display language switching settings, and document scanning unit cleaning display settings. In the present invention, by storing manuals for an image processing device such as an MFP in the data repository 414 of the PC 400, which is an information processing device, operation assistance can be easily provided in an interactive format. The operation information and the like may be stored in an external server device or the like.

[0048] A display language switching explanation item 901 indicates an explanation item for changing the display language setting of the MFP 100. Item 902 indicates an explanation of how to switch the display language, including the operation method for switching and its explanation. Item 903 indicates a template for an API for switching the display language, which is an example of a template that the language model server 300 uses to generate a command that the API generation / execution program 415 of the PC 400 can execute on the MFP 100.

[0049] A remote UI display language switching explanation item 904 indicates an explanation item for changing the remote UI display language of the MFP 100. Item 905 indicates an explanation of how to switch the remote UI display language, and describes the operation method for switching the display language of the remote UI and its explanation. Item 906 indicates a template for an API for switching the remote UI display language, which, like the template 903, is an example of a template that the language model server 300 uses to generate a command that the API generation / execution program 415 of the PC 400 can execute on the MFP 100.

[0050] An auto sleep time setting change explanation item 907 indicates an explanation item for changing the auto sleep time setting of the MFP 100. Item 908 indicates an explanation of how to change the auto sleep time setting, and describes the method for changing the auto sleep time setting. Item 909 indicates a template for an API for changing the auto sleep time setting, which is an example of a template that the language model server 300 uses to generate a command that the API generation / execution program 415 of the PC 400 can execute on the MFP 100.

[0051] Although an example of data according to this embodiment has been described above, the present invention is not limited to the above-described form. For example, the data may be in a vectorized form so that a search based on cosine similarity can be performed.

[0052] <Prompt> An example of a prompt that is generated by the agent program 413 of the PC 400 according to this embodiment and sent to the language model server 300 will be described with reference to FIGS. 10A and 10B.

[0053] Input document No. 1 1001 and input document No. 2 1002 are the results of the search by agent program 413 in S509. In this embodiment, an example is shown in which two documents are searched, but one document may be searched, or three or more documents may be searched. Also, the search result may be zero. If the search result is zero, agent program 413 displays a message to that effect to the user on display unit 410 and terminates the process.

[0054] An instruction statement 1003 is an example of an instruction statement generated by the agent program 413 of the PC 400 and sent to the language model server 300. The content of the instruction statement is a document requesting the creation of an API that changes the settings of the MFP 100 input by the user in the prompt input item 802 from input document 1 1001 and input document 2 1002.

[0055] <Example of Response> An example of a response by the language model server 300 to a prompt sent by the PC 400 according to this embodiment will be described with reference to FIG.

[0056] The response explanation 1101 for input document no. 1 1001 explains how to change the display language switching settings. The explanation 1101 is received by the agent program 413 of the PC 400 via the communication connector 406 and the communication unit I / F 405, and is displayed in the response field 804 by the CPU 401 via the display unit 410. The API explanation 1102 explains how the settings will be changed when the PC 400 executes the API candidate 1103. The agent program 413 of the PC 400 receives the API explanation 1102 via the communication connector 406 and the communication unit I / F 405, and the CPU 401 operates the display unit 410 to display the API candidate field 806. The API candidate 1103 is used by the API generation / execution program 415 of the PC 400 to generate an operation API executable on the MFP 100 in accordance with this description.

[0057] Similarly, the response explanation 1104 for input document 2 1004 explains how to change the settings for switching the remote UI display language. The API explanation 1105 explains how the settings will be changed when the PC 400 executes the API candidate 1106. The API candidate 1106 is used by the API generation / execution program 415 of the PC 400 to generate an operation API that can be executed by the MFP 100 in accordance with this description.

[0058] 12 shows an example of an API generated by the API generation / execution program 415 for the MFP 100 in step S614. An API 1201 is an API generated by the API generation / execution program 415 in accordance with the API candidate 1103 received from the language model server 300 so that the MFP 100 can execute the API. Similarly, an API 1202 is an example of an operation API generated by the API generation / execution program 415 in accordance with the API candidate 1106.

[0059] 18A and 18B , an example of an operation for switching the display language from the display unit 210 of the MFP 100 according to this embodiment will be described. A home screen 1801 is displayed on the display unit 210 of the MFP 100. The display unit 210 is covered by the operation unit 208, such as a touch panel. The home screen 1801 displays various selectable function icons, including a menu icon 1802.

[0060] When the user operates a menu icon 1802 on screen 1801, the operation unit 208 transmits the pressed coordinates to the CPU 201 via the operation unit I / F 207. The CPU 201, which has received the coordinates, runs a display program (not shown) and displays a menu screen 1803 on the display unit 210 via the UI control unit 209. Various setting items are displayed in a selectable manner on the menu screen 1803. When an environment setting button 1804 on the menu screen 1803 is operated, an environment setting screen 1805 is displayed.

[0061] The environment setting screen 1805 displays various setting items related to the environment setting in a selectable manner. When a display setting button 1806 on the environment setting screen 1805 is operated, the screen transitions to a display setting screen 1807. The display setting screen 1807 displays various setting items related to the display setting in a selectable manner. In particular, the display setting screen 1807 includes a display language switching button 1808 and a remote UI display language switching button 1809. When the display language switching button 1808 is operated, the screen transitions to a display language selection screen 1810, and when the remote UI display language switching button 1809 is operated, the screen transitions to a remote UI display language selection screen 1811.

[0062] A display language selection screen 1810 displays various selectable languages, and when a language to be displayed on the display unit 210 is selected, the language of the display screen of the display unit 210 switches to the selected language. Similarly, a remote UI display language selection screen 1811 displays various selectable languages, and when a language to be displayed on the remote UI is selected, the display language of the remote UI (not shown) is displayed in the selected language.

[0063] As described above, the computer program according to this embodiment causes a computer to function to accept a request from a user in natural language, the request being related to the operation of a specified device. The computer program also causes the computer to input a prompt based on the request and device operation information to a language model server. The computer program also causes the computer to provide a response generated by the language model server based on the prompt and device operation information to the user via a display unit. Thus, this embodiment provides a novel mechanism for supporting device operation in an interactive manner, for example, in natural language. Furthermore, the present invention provides interactive support for user operation when remotely operating an MFP device from an information processing device, which is an external device. This eliminates the need for the user to read a manual, providing a more user-friendly operation system.

[0064] Second Embodiment Processing Flow of PC 400 A second embodiment of the present invention will be described below. This embodiment is an example that includes processing for checking the setting status of the MFP 100. The processing flow of the PC 400 of this embodiment will be described with reference to FIGS. 13A to 13C. The processing described below is realized, for example, by the CPU 401 reading a program stored in the ROM 402 or storage 403 into the RAM 404 and executing it. To simplify the description below, various programs executed by the CPU 401 may be described as the subject of operations. Note that the same step numbers are used for controls similar to those in the flowcharts of FIGS. 6A and 6B, and descriptions thereof will be omitted.

[0065] If it is determined in S612 that there is no API (in this embodiment, a setting confirmation API), the agent program 413 displays on the display unit 410 in S1301 that there is no corresponding API, presents a response to the user, and terminates the processing of this flowchart. Also, if it is determined in S613 that there is no MFP connected, the process proceeds to S1302. In S1302, the agent program 413 displays on the display unit 410 that the API cannot be executed because there is no MFP connected, presents only the response from the language model server 300 to the user, and terminates the processing of this flowchart. On the other hand, if it is determined that there is a connection, the process proceeds to S614. In S614, based on the response from the language model server 300, the API generation / execution program 415 generates a setting confirmation API that executes the API execution program 218 of the MFP 100, and displays it on the display unit 410 as a candidate for execution.

[0066] In S616, the agent program 413 determines whether or not the user has instructed the MFP 100 to execute the setting confirmation API. If the user has instructed the MFP 100 to execute the setting confirmation API, the process proceeds to S1303, where the API generation / execution program 415 transmits the setting confirmation API to the MFP 100 via the network 500. On the other hand, if the user has not instructed the MFP 100 to execute the setting confirmation API, the process of this flowchart ends.

[0067] In S1303, the agent program 413 waits for a response from the MFP 100 to the setting confirmation API sent in S617. When a response arrives, the process proceeds to S1304, where the agent program 413 accepts the response from the MFP 100, the CPU 401 displays the response on the display unit 410, and the process of this flowchart ends.

[0068] <Screen Example> An example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 in this embodiment will be described with reference to Fig. 14. Components similar to those of the screen 800 shown in Fig. 8 are assigned the same numbers and will not be described again.

[0069] 14 includes the same components as those of screen 800, as well as a setting confirmation item field 1401 for MFP 100, a setting confirmation item execution button 1402, and a setting confirmation field 1403. The setting confirmation item field 1401 displays the items currently set by MFP 100. The setting confirmation item field 1401 is generated by API generation / execution program 415 from a response from language model server 300, and displays items for which MFP 100 settings can be confirmed via network 500 in an easy-to-understand manner for the user as candidates. When the user operates setting confirmation item execution button 1402, the API generation / execution program 415 executes a confirmation command to MFP 100, and displays the confirmation response results in setting confirmation field 1403.

[0070] 15A and 15B , an example of a response by the language model server 300 to a prompt sent by the PC 400 according to this embodiment will be described. The same components as those in the example of the response shown in Fig. 11 are assigned the same numbers, and their descriptions will be omitted. In this embodiment, in addition to the example of the response (description 1101) shown in Fig. 11 , a confirmation API description 1501, a confirmation API 1502, and a return value description 1503 are added.

[0071] Confirmation API description 1501 is a description of a confirmation API 1502 for confirming the settings of the MFP, and is displayed in the setting confirmation item field 1401. The confirmation API 1502 is used by the API generation / execution program 415 of the PC 400 to generate an API that can be executed on the MFP 100 in accordance with this description. Return value description 1503 indicates a setting confirmation value received as a result of the API generation / execution program 415 executing the confirmation API 1502 on the MFP 100. The agent program 413 references the return value description 1503 and displays it on the display unit 410 as in the setting confirmation field 1403.

[0072] A confirmation API description 1504 is a description of a confirmation API 1505 for confirming the settings of the MFP. The confirmation API 1505 is used by the API generation / execution program 415 of the PC 400 to generate an API that can be executed on the MFP 100 in accordance with this description. A return value description 1506 indicates a setting confirmation value received as a result of the API generation / execution program 415 executing the confirmation API 1502 on the MFP 100. The agent program 413 refers to the return value description 1506 and displays it on the display unit 410 as in the setting confirmation field 1403.

[0073] As described above, the computer program according to this embodiment causes the computer of the information processing device to further function to acquire status information of a specified device from the specified device. Furthermore, the computer program also causes the computer to function to provide the device status information to a user via a display unit. This allows the user to easily check the current status of the specified MFP.

[0074] <Third Embodiment> <Sequence> A third embodiment of the present invention will be described below. In the above embodiment, a form for supporting the setting operation of an MFP was described, but the use of the present invention is not limited to setting and can be applied to various operations. Therefore, in this embodiment, operation support when investigating the cause of an error or warning in the MFP 100 in response to a dialogue with the user will be described. With reference to FIG. 16, a sequence for investigating the cause of an error or warning in the MFP 100 in this embodiment will be described. The processes of S1601 to S1606 are the same as S501 to S506 in FIG. 5, and therefore description thereof will be omitted.

[0075] In S1607, the MFP 100 acquires the status information including the current error and setting status in response to the status request command received in S1606, and returns the information to the PC 400 in S1608. In S1609, the agent program 413 selects data for the specified MFP type from the list of MFP models in the data storage 414 based on the error or warning investigation request made in S1604, and searches for similar information from within that data. For example, if the setting request is "FAX transmission not possible," the search here searches for items that match "FAX" and "transmission not possible."

[0076] Next, in S1610, the agent program 413 generates a prompt to query the language model server 300, and in S1611, sends the generated prompt to the language model server 300. In S1612, the language model program 308 of the language model server 300 receives the prompt generated in S1610 via the communication connector 306 and the communication unit I / F 305, and generates a response. In S1613, the agent program 413 receives the response generated by the language model server 300, and in S1614 generates a screen for the user showing the cause of the error or warning and how to deal with it, based on the received response, and displays the screen on the display unit 410 in S1615.

[0077] 17, an example of a screen displayed on the display unit 410 by the agent program 413 of the PC 400 according to this embodiment will be described. A screen 1700 includes an MFP selection item 1701, a prompt input item 1702, a prompt send button 1703, and an answer field 1704.

[0078] The MFP selection item 1701 is a selection item for the user to select a driver list to be operated by the user. The user selects an MFP in which a problem is occurring from the selection options. When an MFP is selected, the agent program 413 acquires information such as errors and warnings from the selected MFP.

[0079] The prompt input item 1702 is an item in which the user enters problems occurring with the MFP 100 using natural language. When the prompt send button 1703 is operated, the agent program 413 selects data for the MFP set in the MFP selection item 1701 from the data storage 414 and searches for items related to the MFP 100 error information and the problem entered by the user from among the data. Any known search method may be used. The agent program 413 creates a request to generate an API along with the search results and the problem with the MFP 100 entered by the user in the prompt input item 1702, and sends this request to the language model server 300. The answer field 1704 is an area in which an answer from the language model server 300 is displayed.

[0080] As described above, in the computer program according to this embodiment, the request relates to operations for investigating the cause of a device error or warning, and the response to the request includes a method of dealing with the error or warning. In this way, according to this embodiment, the user can be effectively supported not only with regard to operation settings, but also with regard to operation methods or methods of dealing with errors, etc.

[0081] The present invention can also be realized by a process in which a program that realizes one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

[0082] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0083] This application claims priority based on Japanese Patent Application No. 2024-009604, filed January 25, 2024, the entire contents of which are incorporated herein by reference.

[0084] 100: MFP, 200: Internet, 300: Language model server, 400: PC, 500: Network

Claims

1. A computer program, characterized in that the computer is caused to function as: a reception means for receiving a request in natural language from a user, the request being related to an operation of a device that specifies the device; an input means for inputting a prompt based on the request and operation information of the device to a language model server; and a provision means for providing, via a display unit, an answer generated by the language model server based on the prompt and the operation information of the device to the user.

2. The computer program according to claim 1, characterized in that the computer is further caused to function as a search means for searching, from a storage means that holds operation information of the specified device, operation information related to the request.

3. The computer program according to claim 2, characterized in that the storage means stores, for each operation item, as the operation information, an operation method of the device and its description, and identification information of an API that executes the operation method.

4. The computer program according to claim 3, characterized in that the answer includes information indicating an operation method related to the request and identification information of one or more APIs for executing a predetermined process related to the request on the device.

5. The computer program according to claim 4, characterized in that the provision means provides the one or more APIs to be selectable on the display unit.

6. The computer program according to claim 5, characterized in that the provision means provides information to the user by displaying an interactive screen on the display unit, and the interactive screen includes: an area for inputting the request in natural language from the user and an object for executing the request; an area for displaying the answer; and an area for displaying the one or more APIs and an object for selecting the one or more APIs.

7. The computer program according to claim 6, characterized in that the computer is further caused to function as an execution means for executing an API selected according to a user operation from among the one or more provided APIs.

8. Further cause the computer to function as acquisition means for acquiring state information of a specified device from the device, and the providing means further provides the state information of the device to the user via the display unit. The computer program according to any one of claims 1 to 7.

9. The request is related to an operation when investigating the cause of an error or warning of the device, and the response includes a countermeasure method for the error or warning. The computer program according to claim 8.

10. When the request is a setting operation on the device, the response further includes identification information of one or more APIs for checking the setting state of the device for the setting item corresponding to the request, and the providing means further provides the one or more APIs to be selectable on the display unit. The computer program according to any one of claims 1 to 7.

11. An information processing apparatus comprising: reception means for receiving a request in natural language from a user, the request being related to an operation of a specified device; input means for inputting a prompt based on the request and operation information of the device to a language model server; and providing means for providing, via a display unit, a response generated by the language model server based on the prompt and the operation information of the device to the user.

12. An information processing method comprising: a reception step of receiving, as a request in natural language from a user, the request related to an operation of a specified device; an input step of inputting a prompt based on the request and operation information of the device to a language model server; and a providing step of providing, via a display unit, a response generated by the language model server based on the prompt and the operation information of the device to the user.

Images