System, terminal device, information processing device, control method, and program

Abstract

For example, one of the objectives is to provide a system that can appropriately utilize large-scale language models and accept input from information processing devices, etc. [Solution] In System 1, the search server device 20 obtains information about the image forming apparatus 10, such as FAQs and manuals, from the web server device 40, and stores the obtained information about the image forming apparatus 10 in a memory unit or the like. The search server device 20 also receives a question as input text from the terminal device 30, generates a prompt based on the received input text, sends the generated prompt to the chat server device 50, and sends the answer text received from the chat server device 50 to the terminal device 30.

Description

Technical Field

[0001] The present disclosure relates to a system and the like.

Background Art

[0002] For example, as shown in Patent Document 1, when an error occurs in an image forming apparatus, a system is disclosed in which by reading a two-dimensional code displayed on the operation screen of the image forming apparatus into a terminal device, explanatory information corresponding to the error is displayed on the display screen of the terminal device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] One of the objects of the present disclosure is to provide a system and the like that can appropriately utilize a large language model and can receive an input from an information processing apparatus and the like.

Means for Solving the Problems

[0005] The system of the present disclosure includes a model information acquisition unit that acquires model information of an information processing apparatus, a reception unit that receives an input sentence, a generation unit that generates a prompt when the input sentence is related to the model information, and an output unit that outputs a response sentence obtained from a large language model based on the prompt.

[0006] The system of this disclosure includes a model information acquisition unit that acquires model information of an information processing device, a reception unit that receives an input sentence, a generation unit that generates a prompt based on the input sentence, and an output unit that outputs a response sentence and a notification obtained from a large-scale language model based on the prompt. The output unit outputs the response sentence if the input sentence is related to the model information, and does not output the response sentence or outputs a notification that it does not output a response sentence for the input sentence if the input sentence is not related to the model information.

[0007] The system of this disclosure is a system including an information processing device, a terminal device, and a server device, wherein the information processing device transmits model information of the information processing device to the server device, the terminal device obtains connection information from the information processing device for connecting to the server device, the terminal device establishes a communication path with the server device based on the connection information, the terminal device receives an input sentence and transmits it to the server device, the server device generates a prompt based on the input sentence, the server device transmits the prompt to a large-scale language model and obtains a response sentence from the large-scale language model, the server device transmits the response sentence to the terminal device, the terminal device displays the response sentence if it is related to the model information, and if it is not related to the model information, it suppresses the display of the response sentence related to the model information or outputs a notification that it will not display a response sentence for the input sentence.

[0008] The control method of this disclosure includes a model information acquisition step for acquiring model information of an information processing device; a reception step for receiving an input sentence; a generation step for generating a prompt when the input sentence is related to the model information; and an output step for outputting a response sentence obtained from a large-scale language model based on the prompt.

[0009] The program disclosed herein implements a model information acquisition function that acquires model information of an information processing device from a computer, a reception function that accepts an input sentence, a generation function that generates a prompt when the input sentence is related to the model information, and an output function that outputs a response sentence obtained from a large-scale language model based on the prompt.

[0010] The terminal device disclosed herein is a terminal device that displays a response to a user's input text in a conversational format based on model information acquired from an information processing device, and includes a storage unit for storing the model information, The system includes an input unit that receives the input text from the user, and a display unit that displays the input text in conversational format and the corresponding response text. The display unit displays the response text if the input text is related to the model information, and does not display the response text or displays a notification that no response text will be displayed for the input text if the input text is not related to the model information.

[0011] The information processing device disclosed herein is an information processing device that displays a user's input sentence and a response sentence in conversational format, comprising: a storage unit for storing model information; an input unit for receiving the input sentence from the user; and a display unit for displaying the input sentence in conversational format and the response sentence corresponding to the input sentence, wherein the display unit displays the response sentence when the input sentence is related to the model information, and does not display the response sentence or displays a notification that a response sentence will not be displayed for the input sentence when the input sentence is not related to the model information. [Effects of the Invention]

[0012] According to this disclosure, for example, it becomes possible to provide a system that can accept input from an information processing device, etc., and that can appropriately utilize a large-scale language model. [Brief explanation of the drawing]

[0013] [Figure 1] This is a diagram illustrating the system overview in the first embodiment. [Figure 2] This is a diagram for explaining the hardware configuration of the system in the first embodiment. [Figure 3] This is a diagram for explaining the hardware configuration of the system in the first embodiment. [Figure 4] This is a diagram for explaining the software configuration of the system in the first embodiment. [Figure 5] This is a diagram for explaining the software configuration of the system in the first embodiment. [Figure 6] This is a diagram for explaining (a) an example of model information, (b) an example of a prompt, and (c) an example of session information in the first embodiment. [Figure 7] This is a diagram for explaining the processing flow in the first embodiment. [Figure 8] This is a diagram for explaining the processing flow in the first embodiment. [Figure 9] This is a diagram for explaining the processing flow in the first embodiment. [Figure 10] This is a diagram for explaining the processing flow in the first embodiment. [Figure 11] This is a diagram for explaining an operation example in the first embodiment. [Figure 12] This is a diagram for explaining an operation example in the first embodiment. [Figure 13] This is a diagram for explaining the software configuration of the system in the second embodiment. [Figure 14] This is a diagram for explaining the processing flow in the second embodiment. [Figure 15] This is a diagram for explaining the token output processing flow in the second embodiment. [Figure 16] This is a diagram for explaining an operation example in the second embodiment. [Figure 17] This is a diagram for explaining an operation example in the second embodiment. [Figure 18] This is a diagram for explaining the processing flow in the third embodiment. [Figure 19]It is a diagram for explaining the flow of notification processing in the third embodiment. [Figure 20] It is a diagram for explaining an operation example in the third embodiment.

Embodiments for Carrying Out the Invention

[0014] Generally, when a user does not know how to operate an information processing device or the like, or when a problem occurs, the user inputs model information and search words into a database such as FAQ (Frequently Asked Questions) or a manual from the manufacturer's website and performs a search. However, it has been very difficult to present accurate information to the user by simply extracting the operation instructions required by the user based on the input words. In particular, when the target device is a multifunction device or the like, there are many technical terms in the search words, so there is also a problem that it is difficult for general users to come up with appropriate search words.

[0015] Therefore, a method of displaying explanatory information corresponding only when an event such as a problem occurs to the user is known, even if the user cannot come up with a search word. However, in this case, since the explanatory information such as a manual is displayed only when an event occurs, the user cannot freely search using search words when they do not know how to operate an information processing device or the like. Furthermore, when presenting the content of the manual to the user as it is, there is also a problem that it places a burden on the user until they reach the operation instructions they really need.

[0016] To alleviate such user burdens, methods such as Retrieval Augmented Generation (RAG) are known, which involve feeding database information into Large Language Models (LLMs) and obtaining appropriate answers for users based on that information. However, there have been challenges in properly utilizing large language models, such as incurring costs or straining server resources, if, for example, a user inputs information processing equipment other than instructions for operation.

[0017] A system that solves these problems will be described below with reference to the following embodiments. Note that the following embodiments illustrate the invention described in the claims as an example, and the technical scope of this disclosure is not limited to the embodiments described below.

[0018] [1. First Embodiment] [1.1 System Overview] The first embodiment will be described below. In the first embodiment, the following will be described as an example. In the following embodiment, the case in which the information processing device is applied to an image forming apparatus will be described, but it can also be applied to other apparatuses.

[0019] Figure 1 is a diagram illustrating the overview of System 1. System 1 has an image forming apparatus 10 and is connected to a network (NW). The image forming apparatus 10 is, for example, a device called a multifunction printer or an MFP (Multifunction Peripheral / Printer / Product). For example, when the image forming apparatus 10 executes a job (print job), it can form an image on paper, which is a recording medium. The image forming apparatus 10 can execute jobs for multiple processes, such as copying, faxing, scanning, and printing.

[0020] Furthermore, System 1 has a search server device 20. The search server device 20 is connected to a network NW, for example. The search server device 20 is, for example, a server device that can receive a question as input text from a terminal device 30 and generate a prompt based on the received input text. The search server device 20 can send the generated prompt to a chat server device 50 and send the answer text received from the chat server device 50 to the terminal device 30.

[0021] In this embodiment, the input sentence refers to a sentence based on a question, intended to provide the user with the information they need to resolve issues such as not knowing how to operate the image forming apparatus 10 or encountering a malfunction. The answer sentence is a sentence that responds to the question. These sentences are sometimes referred to as messages. In addition to answer sentences, messages may also include, for example, sentences (messages) that inform the user. Here, the sentences (messages) may be input or displayed via a chat application on the terminal device 30, but for example, the control unit may input sentences via voice input or output sentences via voice output.

[0022] The search server device 20 can obtain information about the image forming apparatus 10, such as FAQs and manuals, from the web server device 40, for example. The search server device 20 can store the obtained information about the image forming apparatus 10 in a memory unit, for example.

[0023] Furthermore, System 1 includes a terminal device 30. The terminal device 30 is connected to a network NW, for example. The terminal device 30 can receive questions from users regarding the image forming apparatus 10 using, for example, a chat application. The terminal device 30 can send the questions entered by the user as input text to the search server device 20 and display the answer text received from the search server device 20.

[0024] Furthermore, System 1 may have a web server device 40. The web server device 40 is, for example, connected to a network NW. The web server device 40 is, for example, a server device that stores information about the image forming apparatus 10, such as FAQs and manuals, in a state that can be viewed externally through web services or the like.

[0025] Furthermore, System 1 has a chat server device 50. The chat server device 50 is connected to a network NW, for example. The chat server device 50 is also a server device and may store a large-scale language model 52 as a language model.

[0026] The large-scale language model 52 is one type of language model, commonly known as an LLM. The large-scale language model 22 may be prepared by the service provider building System 1, or an external service may be used. For example, large-scale language models such as GPT® (Generative Pre-trained Transformers, GPT-3, GPT-4, GPT-4o), PaLM® (Pathways Language Model), LLaMA (Large Language Model Meta AI), and tsuzumi® may be used.

[0027] Furthermore, the search server device 20 may store a language model different from the large-scale language model 52, for example, a small language model (SLM) as an edge LLM. That is, a language model with a large amount of training data (LLM) may be stored in the chat server device 50, while a language model with less training data than the large-scale language model 52 (SLM) may be stored in the search server device 20 or other devices. In the following specification, the language model stored in the search server device 20 may be referred to as an edge language model.

[0028] The image forming apparatus 10 may also be an information processing device such as a smartphone or tablet, in addition to an image forming apparatus. It may also be a home appliance equipped with IoT functionality (e.g., an air conditioner, refrigerator, television, etc.), or a portable device or an in-vehicle device (e.g., a car navigation system, etc.).

[0029] [1.2 Hardware Configuration] [1.2.1 Common Configuration] Figures 2 and 3 illustrate the hardware configuration of each device in this embodiment. The image forming apparatus 10, search server device 20, terminal device 30, and web server device 40 each include at least a control unit, a storage unit (ROM: Read Only Memory, RAM: Random Access Memory), and a communication unit. In addition, each device has general functions as needed.

[0030] Here, the control unit, storage, ROM, RAM, display unit, operation unit, and communication unit of each device will be described below.

[0031] The control units (for example, control unit 100, control unit 200, control unit 300, control unit 400) are functional units for controlling the entire system. The control units implement various functions by reading and executing various programs stored in storage or ROM. The control units may be implemented by one or more control devices / arithmetic units (CPU (Central Processing Unit), SoC (System on a Chip)).

[0032] Storage (for example, storage 110, storage 210, storage 310, storage 410) is a non-volatile storage device capable of storing programs and data. For example, it may consist of storage devices such as HDDs (Hard Disk Drives) or SSDs (Solid State Drives). Alternatively, storage may be configured as an externally connectable USB (Universal Serial Bus) memory device. Furthermore, storage may be, for example, a storage area located in the cloud.

[0033] ROM (for example, ROM120, ROM220, ROM320, ROM420) is a non-volatile memory that can retain programs and data even when the power is turned off. ROM may store, for example, the firmware or initial applications of each device.

[0034] RAM (for example, RAM130, RAM230, RAM330, RAM430) is the main memory primarily used by the control unit of each device during execution. RAM is a rewritable memory that temporarily holds data including programs read from the storage or ROM of each device, as well as execution results.

[0035] The display unit (for example, display unit 140, display unit 340) is a display device capable of displaying various information and various screens such as execution screens. The display unit may be a display device such as a liquid crystal display (LCD), an organic electroluminescence (OLED) display, or an electrophoretic display. The display unit also includes an interface to which a display device can be connected. For example, it may consist of an external display device connected via HDMI® (High-Definition Multimedia Interface), DVI (Digital Visual Interface), or DisplayPort.

[0036] The operation unit (for example, operation unit 150, operation unit 350) is an operating device capable of receiving operation input from a user administrator or system administrator. For example, it may be a software key displayed on a touch panel integrated with the display unit, or a hardware key such as an operation button. Alternatively, the operation unit may be an operating device such as a keyboard or mouse. Furthermore, the operation unit may include an interface (for example, USB) to which an operating device can be connected. For example, each device may connect different operating devices (keyboards, mice, etc.) via the interface.

[0037] The communication units (for example, communication units 170, 270, 370, and 470) are communication interfaces that communicate with other devices. For example, a communication unit may be a network interface that can connect to a wireless LAN (Local Area Network), or a network interface that can be connected via a wired Ethernet (registered trademark). Alternatively, a communication unit may have, for example, a near-field communication device and be able to communicate via NFC (Near Field Communication) through the near-field communication device. Furthermore, a communication unit may be a communication device that can connect to a mobile communication network such as LTE (Long Term Evolution) / 4G / 5G / 6G.

[0038] [1.2.2 Image forming apparatus] Next, with reference to Figure 2, the hardware configuration of the image forming apparatus 10 will be described.

[0039] The image forming unit 160 forms an image, for example, on recording paper. The image forming unit 160 includes, for example, an image carrier, forms a toner image on the image carrier, and forms an image by transferring the toner image on the image carrier onto the recording paper. The image forming unit 160 may be configured as an image forming device such as a printer. Alternatively, the image forming unit 160 may form an image electronically as an image file.

[0040] The image reading unit 165 reads the original document (image) and outputs it as image data. The image reading unit 165 may be, for example, a scanner, or a reading device utilizing a CCD (Charge Coupled Device), CIS (Contact Image Sensor), or CMOS (Complementary Metal-Oxide-Semiconductor) image sensor.

[0041] [1.2.3 Terminal Devices] Next, with reference to Figure 3, the hardware configuration of the terminal device 30 will be described.

[0042] The imaging unit 380 reads the image and outputs it as image data. The imaging unit 380 is, for example, a camera, and may be an imaging device that utilizes a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) image sensor. In this embodiment, the terminal device 30 has an imaging unit 380, but for example, the terminal device 30 may not have an imaging unit 380, and an external camera device or the like may be connected to it.

[0043] The configuration of the chat server device 50 is the same as that of the search server device 20 and the web server device 40, so a detailed explanation will be omitted. The chat server device 50 includes, for example, a control unit, a storage unit (storage, ROM, RAM), and a communication unit.

[0044] [1.3 Software Configuration] The main software configurations for each device included in System 1 will be explained with reference to Figures 4 and 5. Figure 4(a) shows the software configuration of the image forming apparatus 10, and Figure 4(b) explains the main software configuration of the search server device 20. Figure 5(a) shows the software configuration of the terminal device 30, and Figure 5(b) explains the main software configuration of the web server device 40. Note that the software configurations shown in Figures 4 and 5 mainly describe the configurations necessary for this embodiment, and other configurations are omitted.

[0045] [1.3.1 Image forming apparatus] First, the main software configuration of the image forming apparatus 10 will be described. For example, the control unit 100 can realize the token generation unit 1010 and the code generation unit 1020 by reading and executing programs stored in the memory units (storage 110, ROM 120).

[0046] The token generation unit 1010 generates a token for, for example, a terminal device 30 to use the search server device 20. Here, the token is, for example, an access token containing connection information for the terminal device 30 to establish communication with the search server device 20. The access token may be a self-contained access token such as a JWT (JSON Web Token). The token generated by the token generation unit 1010 may be encrypted and output to, for example, the code generation unit 1020.

[0047] The code generation unit 1020 may generate a two-dimensional code that can be read by, for example, the imaging unit 180, as identification information for outputting the token generated by the token generation unit 1010 to the terminal device 30. The two-dimensional code may also include the connection destination (for example, URL) of the search server device 20 in addition to the token.

[0048] Here, "code" refers to encoded information. Encoded information may be a graphic or a string (where a string is a string of one or more characters, and characters may include one or more numbers, English letters, kana, hiragana, kanji, symbols, etc.). When the encoded information is a graphic, it may be a one-dimensional code such as a barcode (e.g., EAN code, JAN code, Cobabar, CODE128, etc.) or a two-dimensional code. As for two-dimensional codes, they may be stack-type two-dimensional codes (e.g., PDF417, CODE49, etc.) or matrix-type two-dimensional codes (e.g., Quick Response Code (QR Code®), DataMatrix, VeriCode, Aztec, etc.). In this embodiment, the Quick Response Code (QR Code) will be used as an example.

[0049] In this embodiment, the identification information is read and acquired by the terminal device 30 as a QR code (2D code). However, for example, the image forming apparatus 10 may output identification information including a token via NFC, and the terminal device 30 may acquire the identification information.

[0050] Furthermore, the storage 110 reserves an area for storing the operation application 1110. The operation application is, for example, an application that controls the operation unit 150 of the image forming apparatus 10.

[0051] Furthermore, the storage 110 reserves an area for a model information storage area 1120 that stores model information. Model information includes, for example, the model name, serial number, and firmware version of the image forming apparatus 10. An example of model information stored in the model information storage area 1120 will be explained with reference to Figure 6(a). The model information stores an item (for example, "model name") and a value (for example, "AB123CD").

[0052] [1.3.2 Search Server Device] Next, the main software configuration of the search server device 20 will be described. For example, the control unit 200 can realize the knowledge information acquisition unit 2010, the search processing unit 2020, the token processing unit 2030, the chat processing unit 2040, and the conversation history acquisition unit 2050 by reading and executing programs stored in the memory units (storage 210, ROM 220).

[0053] The knowledge information acquisition unit 2010 acquires knowledge information from, for example, the web server device 40. Knowledge information includes, for example, manuals containing instructions on how to operate the image forming apparatus 10 and how to deal with malfunctions, as well as FAQ information. The knowledge information acquisition unit 2010 vectorizes the knowledge information using methods such as k-nearest neighbors, BoW (Bag of Words), TF-IDF (term frequency-inverse document frequency), and Word2vec, and stores it in the knowledge information DB 2110.

[0054] The search processing unit 2020 searches for the question received from the user in the knowledge information stored in, for example, the knowledge information DB 2110. The search processing unit 2020 vectorizes the question from the user in the same way as the knowledge information. Next, the search processing unit 2020 compares the vectorized question with the knowledge information and identifies the knowledge information whose vectors are similar as the knowledge information related to the question.

[0055] The search processing unit 2020 selects document information that corresponds to the user's question from the identified knowledge information. Document information refers to information used to answer the question received from the user, such as knowledge information related to the question.

[0056] In this case, if the search processing unit 2020 cannot narrow down the document information to a single item, or if the user gives an unintended answer to the first question, it may narrow down the document information in the following way, for example:

[0057] As a first method for narrowing down document information, the search processing unit 2020 may select one or more similar knowledge information from the knowledge information DB 2110 as document information, for example, selecting the top four as document information. The search processing unit 2020 may also display a screen on the display unit 340 of the terminal device 30, indicating that there are multiple or different answers, and have the user input an answer to narrow down the results.

[0058] The search processing unit 2020 may use the conversation history as a second method for narrowing down document information, if the conversation history is stored in the conversation history storage area 2130. The conversation history refers to a combination of questions and answers to past questions, specifically a combination of input sentences and answer sentences.

[0059] The search processing unit 2020 outputs the generated document information to the chat processing unit 2040.

[0060] The token processing unit 2030 performs the following operations: decrypts the token obtained from the terminal device 30 if it is encrypted, reads the decrypted token, and retrieves the connection information contained in the token.

[0061] The chat processing unit 2040 outputs the input text to the large-scale language model 52 and retrieves the response text from the large-scale language model 52. This allows the chat processing unit 2040 to output a response text for the input text.

[0062] Furthermore, the chat processing unit 2040 may have a message output function that can output a message other than the response text obtained from the large-scale language model 52 in place of the response text in response to the input text received from the terminal device 30 via a chat application or the like. The chat processing unit 2040 may output messages other than the response text by obtaining them from, for example, a message storage area reserved in the storage 210. Also, the chat processing unit 2040 may have, for example, a message generation unit capable of generating messages and output the generated messages. Note that the message output unit may have a different configuration from the chat processing unit 2040.

[0063] To describe the chat processing unit 2040 in more detail, it includes a prompt generation unit 2042 and a response text acquisition unit 2044.

[0064] The prompt generation unit 2042 generates prompts to be input to the language model (e.g., the large-scale language model 52) based on the input information. A "prompt" is information that instructs the language model (large-scale language model, small-scale language model) to generate a response sentence.

[0065] The prompt may be in natural language format or in a predetermined format that utilizes tags, etc. The prompt generation unit 2042 generates a prompt based on information such as the input text and document information.

[0066] Figure 6(b) schematically shows an example of a prompt for input to a language model (e.g., a large-scale language model 52) as natural language.

[0067] A prompt preferably includes an input statement and document information. For example, Figure 6(b) includes "How to register in the address book" as an input statement based on a question received from the user, and "When registering a fax destination in the address book, ..." as document information entered from the search processing unit 2020. A prompt may also include one or more contents such as an input statement and document information. Furthermore, a prompt may include a version of that information that has been optimized and reconstructed by the edge LLM.

[0068] The prompt generation unit 2042 may, for example, input the above-mentioned information to the edge LLM once to summarize and generate prompts to be input to the large-scale language model 52.

[0069] The chat processing unit 2040 (prompt generation unit 2042) inputs a prompt containing the input text as an example of a language model to the large-scale language model 52 of the chat server device 50. The large-scale language model 52 outputs a response text to the chat processing unit 2040 according to the input prompt. The large-scale language model 52 is based on the one in the chat server device 50, but if it is stored in the storage 210, for example, the control unit 200 can use the one stored in the storage 210. The control unit 200 may also choose whether to use the large-scale language model 52 or the language model stored in the storage 210 depending on the content of the question and the content of the response text.

[0070] The chat processing unit 2040 (response text acquisition unit 2044) acquires the response text from the large-scale language model 52. The chat processing unit 2040 outputs, for example, the response text to the input text included in the prompt to the terminal device 30.

[0071] The conversation history acquisition unit 2050 stores the conversation history as conversation history information in the conversation history storage area 2130.

[0072] The storage 210 of the search server device 20 reserves an area for the knowledge information DB 2110, which stores knowledge information. The knowledge information DB is, for example, a database that stores manuals and FAQ information for the image forming apparatus 10, and has a manual storage area 2112 for storing the manual for the image forming apparatus 10 and an FAQ information storage area 2114 for storing FAQ information related to the image forming apparatus 10.

[0073] The storage 210 of the search server device 20 reserves an area for a session information storage area 2120 that stores session information. Session information is information that a terminal device 30 uses to start a session with, for example, the search server device 20. A session refers to the establishment of a communication path between the terminal device 30 and the search server device 20, for example, by constructing a WebSocket and performing a series of communications with an application. An example of session information stored in the session information storage area 2120 will be explained with reference to Figure 6(c). The session information stores a device ID (for example, "AB123CD-0123456") and a session ID (for example, "12345").

[0074] Here, the device ID is information that identifies the model that accepts user inquiries, and may include model information (model name, serial number, etc.). Here, the session ID is connection information for starting a session to ask a question about the corresponding device ID, and stores, for example, an arbitrary value that does not overlap with other session IDs, and "Not set" if there is no session corresponding to the corresponding device ID. In addition, the session information may include additional information other than the device ID and session ID, such as the firmware version corresponding to the device ID.

[0075] The storage 210 of the search server device 20 reserves an area for a conversation history storage area 2130 that stores the conversation history. The conversation history storage area 2130 stores combinations of input sentences and answer sentences as conversation history, which are combinations of questions and answers to past questions.

[0076] [1.3.3 Terminal Devices] Next, the main software configuration of the terminal device 30 will be described. For example, the control unit 300 can realize the code processing unit 3010 by reading and executing a program stored in the memory unit (storage 310, ROM 320).

[0077] The code processing unit 3010 performs a process to obtain tokens, etc., from a code displayed on the display unit 140 by the image forming apparatus 10. The code processing unit 3010 decodes a two-dimensional code read by the imaging unit 380, etc., to obtain a token. Here, the code processing unit 3010 may also obtain the connection destination (e.g., URL) of the search server device 20 in addition to the token. The code processing unit 3010 stores the obtained token in the token storage area 3130.

[0078] Furthermore, the storage 310 reserves an area for storing a chat application 3110 that accepts input text as a question from the user and displays an answer to the question from the user, and a camera application 3120 that operates the shooting unit 380. In addition, the storage 310 reserves an area for a token storage area 3130 that stores tokens acquired by the code processing unit 3010.

[0079] [1.3.4 Web Server Device] Furthermore, the main software configuration of the web server device 40 will be described. For example, the control unit 400 can realize the web service unit 4010 by reading and executing programs stored in the storage unit (storage 410, ROM 420).

[0080] The web service unit 4010 can, for example, provide web services to support users via a network. The web service unit 4010 transmits manuals and FAQ information for the image forming apparatus 10 to a search server device 20, for example, a knowledge information acquisition unit 2010.

[0081] Furthermore, the storage 410 has reserved areas for a manual storage area 4110 for storing the manual for the image forming apparatus 10, and an FAQ information storage area 4120 for storing FAQ information for the image forming apparatus 10.

[0082] [1.4 Processing Flow] The following describes the processing flow in this embodiment. Figures 7 to 10 show the processing flow of System 1 in this embodiment.

[0083] The following processing will be explained assuming that the control units (control unit 100, control unit 200, control unit 300) of each device (image forming apparatus 10, search server device 20, terminal device 30) are executed. However, each configuration described in Figure 4 or Figure 5 may also execute the processing for each step.

[0084] First, the control unit 100 displays an operation screen on the display unit 140 of the image forming apparatus 10, which includes a display that allows the user to select options, such as a help button (S102). Next, the control unit 100 determines whether the help button has been selected, and if the help button has been selected, it determines whether the image forming apparatus is registered to use this system (S104; Yes → S106). Meanwhile, the control unit 100 continues to determine whether the help button has been selected until the help button is selected (S104; No).

[0085] Next, if the image forming apparatus is not registered for use with this system, the control unit 100 performs a user registration process (S106; No → S108).

[0086] Here, as a method for determining whether or not this image forming apparatus is registered to use this system, for example, if a device ID is assigned to the image forming apparatus 10, for example, if it has received a device ID from the search server device 20 even once, it may be determined that it is registered to use this system.

[0087] The user registration process is, for example, the process of registering the image forming apparatus 10 with the search server device 20 and making it possible to receive user inquiries about the image forming apparatus 10. For example, the control unit 100 sends model information to the search server device 20, the control unit 200 of the search server device 20 generates a device ID based on the model information, and sends the device ID to the image forming apparatus 10 (S202).

[0088] Here, the model information includes, for example, at least the model name and the serial number. The control unit 200 may generate a device ID by, for example, combining the model name and serial number included in the model information, so that the image forming apparatus 10 for which the help button was selected can be identified.

[0089] Next, if the image forming apparatus is registered for use with this system, the control unit 100 sends the model information to the search server device 20 (S106; Yes → S110). The control unit 200 of the search server device 20 receives the model information (S204) and generates a session ID based on the model information (S206). Here, the control unit 200 may store the session ID in the session information storage area 2120, for example, by associating it with a device ID identified based on the received model information.

[0090] Next, the control unit 200 transmits the generated session ID to the image forming apparatus 10 (S208). When the control unit 100 of the image forming apparatus 10 receives the session ID (S112), it generates a token based on the device ID and the session ID (S114). The generated token may, for example, include at least the device ID and the session ID, and may also include an expiration date.

[0091] The control unit 100 outputs a generated token that can be acquired by the terminal device 30, for example, by displaying a two-dimensional code on the display unit 140 of the image forming apparatus 10, or by making it outputtable via NFC (S116).

[0092] The control unit 300 of the terminal device 30 acquires a token from the image forming apparatus 10, for example, using the imaging unit 380 of the terminal device 30 or NFC (S302). The control unit 300 connects to the search server device 20 via the network NW (S304) and transmits the token acquired in processing S302 (S306).

[0093] When the control unit 200 of the search server device 20 receives a token from the terminal device 30 (S210), it identifies the device ID and session ID from the connection information contained in the token (S212), and determines whether the identified device ID and session ID are valid or not (S214).

[0094] Here, as a method for determining whether the device ID and session ID are valid, for example, the device ID and session ID are compared with the session information storage area 2120, and if both the device ID and session ID stored in the session information storage area 2120 match, they are determined to be valid. The control unit 200 may also determine that the device ID and session ID are not valid if, for example, an expiration time has been set for the token and that expiration time has expired.

[0095] In this embodiment, a self-contained access token is used, but an opaque token may also be used. In that case, the control unit 200 transmits the connection information contained in the token obtained from the terminal device 30 to the image forming apparatus 10, and when it notifies the image forming apparatus 10 that it has generated the connection information, it determines that the device ID and session ID are valid.

[0096] If the control unit 200 determines that the device ID and session ID are valid, it establishes a web socket with the terminal device 30 and starts a session (S214; Yes → S216). At this point, the control unit 200 may invalidate the session ID for which the session has been started, for example, by deleting the corresponding session ID from the session information storage area 2120.

[0097] When a WebSocket is established with the search server device 20 and a session is started, the control unit 300 of the terminal device 30 launches the chat application 3110 and displays the chat application 3110 on the display unit 340 (S308).

[0098] On the other hand, if the control unit 200 determines that the device ID and session ID are invalid, it terminates the process (S214; No).

[0099] Next, the control unit 300 of the terminal device 30 receives input from the user in the launched chat application 3110 (S310). Here, the control unit 300 may, for example, display a display screen on the display unit 340 that accepts input from the user in question form. The control unit 300 then transmits the entered question from the display screen (S312).

[0100] Next, when the control unit 200 of the search server device 20 receives a question (S218), it determines whether the received question is a repeat question or not (S220). If it is not a repeat question, the control unit 200 reconstructs the question as is (S220; No → S222). If it is a repeat question, it reconstructs the question by referring to the conversation history stored in the conversation history storage area 2130 (S220; Yes → S224).

[0101] Here, the control unit 200 reconstructs the question by combining the input sentence, which is a question from the user, with information that identifies the image forming apparatus 10, such as model information, and the conversation history, etc. The control unit 200 may, for example, reconstruct the question using an edge language model. Next, the control unit 200 searches the knowledge information DB 2110 for candidate answers to the question based on the reconstructed question (S226).

[0102] If the control unit 200 has a candidate for answering the question, it generates a prompt from the search results (e.g., input text, document information) (S228; Yes → S230). Then, the control unit 200 sends the generated prompt to the large-scale language model 52 (S232).

[0103] The large-scale language model 52 receives a prompt (S402) and parses the received prompt (S404). Next, it creates a response sentence based on the parsed prompt (S406) and sends the response sentence to the search server device 20 (S408).

[0104] When the control unit 200 of the search server device 20 receives a response (S234), the response This is sent to the terminal device 30 (S236).

[0105] The control unit 300 of the terminal device 30 receives the response (S314), and if the response includes detailed information such as the destination of a manual or FAQ, it displays the details, along with any possible links, on the display unit 340 (S316). Next, the control unit 300 determines whether the details have been selected (S318).

[0106] If the control unit 300 determines that details have been selected, it will, for example, launch a web browser and connect to the web server device 40 to display the details on the display unit 340 (S318; Yes → S320). On the other hand, if details have not been selected, the control unit 300 will determine whether or not there will be a follow-up question (S318; No → S322).

[0107] The control unit 300, if there is a follow-up question, inputs the question again (S322; Yes → S310), or if there is no further question, it terminates the session and ends the process (S322; No).

[0108] On the other hand, if there are no candidates for answering the question, the control unit 200 determines whether the session has ended (S228; No → S240). The session may be determined to have ended, for example, when the control unit 200 receives a message from the terminal device 30 indicating the end of communication, or when there is no response from the terminal device 30 for a certain period of time. If the session has not ended, the control unit 200 accepts the question again (S240; No → S218), and if the session has ended, it terminates the process (S240; Yes).

[0109] The above-described operation flow is merely an example, and some processing may be performed differently. For example, if there are no candidate answers to a question, the control unit 200 may add processing to generate a prompt from the input sentence if the question is related to the image forming apparatus 10, such as by including words like "image forming apparatus" or "copy." In that case, the large-scale language model 52 would generate an answer sentence for the question using the general knowledge it possesses.

[0110] [1.5 Example of Operation] Figures 11 and 12 show examples of display screens, including messages, that are displayed to the user as a result of the operation of the image forming apparatus 10 and the terminal device 30 in this embodiment. Figures 11(a) and 11(b) show examples of the display screen of the image forming apparatus 10, and Figures 12(a), 12(b), and 12(c) show examples of the display screen (chat application 3110) of the terminal device 30.

[0111] Figure 11(a) shows an example of a display screen W100 displayed on the display unit 140 of the image forming apparatus 10. The display screen W100, as the home screen of the image forming apparatus 10, displays icons for selecting various functions such as simple copy, simple fax, and simple scan, as well as a HelpDesk button B100. When the user selects the HelpDesk button B100, the control unit 100 displays the display screen W102 shown in Figure 11(b) on the display unit 140.

[0112] In this embodiment, the help desk button B100 is provided on the home screen, but it may also be always displayed at the edge of the display screen, or it may be provided on the operation unit 150.

[0113] Figure 11(b) shows an example of a display screen W102 displayed on the display unit 140 of the image forming apparatus 10. The display screen W102 shows an area R100 that explains the usage procedure, which includes the message M100, "AI Help Desk: We will solve your problems. We will solve your problems in chat format from the mobile app," and an area R102 that displays information including a token as a QR code.

[0114] Other information obtainable from the QR code besides the token may include, for example, the URL of the server to which the search server device 20 is located. The server's URL may, for example, be pre-stored by the chat application 3110 on the terminal device 30.

[0115] Here, if the user reads the QR code displayed in area R102 with the camera of the terminal device 30, the control unit 300 displays the display screen W300 shown in Figure 12(a) on the display unit 340 of the terminal device 30.

[0116] Here, the token has an expiration date. For example, if the expiration date is set to 30 minutes, after 30 minutes have passed, even if the QR code is read, the display screen W300 shown in Figure 12(a) will not be displayed on the display unit 340 of the terminal device 30.

[0117] In this way, the image forming apparatus 10 is provided to the mobile terminal (terminal device 30) via a two-dimensional code or NFC to a token that allows it to start a session that is valid for a certain period of time. The search server device 20 can determine whether the session is valid or invalid and restrict the terminal device 30's access to the search service.

[0118] Figure 12(a) shows an example of a display screen W300 displayed on the display unit 340 of the terminal device 30. The display screen W300 shows an input form F300 for entering a question and a send button B300 for sending the question entered in the input form F300. The message M300, "Please enter your problem," is displayed in area R300 of the display screen W300.

[0119] Here, if the user enters, for example, "How to register in the address book" into the input form F300 where the user enters a question, and selects the send button B300, the control unit 300 displays the user's question, "How to register in the address book," as message 302 in area R300 of the display screen W300. Next, the control unit 300 continues to display the display screen W302 shown in Figure 12(b).

[0120] Figure 12(b) shows an example of a display screen W302 displayed on the display unit 340 of the terminal device 30. The display screen W302 shows an input form F300 and a send button B300. Furthermore, area R302 of the display screen W302 shows message M300, message M302, and message M304, which is a response to message M302.

[0121] If the user selects the link L300 included in message M304, the control unit 300 displays the display screen W304 shown in Figure 12(c) on the display unit 340. On the other hand, if the user enters a new question in the input form F300 and selects the send button B300, the control unit 300 displays the newly entered question in area R302 of the display screen W302 and displays the answer to the new question.

[0122] For new questions, the search server device 20 can improve accuracy by referring to the conversation history. The terminal device 30 can, for example, refer to messages M302 and M304 related to "how to register in the address book" as part of the conversation history and display an answer to the new question entered by the user.

[0123] Figure 12(c) shows an example of a display screen W304 displayed on the display unit 340 of the terminal device 30. Display screen W304 shows the section of the manual corresponding to the response message M306 in Figure 12(b).

[0124] In this way, since the question content can be entered in natural language, an appropriate answer can be output even if the entered search words do not exactly match the words in the database. Furthermore, based on the keywords entered in the terminal device 30, the search server device 20 performs a search using, for example, variations, on knowledge information such as model manuals and FAQs within the search server device 20. The search server device 20 can then use AI (for example, a large-scale language model) to summarize the results and provide the information to the user in an easy-to-understand format.

[0125] Furthermore, the token may include information such as which operation screen the HELP operation was performed on, representing the state of the image forming apparatus 10 when a HELP operation (such as selecting the help desk button B100) was performed. This allows the search server device 20 to narrow down the content of the question, thereby improving the accuracy of the answer displayed to the user.

[0126] On the other hand, suppose a question completely unrelated to the image forming apparatus 10, such as "Please tell me how to get to Station A," is entered on the display screen W300 in Figure 12(a). In this case, the control unit 200 of the search server device will exclude (block) the generation of prompts for questions completely unrelated to the image forming apparatus 10. As a result, the terminal device 30 will only display the answers to questions related to the image forming apparatus 10.

[0127] Furthermore, suppose the user closes, for example, the display screen W302 in Figure 12(b) or the display screen W304 in Figure 12(c), and ends the session. If the user then tries to start the chat application 3110 again, the search server device 20 restricts the terminal device 30's access to the search service because the token is limited to one session.

[0128] Furthermore, since the token contains model information that identifies the image forming apparatus 10, even if the operating method differs depending on the model of the image forming apparatus, it is possible to display an appropriate response corresponding to that model. In addition, the search server device 20 uses this model information to exclude (block) the generation of prompts based on questions that are completely unrelated to the image forming apparatus 10.

[0129] This eliminates the need for users to manually search for information by considering the model name of the image forming machine. Furthermore, the search server device 20 does not engage in unnecessary communication with, for example, the chat server device 50, thus saving communication and server resources. In addition, if communication with, for example, the chat server device 50 incurs costs, these costs can be saved, and the misuse of the search service can be prevented.

[0130] In this embodiment, no notification was given to the user if communication could not be established or if prompt generation was excluded. However, for example, a notification indicating that it is invalid may be displayed on the display unit 340 of the terminal device 30.

[0131] In this embodiment, the terminal device 30 stores the chat application 3110. In a different embodiment, for example, when the terminal device 30 takes a picture of the QR code displayed on the display unit 140 of the image forming apparatus 10, it may launch a web browser and start the chat application on the web browser.

[0132] [1.6 Effects, etc.] In this way, when a question is received by the terminal device 30 and is related to model information, the search server device 20 searches based on detailed databases for each model, such as various manuals and FAQs, and generates a prompt. As a result, the terminal device 30 displays detailed and appropriate operating instructions for each model as an answer to the question (input text) freely entered by the user, thereby reducing the burden on the user.

[0133] Furthermore, the search server device 20 blocks the generation of prompts based on questions completely unrelated to model information. Since the search server device 20 avoids unnecessary communication with, for example, the chat server device 50, it can save resources such as communication and server resources. Also, if communication with, for example, the chat server device 50 incurs costs, those costs can be saved.

[0134] [2. Second Embodiment] The second embodiment will now be described. The second embodiment is an embodiment in which a token is output by NFC (Near Field Communication).

[0135] The second embodiment will omit explanations of parts where the hardware and software configuration is the same as that of the first embodiment, and will focus on explaining the differences from the first embodiment.

[0136] [2.1 Software Configuration] Figure 13 shows the software configuration of the image forming apparatus 10.

[0137] [2.1.1 Image forming apparatus] The main software configuration of the image forming apparatus 10 will now be described. For example, the control unit 100 can realize the NFC unit 1030 by reading and executing a program stored in the storage unit (storage 110, ROM 120).

[0138] The NFC unit 1030 transmits and receives data via NFC connection through the NFC device of the communication unit 170. The NFC unit 1030 can, for example, transmit a token to the terminal device 30. The NFC unit 1030 may, for example, use the storage unit (storage 110, ROM 120) of the image forming apparatus 10 to establish an NFC connection, or the NFC device may have its own storage unit and transmit and receive data via NFC connection. After the NFC connection is completed, the NFC unit 1030 can erase the transmitted data stored in the storage unit of the NFC device.

[0139] [2.2 Processing Flow] The following describes the processing flow in this embodiment. Figure 14 is a diagram showing the processing flow of System 1 in this embodiment, and Figure 15 is a diagram showing the token output processing of the image forming apparatus 10.

[0140] [2.2.1 Main Processing Flow] Figure 14 is a replacement for Figure 7 of the first embodiment, in which S105 is executed between S104 and S106, and S118 is executed instead of S116. First, the control unit 100 displays, for example, a connection screen on the display unit 140 (S105). Here, the connection screen is a display screen that indicates that the system is transitioning to a waiting state for connection to the NFC device, for example, when registering the token generated in S114 to the storage unit of the NFC device.

[0141] Next, the control unit 100 performs token output processing (S118). Here, token output processing is the process by which the image forming apparatus 10 transmits the generated token to the terminal device 30, for example, via an NFC device provided in the communication unit 170.

[0142] [2.2.2 Token Output Processing Flow] First, the control unit 100 registers the token generated in S114 with the NFC device (S134). Here, the control unit 100 may temporarily store and register the token in a storage unit (storage 110, RAM 130), for example. Note that the token may be stored in a storage unit individually provided by the NFC device.

[0143] Next, the control unit 100 determines whether the NFC device has temporarily stored the token in its memory unit and registered the token (S136). If, for example, the token is registered in the memory unit of the NFC device, the control unit 100 displays a predetermined screen on the display unit 140 (S136; Yes → S138). Here, the predetermined screen refers to a display screen that displays instructions on how to operate the NFC connection with the terminal device 30. The predetermined screen may be, for example, a display screen that displays instructions for NFC connection with the terminal device 30 and includes a notification prompting the user to hold the terminal device 30 over a predetermined location. On the other hand, the control unit 100 continues to display the connection screen on the display unit 140 until the token is registered (S136; No).

[0144] Next, the control unit 100 determines whether the token registered with the NFC device is valid or not (S140). Here, as a method for determining whether the token is valid or not, for example, the control unit 100 may determine that the token has become invalid when the expiration date of the token stored in the memory unit of the NFC device has passed. Alternatively, for example, the control unit 100 may determine that the token has become invalid when the session corresponding to the token becomes invalid, or when a predetermined screen has been displayed for a certain period of time.

[0145] The control unit 100 determines whether the NFC device can connect to the terminal device 30 if the token is valid (S140; Yes → S142), and displays a service termination screen on the display unit 140 if the token is invalid (S140; No → S146). Here, the service termination screen is, for example, a display screen that notifies the user that the token has become unusable and the connection to the search service has been terminated.

[0146] If the NFC device and the terminal device 30 can connect (communicate) via NFC, the control unit 100 sends a token to the terminal device 30 and terminates the display of a predetermined screen to return to the original screen (S142; Yes → S144). Alternatively, the control unit 100 may continue to display a predetermined screen until the NFC device and the terminal device 30 can connect (communicate) via NFC (S142; No). Next, the control unit 100 erases the token from the storage unit of the NFC device (S148).

[0147] [2.3 Example of Operation] Figures 16 and 17 show examples of display screens, including messages, that are displayed to the user as a result of the operation of the image forming apparatus 10 and terminal device 30 in this embodiment.

[0148] Figure 16(a) shows an example of a display screen W120 displayed on the display unit 140 of the image forming apparatus 10. The display screen W120 is a home screen, which is an example of the user interface screen of the image forming apparatus 10, and displays icons for selecting various functions such as simple copy, simple fax, and simple scan, as well as an AI assistant button (first button) B120 and an AI assistant button (second button) B122. At least two buttons for switching to the AI ​​assistant are displayed on the home screen of the image forming apparatus 10. When the user selects either the first button B120 or the second button B122, the control unit 100 displays the display screen W122 shown in Figure 16(b) on the display unit 140.

[0149] Figure 16(b) shows an example of a display screen W122 that the control unit 100 displays on the display unit 140 as a "connecting" screen. The "connecting" screen is a display screen that is displayed on the display unit 140 while the image forming apparatus 10 is connecting with the search server device 20, completing token registration with the NFC device of the communication unit 170, and preparing to send the token to the terminal device 30 via NFC connection. In the display screen W122 of Figure 16(b), an image G120 indicating that the transition is in progress is displayed. Here, the control unit 100 displays the display screen W124 of Figure 17(a), which will be shown next, grayed out on the display screen W122. When the token registration with the NFC device is complete and preparations to send the token to the terminal device 30 via NFC connection are complete, the control unit 100 displays the display screen W124 of Figure 17(a) on the display unit 140.

[0150] Figure 17(a) shows an example of a display screen W124 that is displayed as a predetermined screen on the display unit 140 of the image forming apparatus 10. The display screen W124 in Figure 17(a) shows a message M120 that reads, "You can ask questions using your mobile device. Hold an NFC-compatible mobile device over the screen. Please enable NFC beforehand. This screen will automatically close after the connection is complete," an area R120 that explains the usage procedure, and a close button B124.

[0151] Here, when the user holds the terminal device 30 over the NFC device indicated in area R120, which explains the usage procedure, the control unit 100 transmits a token to the terminal device 30 using NFC. If the user then selects the close button B124, the control unit 100 terminates the display of W124 and displays the original screen, for example, the display screen W120 shown in Figure 16(a), on the display unit 140.

[0152] In this way, the designated screen displays instructions for connecting the terminal device 30 via NFC. The designated screen may also include a notification prompting the user to hold the terminal device 30 over a designated location. Furthermore, the designated screen may include a notification prompting the user to enable the NFC function of the terminal device 30 before holding it over the location. In addition, a notification may be displayed indicating that the display of the designated screen will end once the connection between the image forming apparatus 10 and the terminal device 30 is complete, or that the display will end when the connection between the image forming apparatus 10 and the terminal device 30 is complete.

[0153] Furthermore, the predetermined screen may include an external view of the device showing the position where the terminal device 30 should be held. In addition, the predetermined screen may also display a magnified view of a part of the image forming apparatus 10 along with a mark indicating the position for receiving NFC. The predetermined screen may also display a notification that questions can be asked via the terminal device 30, and a close button to exit the display of the predetermined screen. After transitioning to the predetermined screen, the first button B120 and the second button B122 are not displayed, and the close button B124 is displayed in the position of the second button B122.

[0154] Figure 17(b) shows an example of a display screen W126 that is displayed on the display unit 140 of the image forming apparatus 10 as a service termination screen when the token becomes invalid or the service cannot be continued for any reason. Display screen W126 displays the message M122, "This service has ended. Thank you for using our service," and a close button B126. When the user selects the close button B126, the control unit 100 terminates the display of W122 and displays the original screen, for example, display screen W120 of Figure 16(a), on the display unit 140.

[0155] Thus, the image forming apparatus 10 can transmit tokens not only as QR codes (two-dimensional codes) but also via NFC connection to the terminal device 30. The image forming apparatus 10 can display a predetermined screen to provide a user-friendly interface.

[0156] [3. Third Embodiment] The third embodiment will now be described. The third embodiment is one in which, if an inappropriate question unrelated to the image forming apparatus 10 is entered, the chat application displays a message refusing to answer the question and notifies the user.

[0157] The third embodiment will omit explanations of parts where the hardware and software configuration is the same as that of the first embodiment, and will focus on explaining the differences from the first embodiment.

[0158] [3.1 Processing Flow] The following describes the processing flow in this embodiment. Figure 18 is a diagram showing the processing flow of System 1 in this embodiment, and Figure 19 is a diagram showing the notification processing between the search server device 20 and the terminal device 30.

[0159] [3.1.1 Main Processing Flow] Figure 18 replaces Figure 9 of the first embodiment, and S219 is executed after S218 or S312.

[0160] The control unit 200 performs notification processing (S219). Here, notification processing is the process of displaying a screen containing a message informing the terminal device 30 that it is refusing to answer the question in response to the input text entered into the terminal device 30. In S218, the control unit 200 notifies that it is refusing to answer the question if the question received from the terminal device 30 is, for example, an inappropriate question unrelated to the image forming apparatus 10, or if a large number of questions have been sent. In this embodiment, the message includes the input text, the answer text, and notifications other than the input text and the answer text.

[0161] [3.1.2 Notification Processing Flow] First, the control unit 200 of the search server device 20 displays a processing wait indicator on the display unit 340 (S362). Here, the processing wait indicator is a message displayed while the search server device 20 is generating an answer after the user has entered a question (input) sentence, informing the user that the search server device 20 is processing the request.

[0162] The processing waiting indicator may be temporarily displayed on the display unit 340, for example, until the control unit 300 displays a message indicating an answer to the question or a message indicating a refusal to answer the question. Alternatively, in that case, the control unit 300 may display a message indicating an answer to the question or a message indicating a refusal to answer the question on the display unit 340 instead of the processing waiting indicator.

[0163] Meanwhile, the control unit 200 of the search server device 20 determines whether the question is related to model information (S262). Here, the following methods can be considered as ways to determine whether the question is related to model information.

[0164] As a first method, the control unit 200 may extract knowledge information related to the model information from the model information obtained from the token, and when the number of knowledge information items that approximate the model information compared with the question is greater than or equal to a predetermined threshold, the control unit 200 may determine that the question is related to the model information.

[0165] As a second method, the control unit 200 may reconstruct the question, vectorize it, compare it with knowledge information to search for candidates, and if a candidate is found, determine that the question is related to model information. The reconstructed question may be used in subsequent processing (for example, S222, S224, S226).

[0166] As a third method, if the question contains the model information itself, the control unit 200 may determine that the question is related to the model information.

[0167] If the control unit 200 determines that the question is not related to the model information, it sends a rejection notification to the terminal device 30 (S262; No → S264). Here, the control unit 200 can, for example, enable the reception of questions from the terminal device 30.

[0168] Meanwhile, the control unit 300 of the terminal device 30 determines whether it has received a rejection notification (S364). If it has received a rejection notification, it displays a screen on the display unit 340 of the terminal device 30 that includes a message rejecting the answer to the question (S364; Yes → S366). Furthermore, the control unit 300 displays a screen on the display unit 340 that allows the user to input a question, and accepts the input of a question again (S368).

[0169] On the other hand, if the control unit 200 determines that the question is related to model information, it determines whether a token over has occurred (S262; Yes → S266). Here, a token over means, for example, that the number of characters or words in the conversation exceeds a predetermined upper limit, such as when the conversation history continues for a long time. Also, a token here refers to the basic unit (word unit or character unit) of text in the conversation (chat) history processed by the large-scale language model. The control unit 300 may determine a token over if, for example, the number of characters or words in the conversation exceeds a predetermined upper limit, or if the per-minute token consumption limit is reached. If a token over has occurred, for example, the control unit 200 sends a token over notification to the terminal device 30 (S266; Yes → S268).

[0170] On the other hand, if the control unit 300 of the terminal device 30 determines that it has not received a rejection notification, it determines whether it has received a token over notification (S364; No → S370). If the control unit 300 receives a token over notification, it displays a display screen on the display unit 340 that notifies the user that a token over has occurred and requests that the conversation history be cleared (deleted) (S370; Yes → S372). The control unit 300 may also display a display screen on the display unit 340 that includes text such as, for example, "The conversation length has reached its limit. You need to clear the conversation to continue."

[0171] Next, the control unit 300 displays a clear instruction on the display screen, indicating that the conversation history can be cleared (S374). Then, the control unit 300 determines whether, for example, the clear instruction has been selected by the user (S376). If it determines that the clear instruction has not been selected, the clear instruction continues to be displayed (S376; No).

[0172] On the other hand, if it is determined that the clear instruction display has been selected, the control unit 300 clears the conversation history (S376; Yes → S378). Subsequently, the control unit 300 displays a conversation initial screen on the display unit 340 in which the conversation history has been cleared and questions can be entered, and can accept question input again (S380). Here, the conversation initial screen may be, for example, a display screen that is displayed when a session is started and in which questions can be entered. The conversation initial screen may be, for example, the initial screen of a chat application, which displays an example of a question.

[0173] Meanwhile, the control unit 200 of the search server device 20 clears the conversation history corresponding to the session (S270). By deleting the conversation history from the conversation history storage area 2130, the control unit 200 can receive questions from the terminal device 30 in the same state as when the session was started.

[0174] Furthermore, the control unit 200 of the search server device 20 terminates processing if, for example, a token over error does not occur (S266; No). Similarly, the control unit 300 of the mobile terminal device 30 terminates processing if it does not receive a token over error notification (S370; No).

[0175] [3.2 Example of Operation] Figure 20 shows an example of a display screen, including a message, that is displayed to the user as a result of the operation of the terminal device 30 in this embodiment.

[0176] Figure 20(a) is an example of the display screen W330 displayed on the display unit 340 of the terminal device 30 as the initial chat screen (initial conversation screen) displayed when the chat application is launched on the terminal device 30 at S308 in Figure 8. The display screen W330 displays an input form F330 for entering a question, a send button B330 for sending the question entered in the input form F330, and an area R331 that displays the upper limit of the number of characters that can be entered in the question (for example, 300 characters in this embodiment) and the number of characters that have been entered. Furthermore, the display screen W330 displays the message M330 that says "AI Assistant: When you enter a question, the AI ​​will create an answer," and an area R330 that displays an example of a question.

[0177] If the user enters, for example, "I don't know how to print on an envelope" into the input form F330 where they enter their question, and then selects the send button B330, the control unit 300 displays the display screen W332 shown in Figure 20(b) on the display unit 340.

[0178] Figure 20(b) shows an example of a display screen W332 displayed on the display unit 340 of the terminal device 30. Display screen W332 shows the message M332, which was entered by the user in the input form F330 of the display screen W330 in Figure 20(a), and reads "I don't know how to print on an envelope."

[0179] Here, the display screen W332 further displays the message M334, "...", on the display unit 340 as a processing waiting indicator. In this way, the control unit 300 can indicate a processing waiting state by displaying text such as "..." on the display unit 340 of the terminal device 30 while the search server device 20 is generating the answer text after the user has entered a question.

[0180] Furthermore, when the control unit 300 receives, for example, a response to a question or a rejection notification from the search server device 20, it may replace message M334 with a new message based on the response or rejection notification and display it on the display unit 340.

[0181] Figure 20(c) shows an example of a display screen W334 displayed on the display unit 340 of the terminal device 30. The display screen W334 shows an input form F332 for entering a question and a send button B334 for sending the question entered in input form F332. Here, for example, the user enters an inappropriate question in input form F332, for example, a question completely unrelated to the image forming apparatus 10, and then selects the send button B334 to send the question.

[0182] The control unit 300 displays the entered question as message M336 on the display screen W334. Next, the control unit 300 displays message M338, which includes the message "Sorry, but we cannot answer such a question," informing the user that it has received a response to the question sent to the search server device 20 for an inappropriate question, and a button B332 to start a new conversation. It also displays an input form F332 on the display unit 340 and accepts the input of another question. Furthermore, if the button B332 to start a new conversation is pressed, the control unit 300 displays the display screen W330 on the display unit 340 as the initial chat screen (initial conversation screen) as shown in Figure 20(a).

[0183] In this way, if a question unrelated to the image forming apparatus 10 (an inappropriate question) is entered, the question can be rejected. The control unit 300 can display a message to inform the user, for example, "I'm sorry, but I cannot answer that question." In addition to the above, it is also possible to display a button that allows the user to clear the chat history displayed on the display unit 340 and start a new chat, displaying the initial chat screen.

[0184] Figure 20(d) shows an example of a display screen W336 displayed on the display unit 340 of the terminal device 30. The display screen W336 shows an input form F334 for entering a question, a message M340 on the input form F334 that reads "The conversation length has reached its limit. You need to clear the conversation to continue," and a button B336 for starting a new conversation. Here, message M340 is a message that includes notification of a token overrun and a request to clear (delete) the conversation history. The button B336 for starting a new conversation is a clear instruction display. Furthermore, the display screen W336 also shows an area R332 that displays an icon indicating that the input form F334 is tappable, for example, instead of a send button for sending a question.

[0185] Here, even if the user enters a question into input form F334, for example, the control unit 300 does not accept the input of the question into input form F334. On the other hand, if the user taps the button B336 to start a new conversation or any part of input form F334, for example, the control unit 300 clears the chat history displayed on the display unit 340 and starts a new chat. For example, the control unit 300 displays the initial chat screen (initial conversation screen) W330 shown in Figure 20(a) on the display unit 340.

[0186] Thus, the control unit 300 can similarly reject a request for a question as a token overflow if the number of questions reaches the upper limit, for example, if the conversation history is long and the number of characters in the conversation exceeds the upper limit.

[0187] The control unit 300 can also reject a request if the maximum token consumption limit per minute is reached, indicating a token overflow. The control unit 300 may also display a message to the user, such as, "The maximum number of question requests has been reached. Please try again later." The control unit 300 may also display a "Start a new conversation" button after a certain period of time, for example, about one minute.

[0188] [4. Fourth Embodiment] The following describes a fourth embodiment, in which the chat application can be displayed on the image forming apparatus 10 instead of the terminal device 30.

[0189] For example, in the first to third embodiments, the terminal device 30 was configured to receive input text of questions from the user and display answer texts to those input texts in a conversational format. In this embodiment, tokens are not transmitted from the image forming apparatus 10 to the terminal device 30, and the chat application is displayed on the display unit 140 of the image forming apparatus 10 to input questions and answer questions.

[0190] In the flow chart of Figure 7, for example, instead of the image forming apparatus 10 executing S116 and the terminal device 30 executing S302, the image forming apparatus 10 executes the processing from S304 onward on behalf of the terminal device 30. This makes it possible for the image forming apparatus 10 to include the functions of the terminal device 30.

[0191] [5. Variant] This disclosure is not limited to the embodiments described above, and various modifications are possible. In other words, embodiments obtained by combining technical means that are appropriately modified within the scope of this disclosure are also included in the technical scope.

[0192] For example, in Figure 20(c), the control unit 200 may terminate the session if an answer to a question is rejected due to an inappropriate question, if the total number of answers to questions due to inappropriate questions reaches a predetermined number or a predetermined number consecutively, if a predetermined amount of time has elapsed since a token was output from the image forming apparatus, or if a predetermined amount of time has elapsed since the previous question. This allows the control unit 200 to prevent unauthorized use of the chat application until the terminal device 30 obtains a token again and a session is started.

[0193] As an example of operation, in Figure 20(c), even if the user selects, for example, the send button B334, the control unit 300 of the terminal device 30 may not display the question on the display screen W334 and may not accept the question. Also, if the user taps, for example, the button B332 to start a new conversation in Figure 20(c), the control unit 300 may terminate the chat application or activate the camera for reading the QR code.

[0194] Furthermore, although the embodiments described above are explained separately for the sake of explanation, they can be combined and implemented to the extent possible. In addition, we intend to obtain rights to any of the technologies described in this specification through amendments or divisional applications.

[0195] Furthermore, in each embodiment, the program that operates in each device is a program that controls the CPU and other components (a program that makes the computer function) in order to realize the functions of the embodiments described above. The information handled by these devices is temporarily stored in a temporary storage device (for example, RAM) during processing, and then stored in various ROMs or HDDs, and read, modified, and written by the CPU as needed.

[0196] Here, the recording medium for storing the program may be any of the following: semiconductor media (e.g., ROM or non-volatile memory card), optical recording medium or magneto-optical recording medium (e.g., DVD (Digital Versatile Disc), CD (Compact Disc), BD (Blu-ray® Disc)), magnetic recording medium (e.g., magnetic tape, flexible disk), etc.

[0197] Furthermore, when distributing the program to the market, it can be stored on a portable recording medium and distributed, or transferred to a server computer connected via a network such as the Internet. In this case, the storage device of the server device is, of course, also included in this disclosure.

[0198] Furthermore, the data mentioned above may not be stored within the device itself, but rather stored on an external device and retrieved as needed. For example, the data may be stored on a NAS (Network Attached Storage) or on the cloud.

[0199] Furthermore, the scope of this disclosure is not limited to the configurations explicitly described in the specification, but also includes combinations of the technologies disclosed herein. While the configurations for which patent protection is sought are described in the attached claims, there is no intention to exclude them from the technical scope simply because they are not described in the claims.

[0200] Furthermore, the phrases "in the case of..." and "when..." in the above-mentioned specification are explained as examples only, and do not represent a configuration limited to those described. Even for configurations other than those described, we disclose information that would be obvious to a person skilled in the art, and we intend to acquire rights to such information.

[0201] Furthermore, the descriptions of the processes and data flows described in the specification are not limited to the order in which they are described. For example, configurations in which parts of the process are deleted or the order is rearranged are also disclosed, and the company intends to acquire rights to them.

[0202] Furthermore, although the functions described in the embodiments are explained as being performed by each device, they may also be implemented by a single device or by utilizing an external server.

[0203] Furthermore, each functional block or feature of the apparatus used in the embodiments described above may be implemented or executed by an electrical circuit, such as an integrated circuit or a plurality of integrated circuits. An electrical circuit designed to perform the functions described herein may include a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gates or transistor logic, discrete hardware components, or a combination thereof. The general-purpose processor may be a microprocessor, a conventional processor, controller, microcontroller, or state machine. The aforementioned electrical circuit may consist of digital circuits or analog circuits. Also, if advances in semiconductor technology lead to the emergence of integrated circuit technologies that replace current integrated circuits, one or more aspects of this disclosure may use new integrated circuits based on such technologies. [Explanation of Symbols]

[0204] 1 System 10 Image forming apparatus 20 Search Server Device 30 Terminal devices 40 Web server device 50 Chat Server Device 52 Large-scale language models 100, 200, 300, 400 Control Unit 110, 210, 310, 410 storage 120, 220, 320, 420 ROM 130, 230, 330, 430 RAM 340, 440 display section 350, 450 operation unit 160 Image forming unit 165 Image reading unit 170, 270, 370, 470 Communications Department

Claims

1. A model information acquisition unit that acquires model information of an information processing device, A reception unit that receives input text, A generation unit that generates a prompt when the input statement relates to the model information, Based on the aforementioned prompt, an output unit outputs a response sentence obtained from a large-scale language model, A system equipped with this feature.

2. The device further has a storage unit for storing document information relating to the aforementioned information processing device, The generation unit generates the prompt based on the input sentence and the document information corresponding to the model information. The system according to claim 1.

3. The memory unit stores the combination of the input sentence and the response sentence as a conversation history. The generation unit generates the prompt based on the conversation history when the reception unit receives the input sentence again. The system according to claim 2.

4. The aforementioned model information acquisition unit acquires identification information including the aforementioned model information from the information processing device, The reception unit receives input text via a terminal device. The terminal device receives the input message after obtaining the identification information. The system according to claim 1.

5. A system including an information processing device, a terminal device, and a server device, The information processing device transmits the model information of the information processing device to the server device. The terminal device obtains connection information from the information processing device for connecting to the server device. The terminal device establishes a communication path with the server device based on the connection information. The terminal device receives an input message and transmits it to the server device. The server device generates a prompt when the input statement is related to the model information. The server device sends the prompt to the large-scale language model and obtains a response from the large-scale language model. The server device transmits the response text to the terminal device. The terminal device displays the answer text. system.

6. The information processing device transmits the model information of the information processing device to the server device. The server device generates the connection information based on the model information and transmits it to the information processing device. The system according to claim 5.

7. A step to acquire model information for information processing equipment, A reception step that accepts the input text, A generation step that generates a prompt when the input statement relates to the model information, Based on the aforementioned prompt, an output step outputs a response sentence obtained from a large-scale language model, A control method including

8. to the computer A model information acquisition function that acquires model information of information processing equipment, A reception function that accepts input text, A generation function that generates a prompt when the input statement relates to the model information, Based on the aforementioned prompt, the output function outputs a response sentence obtained from a large-scale language model, A program that achieves this.

9. A model information acquisition unit that acquires model information of an information processing device, A reception unit that receives input text, A generation unit that generates a prompt based on the input statement, Based on the aforementioned prompt, an output unit outputs a response sentence and notification obtained from a large-scale language model, Equipped with, The output unit is, If the input statement relates to the model information, the response statement is output. If the input statement is not related to the model information, the system will not output the response statement, or it will output a notification that it will not output a response statement for the input statement. system.

10. The generating unit is If the input statement relates to the model information, the prompt is generated. If the input statement is not related to the model information, the prompt will not be generated. The system according to claim 9.

11. A system including an information processing device, a terminal device, and a server device, The information processing device transmits the model information of the information processing device to the server device. The terminal device obtains connection information from the information processing device for connecting to the server device. The terminal device establishes a communication path with the server device based on the connection information. The terminal device receives an input message and transmits it to the server device. The server device generates a prompt based on the input statement, The server device sends the prompt to the large-scale language model and obtains a response from the large-scale language model. The server device transmits the response text to the terminal device. The terminal device displays the response text if it is related to the model information, and if it is not related to the model information, it suppresses the display of the response text related to the model information, or outputs a notification that it will not display a response text for the input text. system.

12. The terminal device acquires the connection information from the information processing device via short-range wireless communication. The system according to claim 11.

13. The terminal device obtains the connection information from the information processing device as encoded information. The system according to claim 11.

14. In a mobile terminal that displays a response to a user's input in a conversational format based on model information acquired from an information processing device, A storage unit that stores the model information acquired from the information processing device, An input unit that receives the aforementioned input text from the user, The system includes a display unit that displays the input text in conversational format and the corresponding response text, The aforementioned display unit is If the input statement relates to the model information, the response statement will be displayed. If the input statement is not related to the model information, the response statement will not be displayed, or a notification will be displayed indicating that a response statement will not be displayed for the input statement. Terminal device

15. In an information processing device that displays user input text and response text in conversational format, A memory unit that stores model information of the information processing device, An input unit that receives the aforementioned input text from the user, The system includes a display unit that displays the input text in conversational format and the corresponding response text, The aforementioned display unit is If the input statement relates to the model information, the response statement will be displayed. If the input statement is not related to the model information, the response statement will not be displayed, or a notification will be displayed indicating that a response statement will not be displayed for the input statement. Information processing device.

Images