Computer program, apparatus and method for causing image processing apparatus to execute image processing

Abstract

To disclose a technique for improving user's convenience.SOLUTION: The computer program causes the computer to function as a first reception unit that receives an input of a first natural sentence, a first supply unit that supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is received, and a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit. The first object information is caused to function as: a first acquisition unit which includes first identification information for identifying the first object for each of M1 screens among the M screens; and a second display control unit which, when the first object information is acquired from a natural sentence processing unit, causes a display device to sequentially display M screens including the M1 screens according to the first object information, wherein the first object is emphatically displayed on each of the M1 screens.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present specification relates to a technique in a situation where processing relating to image processing should be performed. [Background technology]

[0002] Patent Document 1 discloses a technique for causing a printer to execute printing using an application installed on a terminal. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-085588 Summary of the Invention [Problem to be solved by the invention]

[0004] This specification discloses a technique for improving user convenience. [Means for solving the problem]

[0005] This specification discloses a computer program for causing an image processing device to execute image processing. The computer program includes a computer including the following components: a first display control unit for sequentially displaying M screens (M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device on a display device, each of the M screens including one or more objects, and the first process being executed when a first object corresponding to the first process is selected from the one or more objects on each of the M screens; a first accepting unit for accepting an input of a first natural sentence related to the first process from a user; a first supply unit for supplying first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is accepted; and a second supply unit for supplying first natural sentence data representing the first natural sentence to a natural sentence processing unit from the natural sentence processing unit. The first acquisition unit may function as a first acquisition unit that acquires first object information based on first natural language data, wherein the first object information includes, for each of M1 screens out of the M screens (M1 is an integer greater than or equal to 2 and less than or equal to M), first identification information that identifies the first object among the one or more objects included in that screen; and a second display control unit that, when the first object information is acquired from the natural language processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, wherein the first object identified by the first identification information is highlighted on each of the M1 screens.

[0006] According to the above configuration, when the computer accepts input of a first natural sentence, it supplies the first natural sentence data to the natural sentence processing unit and acquires first object information from the natural sentence processing unit. Then, the computer sequentially displays M screens including M1 screens according to the first object information. On each of the M1 screens, the first object identified by the first identification information is highlighted. This allows the user to easily recognize that the first object should be selected on each of the M1 screens. This improves user convenience.

[0007] A computer-readable recording medium storing the above computer program, an apparatus implemented by the above computer program, and a method executed by the apparatus are also novel and useful. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a communication system. [Figure 2] 10A to 10C are diagrams illustrating screen transitions when causing the multi-function device to execute printing. [Figure 3] This is a continuation of the screen transition diagram in Figure 2. [Figure 4] This is a screen transition diagram continuing from Figure 3. [Figure 5] 10A to 10C are diagrams illustrating screen transitions when a multi-function device is caused to execute a scan. [Figure 6] 10 is a flowchart of a process executed by an application. [Figure 7] This is a screen transition diagram for Case A. [Figure 8] This is a screen transition diagram continuing from Figure 7. [Figure 9] This is a screen transition diagram continuing from Figure 8. [Figure 10] This is a screen transition diagram continuing from Figure 9. [Figure 11] This is a screen transition diagram for Case B. [Figure 12] This is a screen transition diagram for Case C. [Figure 13]This is a screen transition diagram continuing from Figure 12. [Figure 14] This is a screen transition diagram for Case D. DETAILED DESCRIPTION OF THE INVENTION

[0009] (First Example) (Communication system configuration; Figure 1) As shown in FIG. 1, the communication system 2 includes a terminal 10, a multi-function peripheral (MFP) 100, and an artificial intelligence (AI) server 200. The terminal 10 and the MFP 100 are connected to a local area network (LAN) 4. The terminal 10 and the MFP 100 can communicate with each other via the LAN 4. The LAN 4 may be a wired LAN or a wireless LAN. The AI ​​server 200 is installed on the Internet 6. The devices 10, 100 and the AI ​​server 200 can communicate with each other via the LAN 4 and the Internet 6.

[0010] (Configuration of Terminal 10) The terminal 10 is a portable terminal device such as a mobile phone, smartphone, PDA, or tablet PC. In a modified example, the terminal 10 may be a desktop PC, laptop PC, or the like. The terminal 10 includes an operation unit 12, a display unit 14, a communication interface 16, and a control unit 30. The units 12 to 30 are connected to a bus line. Hereinafter, the interface will be referred to as "I / F."

[0011] The operation unit 12 is a user interface that allows the user to input various information to the terminal 10. The operation unit 12 includes, for example, a touch panel for displaying software keys (operation objects), hardware keys, or both. The hardware keys include, for example, buttons or switches. The display unit 14 is a display or panel for displaying various information. The panel may or may not be a touch panel. The panel is, for example, a liquid crystal panel or an organic EL panel. The communication I / F 16 is connected to the LAN 4.

[0012] The control unit 30 includes a CPU 32 and a memory 34. The memory 34 includes a main storage device and an auxiliary storage device. For example, the main storage device includes a RAM and a cache memory. For example, the auxiliary storage device may be a ROM, a flash memory, a solid-state drive (SSD), a hard disk drive (HDD), or a combination thereof. An OS program 36 and an application program 38 are stored in the auxiliary storage device of the memory 34. Hereinafter, the OS program and the application program will be referred to as "OS" and "apps," respectively. The OS 36 controls the basic operations of the terminal 10. For example, the app 38 enables the MFP to perform multiple functions, such as printing and scanning. The app 38 is downloaded from a server on the Internet 6 and installed on the terminal 10. In other words, the app 38 is a so-called native app. The CPU 32 performs various processes according to the programs loaded from the auxiliary storage device to the main storage device. In the following description, various processes that are implemented by the CPU 32 according to a program may be described as being performed primarily by the OS 36 or the application 38.

[0013] The application 38 can cause the MFP 100 to perform multiple functions such as a print function, a scan function, etc. In particular, the application 38 of this embodiment can cooperate with the AI ​​server 200 to guide the user through the operations that should be performed when causing the MFP 100 to perform a print function, etc.

[0014] (MFP100 configuration) The MFP 100 is a peripheral device capable of performing multiple functions such as printing, scanning, and facsimile, for example, a peripheral device of the terminal 10. In a modified example, the MFP 100 may be a single-function device such as a printer capable of performing only printing or a scanner capable of performing only scanning. The MFP 100 is assigned a device ID "dv1" that identifies the MFP 100. The device ID is, for example, a model name, an IP address, or a combination thereof. In this specification, the device ID is the model name. The model name is model identification information for the device. The model name may be a higher-level model name that encompasses multiple model names. For example, the model name may be "ABC Series," which encompasses model names ABC1 and ABC2. For example, the model name may be a serial number. The MFP 100 includes an operation unit 112, a display unit 114, a communication I / F 116, a print execution unit 118, a scan execution unit 120, and a control unit 130. Each of the units 112 to 130 is connected to a bus line.

[0015] The operation unit 112 is a user interface that allows a user to input various information to the MFP 100. The operation unit 112 includes, for example, a touch panel for displaying software keys, hardware keys, or both. Hardware keys include, for example, buttons or switches. The display unit 114 is a display or panel for displaying various information. The panel may or may not be a touch panel. The panel may also be, for example, a liquid crystal panel or an organic EL panel. The communication I / F 116 is connected to the LAN 4. The print execution unit 118 includes an electrophotographic, inkjet, or thermal print engine. An electrophotographic print engine includes a photosensitive member and an exposure device that emits light to expose the photosensitive member. An inkjet print engine includes a print head that ejects ink droplets. A thermal print engine includes a print head that generates heat using a heater. The scan execution unit 120 includes a scanner engine having an image sensor such as a Charge-Coupled Device (CCD) image sensor or a Contact Image Sensor (CIS) image sensor.

[0016] The control unit 130 includes a CPU 132 and a memory 134. The memory 134 includes a main storage device and an auxiliary storage device. As an example, the main storage device includes a RAM and a cache memory. As an example, the auxiliary storage device may be a ROM, a flash memory, an SSD, an HDD, or a combination thereof. A program 136 is stored in the auxiliary storage device of the memory 134. The CPU 132 performs various processes in accordance with the program loaded from the auxiliary storage device to the main storage device.

[0017] (Configuration of AI server 200) The AI ​​server 200 generates information (specifically, text and guidance information) for guiding the user through operations. The servers described herein (e.g., the AI ​​server 200, the cloud print server (not shown)) are installed on the Internet 6 by the MFP 100 vendor (hereinafter simply referred to as the "vendor"). While the AI ​​server 200 will be described in particular below, the same applies to other servers described herein, such as the cloud print server. The AI ​​server 200 may be a physical server, a cloud virtual server, a virtual machine, or the like. The AI ​​server 200 may be a single server or a collection of multiple servers. The AI ​​server 200 may be a server on an intranet or a cloud server on the Internet 6. In a variant, the AI ​​server 200 may be installed on the Internet 6 by a business operator other than the vendor. In another variant, the vendor may not prepare the AI ​​server 200 hardware on its own, but may use an environment provided by an external cloud computing service. In this case, the vendor may prepare a program (i.e., software) for the AI ​​server 200 and install it in the above environment to realize the AI ​​server 200. In this embodiment, as an example, a form in which the AI ​​server 200 is a physical server will be described.

[0018] The AI ​​server 200 includes a communication I / F 216 and a control unit 230. The communication I / F 216 and the control unit 230 are connected to a bus line. The communication I / F 216 is connected to the Internet 6.

[0019] The control unit 230 includes a processor 232 and a memory 234. The processor 232 includes a CPU, a graphics processing unit (GPU), a neural processing unit (NPU), or a combination thereof. The memory 234 includes a main storage device and an auxiliary storage device. For example, the main storage device includes a RAM and a cache memory. For example, the auxiliary storage device may be a ROM, a flash memory, an SSD, an HDD, or a combination thereof. A program 236 is stored in the auxiliary storage device of the memory 234. The processor 232 performs various processes in accordance with the program loaded from the auxiliary storage device to the main storage device.

[0020] A trained model is incorporated into the program 236. The AI ​​server 200 generates information for guiding a user's operation based on the trained model incorporated into the program 236. That is, the trained model has previously learned the correspondence between text and guidance information. The trained model includes a program and various parameters. The trained model includes, for example, language models such as a large-scale language model (LLM) and a small-scale language model (SLM).

[0021] (Printing process: Figures 2 to 4) Next, with reference to FIGS. 2 to 4, screens sequentially displayed on the terminal 10 when the user uses the application 38 to cause the MFP 100 to print an image will be described. In particular, FIGS. 2 to 4 describe a situation in which the user desires to print a document file (i.e., a document) in monochrome on both sides. Each screen displayed in this embodiment is displayed on the display unit 14 of the terminal 10 by the CPU 32 of the terminal 10 executing the application 38. In the following description of each screen, for ease of understanding, the CPU 32 or the application 38 will not be described as the subject of processing, but the terminal 10 will be described as the subject of processing. Also, in the initial state of FIG. 2 (i.e., the state before the home screen SC1 is displayed), information about one or more MFPs including the MFP 100 (e.g., the device ID, IP address, etc. of the MFP) has already been registered in the memory 34 of the terminal 10 (i.e., the application 38).

[0022] When the terminal 10 receives an application launch operation from the user to launch the application 38, the terminal 10 launches the application 38 and displays the home screen SC1 of Fig. 2 on the display unit 14. The home screen SC1 includes one or more device objects OB11, OB12, a plurality of function objects OB13 to OB18, and a bot button OB19.

[0023] The device object OB11 is an object corresponding to the registered MFP 100 and includes the device ID "dv1" of the MFP 100. As described above, the device ID "dv1" is the model name of the MFP 100. Note that the device ID "dv1" may include the IP address of the MFP 100 in addition to the model name of the MFP 100. Although partially hidden in FIG. 2, the device object OB12 is an object corresponding to another MFP registered in the terminal 10 and includes the device ID of the other MFP. Note that in a situation where only one device (i.e., the MFP 100) is registered in the terminal 10, the device object OB12 may be an object for registering a new MFP. The function objects OB13 to OB18 are objects corresponding to a document printing function, a photo printing function, a voice printing function, a scan function, a service registration function, and a consumables purchasing function, respectively. The document printing function is a function for printing an image represented by a document file (e.g., a Word file, a PDF file, a Text file, etc.). The photo printing function is a function for printing an image represented by a photo file (e.g., a JPEG file). The audio printing function is a function for printing an image represented by an audio file (e.g., an MP3 file). Here, an image represented by an audio file is an image represented by a document file obtained by transcribing the audio represented by the audio file. The service registration function is a function for registering an MFP with, for example, a flat-rate printing service. The consumables purchasing function is a function for ordering (i.e., purchasing) consumables used by the MFP (e.g., ink cartridges, printing paper, etc.). The bot button OB19 is a button for starting the operation guidance described below.

[0024] When the terminal 10 receives, at T20, a selection of a function object OB13 corresponding to document printing from the user, it displays a file selection screen SC2 on the display unit 14. The file selection screen SC2 includes a plurality of file objects OB21 to OB23 corresponding to a plurality of document files stored in the memory 34, a cancel object OB24, and a bot button OB19. Each of the file objects OB21 to OB23 includes a file name (e.g., F1 to F3).

[0025] When the terminal 10 receives a selection of a file object OB21 corresponding to the file name F1 from the user at T20, the terminal 10 displays a print preview screen SC3 on the display unit 14. The print preview screen SC3 includes a preview image P, a device object OB31, a settings object OB32, a print object OB33, and a bot button OB19. The preview image P corresponds to the document image data of the first page of the document image data for multiple pages (20 pages in this embodiment) included in the target file corresponding to the file name F1. The character string "1 / 20" in the print preview screen SC3 indicates that the currently displayed preview image P corresponds to the document image data of the first page of the document image data for the 20 pages. Although not shown in the figure, the user can display different preview images corresponding to different pages of document image data by sliding the area where the preview image P is displayed left or right.

[0026] The device object OB31 includes the device ID of the MFP100 corresponding to the device object OB11 whose entirety is displayed when the function object OB13 is selected on the home screen SC1, and the paper size. That is, the device object OB31 is an object that indicates the device that prints the image corresponding to the preview image and the size of the print medium. In a modified example, the device object OB31 may not include the paper size, or may include information other than the paper size (e.g., paper type (plain paper, glossy paper, etc.)). Furthermore, when the selection of the device object OB31 is accepted by the user, the device object OB31 may be configured to allow settings such as the paper size to be changed. The setting object OB32 is an object for displaying a print setting screen. The print object OB33 is an object for executing image printing.

[0027] When the terminal 10 accepts selection of a setting object OB32 from the user at T30, it displays the print setting screen SC4 of Fig. 3 on the display unit 14. The print setting screen SC4 includes a plurality of item objects OB41 to OB46, a back button OB47, and a bot button OB19. Each of the item objects OB41 to OB46 includes the current setting value of the corresponding item.

[0028] The item object OB41 is an object for specifying the number of copies to be printed. The item object OB42 is an object for specifying layout settings (for example, "1 page / sheet" indicating that one page of images will be printed on one sheet of printing medium, or "2 pages / sheet" indicating that two pages of images will be printed on one sheet of printing medium).

[0029] The item object OB43 is an object for specifying a double-sided printing setting (i.e., one of "single-sided printing," "long-edge binding," and "short-edge binding"). "Long-edge binding" means that, in a double-sided printing medium, the up-down orientation of the image of page N printed on the first side of the printing medium matches the up-down orientation of the image of page (N+1) printed on the second side of the printing medium, which is the reverse side of the first side. "Short-edge binding" means that, in a double-sided printing medium, the up-down orientation of the image of page N printed on the first side of the printing medium is opposite to the up-down orientation of the image of page (N+1) printed on the second side of the printing medium.

[0030] The item object OB44 is an object for specifying a color / monochrome setting (i.e., either "color" or "monochrome"). In a modified example, the object may be configured to allow selection of an option other than "color" or "monochrome" (for example, "grayscale") to be specified.

[0031] The item object OB45 is an object for specifying the image quality setting (e.g., "normal," "high quality," etc.). The item object OB46 is an object for selecting the range of pages desired to be printed from the file corresponding to the selected file name F1. The back button OB47 is a button for displaying a print preview screen (e.g., SC3).

[0032] When the terminal 10 receives from the user at T40 the selection of the item object OB43 corresponding to the double-sided printing setting, it displays a dialog D1 on the display unit 14, superimposed on the print setting screen SC4. The dialog D1 includes setting value objects OB51 to OB53 corresponding to each setting value of the double-sided printing setting, and a cancel button OB54. The setting value object OB51 is an object corresponding to the option "single-sided." The setting value object OB52 is an object corresponding to the option "long-edge binding." The setting value object OB53 is an object corresponding to the option "short-edge binding." Each setting value object OB51 to OB53 includes a radio button for specifying the corresponding option. The cancel button OB54 is a button for canceling the change to the double-sided printing setting and returning to the print setting screen SC4. When the dialog D1 is displayed, the radio button included in the setting value object OB51 corresponding to the current setting value (i.e., "single-sided") is selected in the dialog D1.

[0033] When the terminal 10 receives the selection of the radio button included in the setting value object OB52 corresponding to the option "long edge binding" in T50, it displays the print setting screen SC5 on the display unit 14. The print setting screen SC5 is the same as the print setting screen SC4, except that the setting values ​​included in the item object OB43 are different.

[0034] When the terminal 10 receives from the user in T60 the selection of the item object OB44 corresponding to the color / monochrome setting, it displays a dialog box D2 on the display unit 14, superimposed on the print setting screen SC5, as shown in FIG. 4. The dialog box D2 includes setting value objects OB71 and OB72 corresponding to each setting value of the color / monochrome setting, and a cancel button OB73. The setting value object OB71 is an object corresponding to the option "color." The setting value object OB72 is an object corresponding to the option "monochrome." Each setting value object OB71 and OB72 includes a radio button for specifying the corresponding option. The cancel button OB73 is a button for canceling the change to the color / monochrome setting and returning to the print setting screen SC5. When the dialog box D2 is displayed, the radio button included in the setting value object OB71 corresponding to the current setting value (i.e., "color") is selected in the dialog box D2.

[0035] When the terminal 10 receives selection of the radio button included in the setting value object OB72 corresponding to the option "Monochrome" at T70, it displays the print setting screen SC6 on the display unit 14. The print setting screen SC6 is the same as the print setting screen SC5, except that the setting values ​​included in the item object OB44 are different.

[0036] When terminal 10 accepts selection of back button OB47 in print setting screen SC5 from the user in T80, it re-displays print preview screen SC3 on display unit 14. Then, when terminal 10 accepts selection of print object OB33 from the user in T90, it converts image data included in the target file into print data in accordance with each specified setting value and sends the converted print data to MFP 100 via communication I / F 16. The print data has a data format that can be interpreted by MFP 100. As a result, MFP 100 causes print execution unit 118 to print the image represented by the print data. In particular, because each specified setting value includes "long edge binding" and "monochrome," the user can obtain print media on which images included in the document file are printed in monochrome on both sides (in this case, 10 sheets of print media because 20 pages of images are printed on both sides of the print media).

[0037] (Scanning process; Figure 5) Next, with reference to FIG. 5, screens sequentially displayed on the terminal 10 when the user uses the application 38 to cause the MFP 100 to scan a document will be described. In particular, FIG. 5 describes a situation in which the user desires to scan a document in monochrome. The user places the document to be scanned on the MFP 100. Thereafter, when the terminal 10 accepts, from the user at T100 in FIG. 2, a selection of a function object OB16 corresponding to scan on the home screen SC1 (see FIG. 2), the terminal 10 displays a scan setting screen SC11 in FIG. 5 on the display unit 14. The scan setting screen SC11 includes a plurality of item objects OB111 to OB113, a scan object OB114, a back button OB115, and a bot button OB19. Each of the item objects OB111 to OB113 includes a current setting value for the corresponding item.

[0038] The item object OB111 is an object for specifying color / monochrome settings, and is similar to the item object OB44 of the print setting screen SC4, etc. The item object OB112 is an object for specifying a document size setting (e.g., "A4," "B5," "Letter," etc.). The item object OB113 is an object for specifying a resolution setting (e.g., "300 dpi," "600 dpi," etc.). The scan object OB114 is an object for scanning a document. The back button OB115 is a button for displaying a home screen (e.g., SC1).

[0039] When the terminal 10 receives a selection of an item object OB111 corresponding to color / monochrome setting from the user at T110, the terminal 10 displays a dialog D11 superimposed on the scan setting screen SC11 on the display unit 14. The dialog D11 is similar to the dialog D2 (see FIG. 4).

[0040] When the terminal 10 accepts the selection of the radio button included in the setting value object OB122 corresponding to the option "Monochrome" at T120, it displays the scan setting screen S12 on the display unit 14. The scan setting screen SC12 is the same as the scan setting screen SC11 except that the setting value included in the item object OB111 is different.

[0041] When terminal 10 accepts selection of scan object OB114 from the user at T130, it transmits a scan instruction including each specified setting value to MFP 100 via communication I / F 16. As a result, MFP 100 causes scan execution unit 120 to scan the document to be scanned that is installed in MFP 100, and generates scan data representing an image of the document to be scanned. In particular, the scan data is generated according to the setting values ​​included in the scan instruction. Then, the scan data is transmitted to terminal 10 (not shown), allowing the user to confirm the image represented by the scan data.

[0042] A user who is proficient in operating the application 38 can cause the MFP 100 to execute printing or the like by using the application 38 as described above. On the other hand, a user who is not proficient in operating the application 38 may have difficulty in causing the MFP 100 to execute printing or the like by using the application 38. For example, the user needs to operate the application 38 while referring to a web page or the like that contains an explanation of how to operate the application 38, which is time-consuming for the user. The application 38 of this embodiment realizes the guidance process of FIG. 6 in order to improve such user convenience.

[0043] (Guidance process; Figure 6) Next, the guidance process realized by the application 38 will be described with reference to Fig. 6. The process of Fig. 6 is triggered by receiving selection of the bot button OB19 from the user on any of the above screens SC1 to SC12. Hereinafter, the screen including the bot button OB19 selected as the trigger for the process of Fig. 6 will be referred to as the "guidance start screen."

[0044] In S10, the CPU 32 divides the guidance start screen into two areas. In one of the divided areas (for example, the main area A1 of the home screen SC20 in FIG. 7), the guidance start screen is displayed (i.e., each object included in the guidance start screen is arranged). In the other area (for example, the guidance area A2 of the home screen SC20 in FIG. 7), messages are displayed as needed to guide the user through operations to be performed. Furthermore, as shown in the home screen SC20 in FIG. 7, the guidance area A2 is arranged with an area R1 for receiving input of a character string from the user and a transmission object OB100 for transmitting the character string input in area R1 to the AI ​​server 200.

[0045] In S12, the CPU 32 monitors whether the selection of the transmission object OB100 is accepted after the input of a character string into the input area R1 is accepted. The character string entered here corresponds to the process desired by the user (for example, "I would like to print the document file in black and white on both sides"). If the selection of the transmission object OB100 is accepted after the input of the character string is accepted, the CPU 32 determines YES in S12 and proceeds to S14.

[0046] In S14, the CPU 32 transmits the screen ID and the input character string to the AI ​​server 200 via the communication I / F 16. The screen ID is information that identifies the guidance start screen. When the AI ​​server 200 receives the screen ID and the character string from the terminal 10, it determines whether it is possible to start providing guidance to the user based on the received character string. Specifically, the AI ​​server 200 determines whether it is possible to uniquely identify the process desired by the user from the received character string. For example, if the input character string is "I want to print a document in black and white on both sides," the AI ​​server 200 determines that the user wants to print a document file on both sides. On the other hand, if the input character string is "I want to print in black and white on both sides," the AI ​​server 200 determines that it is impossible to uniquely identify the process desired by the user. This is because in such a case, the AI ​​server 200 cannot determine which type of file the user wants to print—a document file, a photo file, or an audio file—is desired to be printed. If it determines that it is possible to start providing guidance, the AI ​​server 200 transmits the text, guidance information, and a guidance ID (not shown) to the terminal 10. The text includes a message corresponding to each of the one or more screens and a completion message displayed when a series of processes is completed. The guidance information includes, for each of the one or more screens, a combination of a screen ID identifying the screen and an object ID identifying an object to be highlighted on the screen (hereinafter referred to as a "target object"). Note that, as will be described in detail later, some of the one or more screens may not include an object to be highlighted. In this case, the combination information corresponding to the screen is a combination of a screen ID indicating the screen and information indicating that no object to be highlighted exists on the screen. As described above, the trained model has previously learned the correspondence between text and guidance information (i.e., the combination information of a screen ID and an object ID) for a first natural sentence. For example, the trained model has previously trained using combinations of text and guidance information for various natural sentences assumed to be the first natural sentence as training data. Similarly, the trained model has previously learned the correspondence between text and guidance information (i.e., the combination information of a screen ID and an object ID) for a second natural sentence.In a modified example, a trained model may implement a similar function by using Retrieval-Augmented Generation (RAG) and referencing the correspondence between text and guidance information for the first natural language sentence or the second natural language sentence. In another modified example, the present technology may be implemented without using a trained model (i.e., an AI function), but by referencing the correspondence between text and guidance information for the first natural language sentence or the second natural language sentence (e.g., a table, a database, etc.) in the same way as a chatbot. The guidance ID is information that identifies the collection of the above text and guidance information.

[0047] On the other hand, if the AI ​​server 200 determines that it is not possible to start guidance, it sends an additional question to the terminal 10 to request information necessary to enable guidance to begin. Examples of the additional question are as follows: For example, consider a situation in which the guidance start screen is the home screen SC1 and the user inputs the string "I would like to print double-sided in black and white" into the input field R1. As described above, in this situation, the AI ​​server 200 cannot determine what type of file the user wants to print. In this case, the AI ​​server 200 sends the additional question, for example, "Is the file you want to print a document file?" to the terminal 10.

[0048] In S16, the CPU 32 determines whether text and guidance information or an additional question has been received from the AI ​​server 200. If text and guidance information has been received from the AI ​​server 200 (YES in S16), the CPU 32 proceeds to S20, and if an additional question has been received from the AI ​​server 200 (NO in S16), the CPU 32 proceeds to S40.

[0049] In S20, the CPU 32 first identifies a message corresponding to the first screen from the text and identifies combination information corresponding to the first screen from the guidance information. Next, the CPU 32 determines whether the identified combination information includes an object ID. If the identified combination information includes an object ID, the CPU 32 highlights the target object identified by the object ID on the guidance start screen displayed in the main area A1. In this embodiment, as shown in FIG. 7 and other figures, the CPU 32 displays the target object in a lighter color (white in this case) than the background color of the main area A1 (gray in this case). Furthermore, the CPU 32 displays the identified message in the guidance area A2. On the other hand, if the identified combination information does not include an object ID, the CPU 32 does not highlight the target object in the main area A1. Furthermore, the CPU 32 displays the identified message in the guidance area A2.

[0050] In S22, the CPU 32 determines whether selection of the transmission object OB100 is accepted after input of a character string into the input area R1 of the guidance area A2 is accepted. If selection of the transmission object OB100 is accepted after input of a character string into the input area R1 is accepted, the CPU 32 determines YES in S22 and returns to S14. Here, a situation in which a YES determination is made in S22 is assumed to be a situation in which the user desires a process different from the process already input in S12 (see case C described below). On the other hand, if input of a character string into the input area R1 is not accepted, the CPU 32 determines NO in S22 and proceeds to S24.

[0051] In S24, the CPU 32 determines whether the selection of the correct object has been accepted. Specifically, the CPU 32 determines whether the selection of the highlighted target object has been accepted. If the selection of the target object has been accepted, the CPU 32 determines YES in S24 and proceeds to S30. On the other hand, if the selection of an object other than the target object has been accepted, the CPU 32 determines NO in S24 and proceeds to S50. Note that if no highlighting is performed in the main area A1, the CPU 32 determines YES in S24 regardless of which object is selected on the screen displayed in the main area A1. Note that the CPU 32 displays a screen corresponding to the selected object in the main area A1 regardless of the determination result in S24.

[0052] In S30, the CPU 32 determines whether a series of operations has been completed. Specifically, the CPU 32 determines whether all screens have been displayed in accordance with the text and guidance information received in S16. If all screens have been displayed, the CPU 32 determines YES in S30 and proceeds to S32. On the other hand, if all screens have not been displayed, the CPU 32 executes the process of S20 in accordance with the message and combination information corresponding to the next screen.

[0053] In S32, the CPU 32 displays the completion message included in the text received in S16 in the guidance area A2. When the process of S32 ends, the process of FIG.

[0054] In addition, in S40, the CPU 32 displays the follow-up question received in S16 in the guidance area R2. When the process of S40 ends, the process returns to the monitoring process of S12.

[0055] Furthermore, in S50, the CPU 32 identifies the screen ID of the screen displayed in the main area A1 in response to the selection of an incorrect object, and transmits the screen ID and a guidance ID to the AI ​​server 200 via the communication I / F 16. Upon receiving the screen ID and the guidance ID, the AI ​​server 200 transmits warning text to the terminal 10 in response to determining that the screen ID is not included in the guidance information identified by the guidance ID. The warning text includes a warning message to notify the user that an incorrect object has been selected by the user.

[0056] In S52, the CPU 32 receives the warning text from the AI ​​server 200 via the communication I / F 16.

[0057] In S54, the CPU 32 displays the warning message included in the received warning text in the guidance area A2. When the process of S54 ends, the process returns to the monitoring process of S12.

[0058] (Case A; Figures 7 to 10) Next, with reference to Figures 7 to 10, each screen displayed by the terminal 10 as a result of the execution of the guidance process of Figure 6 will be described. When the terminal 10 accepts selection of the bot button OB19 in the home screen SC1 (see Figure 2) at T200 of Figure 2 (trigger of Figure 6), it displays the home screen SC20 of Figure 7 on the display unit 14 (S10). As shown in Figure 7, the home screen SC20 is divided into a main area A1 and a guidance area A2.

[0059] The main area A1 of the home screen SC20 displays a screen that is the same as a portion of the home screen SC1. The remaining portion of the home screen SC1 is not displayed in the main area A1, but the user can slide the main area A1 up or down to display the non-displayed portion of the home screen SC1 in the main area A1. This also applies to the main areas A1 of the subsequent screens.

[0060] A message MB1, an input area R1, and a transmission object OB100 are arranged in the guidance area A2 of the home screen SC20. The message MB1 is a message that asks the user what kind of processing the user wants to perform.

[0061] In T201, the terminal 10 accepts input of the character string "I want to print the document in black and white on both sides" into the input area R1 from the user, and then accepts selection of the transmission object OB100 (YES in S12). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). In this case, the text includes nine messages MB2 to MB10 and a completion message MB11. Furthermore, the guidance information includes nine screen IDs (i.e., information identifying each of the screens SC1 to SC4, the dialog D1, the screen SC5, the dialog D2, and the screens SC6 and SC3) and six object IDs (i.e., information identifying each of the objects OB13, OB32, OB43, OB44, OB47, and OB33). Of the nine screen IDs, three screen IDs that identify the screen SC22, the dialogs D1, and D2 are combined with information indicating that no object to be highlighted exists.

[0062] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies from the text a message (i.e., message MB2) corresponding to the first screen (i.e., home screen SC1) and identifies combination information corresponding to the first screen. Since the combination information includes an object ID for identifying the functional object OB13, the functional object OB13 identified by the object ID included in the combination information is highlighted in the home screen SC1 displayed in the main area A1 of the home screen SC21 (S20). Furthermore, in the guidance area A2 of the home screen SC21, a message MU1 indicating the entered character string and a message MB2 corresponding to the first screen are further displayed, compared to the guidance area A2 of the home screen SC20 (S20). By looking at the message MB2, the user can recognize the operation to be performed by the user. In particular, in the guidance area A2, new messages are displayed in order below the older messages, and the new messages are automatically scrolled so that the user can check them. This allows the user to easily check the latest message. Although some of the old messages MB1 are not displayed in the guidance area A2 of the home screen SC21, the old messages can be displayed by the user sliding the guidance area A2 up or down. This also applies to the guidance areas A2 of the following screens.

[0063] The processing of T210 is the same as the processing of T10 in Fig. 2. In this case, the terminal 10 displays the file selection screen SC22 on the display unit 14. A part of the screen of the file selection screen SC2 is displayed in the main area A1 of the file selection screen SC22. In T210, the highlighted functional object OB13 (i.e., the correct object) is selected (NO in S22 and YES in S24 in Fig. 6), and not all of the screens have been displayed according to the received text and guidance information (NO in S30). Therefore, the terminal 10 determines not to highlight the next screen (i.e., the file selection screen SC2) in the main area A1 of the file selection screen SC22 according to the message and combination information corresponding to the next screen, and displays a message MB3 in the guidance area A2 of the file selection screen SC22.

[0064] The subsequent processes of T220 to T290 in FIGS. 7 to 10 are the same as the processes of T20 to T90 in FIGS. 2 to 4. Furthermore, in each of screens SC23 to SC27 in FIGS. 8 to 10, screens similar to SC3 to SC6 and SC3 in FIGS. 2 to 4 are displayed in the main area A1. Furthermore, in the main area A1 of each of screens SC24 and SC25, the dialogues D1 and D2 displayed superimposed on each of screens SC24 and SC25 are the same as the dialogues D1 and D2 displayed superimposed on each of screens SC4 and SC5. Furthermore, in the main area A1 and each of dialogs D1 and D2 of each of screens SC23 to SC27, it is determined whether or not to highlight based on the corresponding combination information included in the received guidance information. Furthermore, the corresponding message included in the received text is newly displayed in the guidance area A2 of each of screens SC23 to SC27.

[0065] When terminal 10 accepts selection of highlighted print object OB33 from the user at T290 in FIG. 10, it sends the print data to MFP 100 and determines that the series of operations is complete (i.e., all screens have been displayed) (YES in S30 in FIG. 6), and displays home screen SC28 on display unit 14. Main area A1 of home screen SC28 is the same as main area A1 of home screen SC20. In guidance area A2 of home screen SC28, a completion message MB11 included in the received text is displayed (S32).

[0066] Thus, with the above configuration, when the terminal 10 receives input of the character string "I want to print the document in black and white on both sides" (YES in S12 of FIG. 6), it supplies the character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). Then, the terminal 10 sequentially displays each of the screens SC21 to SC28 of FIGS. 7 to 10 in accordance with the text and guidance information. As shown in FIGS. 7 to 10, an object identified by the object ID included in the guidance information (i.e., the target object) is highlighted on at least a portion of each of the sequentially displayed screens. This allows the user to easily recognize that the highlighted object should be selected. This improves user convenience.

[0067] Also, consider a comparative example in which a chatbot is used to sequentially display print operation guidance to a terminal user via chat. In this comparative example, the user can execute printing of an image by sequentially inputting print settings, etc., on the chatbot's screen in response to requests (i.e., questions) from the chatbot. However, in this comparative example, the user must interact with the chatbot each time printing is executed. On the other hand, in the configuration of this embodiment, the object to be operated is highlighted in the main area A1 where the normal screen shown in FIGS. 2 to 4 is located. This allows the user to become familiar with operations on the normal screen. As a result, the user may be able to execute printing on the MFP 100 without needing guidance the next time the user executes printing on the MFP 100.

[0068] (Case B; Figure 11) Next, a case B in which an incorrect object is selected will be described with reference to Fig. 11. The home screen SC21 in Fig. 11 is the same as the home screen SC21 in Fig. 7.

[0069] In T310, the terminal 10 accepts selection of a functional object OB16 corresponding to scanning from the user. As shown in FIG. 11, the functional object OB16 is not highlighted in the main area A1 of the home screen SC21. That is, the functional object OB16 is an incorrect object (NO in S24 of FIG. 6). In response to the selection of the functional object OB16, the terminal 10 displays a scan setting screen SC31 on the display unit 14. The scan setting screen SC11 (see FIG. 5) is displayed in the main area A1 of the scan setting screen SC31.

[0070] Furthermore, in response to the selection of an incorrect object, the terminal 10 transmits the screen ID of the scan setting screen SC11 displayed in the main area A1 of the scan setting screen SC31 to the AI ​​server 200 (S50), and receives warning text including a warning message MB101 from the AI ​​server 200 (S52). The terminal 10 then displays the warning message MB101 in the guidance area A2 of the scan setting screen SC31 (S54). By viewing the warning message MB101, the user can recognize that an incorrect object has been selected. Note that the user can also continue to perform an operation on the scan setting screen SC31 to cause the MFP 100 to scan the document.

[0071] In T320, the terminal 10 accepts the user's input of the character string "Return to guide" into the input area R1, and then accepts the selection of the transmission object OB100 (YES in S12 after S54). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). The text of this case includes the messages included in the text of case A as well as message MB102. Furthermore, the guidance information includes combination information that combines a screen ID that identifies the scan setting screen SC11 and an object ID that identifies the back button OB115, in addition to the combination information included in the guidance information of case A.

[0072] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies a message corresponding to the first screen (i.e., the scan setting screen SC11) and identifies combination information corresponding to the first screen. Because the combination information includes an object ID that identifies the back button OB115, the back button OB115 is highlighted in the main area A1 of the scan setting screen SC32. Also, in the guidance area A2 of the scan setting screen SC32, a message MU101 indicating the entered character string and a message MB102 corresponding to the first screen included in the received text are displayed.

[0073] When the terminal 10 receives the user's selection of the highlighted back button OB115 at T330, it determines that the correct object has been selected (YES in S24 of FIG. 6), and displays a screen similar to the home screen SC21 of FIG. 7 on the display unit 14. The screen displayed here differs from the home screen SC21 of FIG. 7 in the message displayed in the guidance area A2. In this way, when an incorrect object is selected by the user, the terminal 10 notifies the user of this and allows the user to return to the guidance. This allows the processing desired by the user to be executed appropriately.

[0074] (Case C; Figures 12 and 13) Next, a case C in which the user changes the processing desired will be described with reference to Fig. 12 and Fig. 13. The home screen SC21 in Fig. 12 is the same as the home screen SC21 in Fig. 7.

[0075] In T410, the terminal 10 accepts input of the character string "I want to scan in monochrome" into the input field R1 from the user, and then accepts selection of the transmission object OB100 (YES in S22). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). In this case, the text includes four messages MB201 to MB204 and a completion message MB205. The guidance information also includes four screen IDs (i.e., information identifying screens SC1 and SC11, dialog D11, and screen SC12, respectively) and three object IDs (i.e., information identifying objects OB16, OB111, and OB114, respectively). Of the four screen IDs, the screen ID identifying dialog D11 is combined with information indicating that no object to be highlighted exists.

[0076] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies a message corresponding to the first screen (i.e., the home screen SC1) and also identifies combination information corresponding to the first screen. Since the combination information includes an object ID that identifies the function object OB16, the function object OB16 is highlighted in the main area A1 of the home screen SC41. Furthermore, in the guidance area A2 of the home screen SC41, a message MU201 indicating the entered character string and a message MB201 corresponding to the first screen included in the received text are displayed.

[0077] The processing of T420 is the same as the processing of T100 in FIG. 2. In this case, the terminal 10 displays the scan setting screen SC42 on the display unit 14. The scan setting screen SC11 is displayed in the main area A1 of the scan setting screen SC42. In T420, the highlighted function object OB16 (i.e., the correct object) is selected (NO in S22 and YES in S24 in FIG. 6), and not all screens have been displayed according to the received text and guidance information (NO in S30). Therefore, the terminal 10 determines to highlight the item object OB111 in the main area A1 of the scan setting screen SC42 according to the message and combination information corresponding to the next screen (i.e., the scan setting screen SC11), and displays a message MB202 in the guidance area A2 of the scan setting screen SC42.

[0078] The subsequent processing at T430 in FIG. 12 and the processing at T440 and T450 in FIG. 13 are respectively the same as the processing at T110 to T130 in FIG. 5. Furthermore, in each of screens SC42 and SC43 in FIG. 13, a screen similar to SC11 and SC12 in FIG. 5 is displayed in the main area A1. Furthermore, in the main area A1 of scan setting screen SC42, the dialog D11 displayed superimposed on said screen SC42 is the same as the dialog D11 displayed superimposed on scan setting screen SC11. Furthermore, in the main area A1 and dialog D11 of each of screens SC42 and SC43, whether or not to highlight is determined based on the corresponding combination information included in the received guidance information. Furthermore, a corresponding message included in the received text is newly displayed in the guidance area A2 of each of screens SC42 and SC43.

[0079] When the terminal 10 receives selection of the highlighted scan object OB114 from the user at T450 in FIG. 13, it sends a scan instruction to the MFP100 and determines that the series of operations is complete (i.e., all screens have been displayed) (YES in S30 in FIG. 6), and displays the home screen SC44 on the display unit 14. The main area A1 of the home screen SC44 is the same as the main area A1 of the home screen SC20 (see FIG. 7). The guidance area A2 of the home screen SC44 displays a completion message MB205 included in the received text (S32). In this way, even if the process desired by the user is changed midway, the terminal 10 can appropriately guide the user on the operation to execute the changed process.

[0080] (Case D; Figure 14) Next, case D will be described with reference to Fig. 14. Case D differs from case A in the screen on which the bot button OB19 is selected (that is, the guidance start screen).

[0081] When the terminal 10 receives a selection of the bot button OB19 included in the print preview screen SC3 from the user at T500 in Fig. 2 (trigger in Fig. 6), the terminal 10 displays the print preview screen SC50 in Fig. 14 on the display unit 14 (S10). As shown in Fig. 14, the print preview screen SC50 is divided into a main area A1 and a guide area A2.

[0082] The main area A1 of print preview screen SC50 displays the same screen as part of print preview screen SC3. Message MB301 displayed in guide area A2 of print preview screen SC50 is the same as message MB1 in case A (see FIG. 7).

[0083] In T510, the terminal 10 accepts the user's input of the character string "I want to print the document in black and white on both sides" into the input area R1, and then accepts the selection of the transmission object OB100 (YES in S12). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). In this case, the text includes seven messages MB302, MB5 to MB10, and a completion message MB11. The guidance information also includes seven screen IDs (i.e., information identifying each of the screens SC3, SC4, dialog D1, screen SC5, dialog D2, screen SC6, and SC3) and five object IDs (i.e., information identifying each of the objects OB32, OB43, OB44, OB47, and OB33). Of the seven screen IDs, the two screen IDs identifying the dialogs D1 and D2 are combined with information indicating that there are no objects to be highlighted. If the AI ​​server 200 receives input of the string "I want to print double-sided in black and white" from the user at step T510, the AI ​​server 200 determines that it is possible to start guidance, because at step T510, the document file corresponding to the file name F1 has already been selected.

[0084] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies from the text a message (i.e., message MB302) corresponding to the first screen (i.e., print preview screen SC3) and also identifies combination information corresponding to the first screen. Since the combination information includes an object ID that identifies the setting object OB32, the setting object OB32 is highlighted in the main area A1 of the print preview screen SC51. Furthermore, in the guidance area A2 of the print preview screen SC51, compared to the guidance area A2 of the print preview screen SC50, a message MU301 indicating the entered character string and a message MB302 corresponding to the first screen included in the received text are further displayed. Subsequent processing is the same as in Case A.

[0085] As shown in FIGS. 2 to 4, each of the screens SC1 to SC6 in FIGS. 2 to 4 includes a bot button OB19. Therefore, operation guidance can be started from any of the screens SC1 to SC6 in FIGS. 2 to 4. For example, as in case D, operation guidance can be started from the print preview screen SC3. Therefore, the user can select the bot button OB19 on the screen that is displayed when the user becomes unsure of the operation to perform. Therefore, compared to the comparative example in which only the home screen SC1 includes the bot button OB19, when the user becomes unsure of the operation to perform, the user does not need to return to the home screen SC1 to start guidance. This improves user convenience.

[0086] (Correspondence) The MFP 100 is an example of an "image processing device." The CPU 32 and the application 38 are examples of a "computer" and a "computer program," respectively. The AI ​​server 200 is an example of a "natural language processing unit." The display unit 14 of the terminal 10 is an example of a "display device" and an "output device." The process of performing double-sided printing of a document in black and white is an example of a "first process." In Case A, the home screen SC1 is an example of a "second screen." In Case A, the screens SC1 to SC6 and the dialogs D1 and D2 in Figures 2 to 4 are examples of "M screens." In Case A, the screens SC1 to SC6 are an example of "M1 screens." In Case D, the print preview screen SC3 is an example of a "second screen." In Case D, the screens SC3 to SC6 and the dialogs D1 and D2 in Figures 2 to 4 are examples of "M screens." In Case D, the print preview screen SC3 is an example of a "second screen." In Case D, the screens SC3 to SC6 and the dialogs D1 and D2 in Figures 2 to 4 are examples of "M screens." In Case D, the print preview screen SC3 is an example of a "second screen." A function object OB13 on the home screen SC1 is an example of a "first object." The character string "I want to print the document in black and white on both sides" is an example of a "first natural sentence." The guidance information received from the AI ​​server 200 after processing T201 in FIG. 7 is an example of a "first object information." In particular, the object ID included in the guidance information is an example of a "first identification information." The main area A1 and the guidance area A2 are examples of a "first area" and a "second area," respectively. The home screen SC21 is an example of a "first screen." The function object OB16 selected in T310 in FIG. 11 is an example of a "different object." The warning message MB101 is an example of "warning information." The bot button OB19 is an example of a "predetermined object."

[0087] The process of scanning in monochrome is an example of a "second process." The home screen SC1 in FIG. 2 and the screen SC11, dialog D11, and screen SC12 in FIG. 5 are examples of "N screens." The home screen SC1 in FIG. 2 and the screens SC11 and SC12 in FIG. 5 are examples of "N1 screens." The function object OB16 on the home screen SC1 is an example of a "second object." The natural sentence "I would like to scan in monochrome" is an example of a "second natural sentence." The guidance information received from the AI ​​server 200 after the process of T410 in FIG. 12 is an example of "second object information." In particular, the object ID included in the guidance information is an example of "second identification information."

[0088] The process of displaying each screen in FIGS. 2 to 4 is an example of a process executed by a "first display control unit." The process of S12 in FIG. 6 is an example of a process executed by a "first receiving unit." The process of S14 in FIG. 6 is an example of a process executed by a "first (and second) supply unit." The process of S16 in FIG. 6 is an example of a process executed by a "first (and second) acquisition unit." The processes of S20 to S30 in FIG. 6 and the process of displaying each screen in FIGS. 7 to 10 are examples of a process executed by a "second display control unit." In particular, the process of S20 in FIG. 6 is an example of a process executed by a "first output control unit." The process of S54 in FIG. 6 is an example of a process executed by a "second output control unit." The process of displaying the home screen SC1 in FIG. 2 and each screen in FIG. 5 is an example of a process executed by a "third display control unit." The process of S22 in FIG. 6 is an example of a process executed by a "second receiving unit." The processes of S20 to S30 in Fig. 6 and the processes of displaying the screens in Fig. 12 and Fig. 13 are examples of processes executed by a "fourth display control unit." The process of S10 in Fig. 6 is an example of processes executed by a "fifth display control unit."

[0089] Although specific examples of the technology disclosed in this specification have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples exemplified above. Modifications of the above examples are listed below.

[0090] (Variation 1) In the above embodiment, the terminal 10 has been described as providing guidance on operations when causing the MFP 100 to execute printing or scanning. The technology of this embodiment is also useful for providing guidance on operations when executing processes other than printing or scanning. Here, "processes other than printing or scanning" refer to processes related to the MFP 100, such as processes for registering the MFP 100 with a subscription service and processes for establishing a wireless connection between the MFP 100 and an external device. Generally speaking, the "first process" may be any "process related to an image processing device."

[0091] (Modification 2) Terminal 10 may accept voice input from the user instead of inputting a character string into input area R1 of guidance area A2.

[0092] (Variation 3) In the above embodiment, an example of application 38 being a native application installed on terminal 10 has been described. In a variation, application 38 may be an application installed on MFP 100, for example. In this case, MFP 100 may display the various screens of the above embodiment on display unit 114 of MFP 100. In this variation, CPU 132 of MFP 100 is an example of a "computer," and display unit 114 of MFP 100 is an example of a "display device."

[0093] (Variation 4) Instead of app 38, which is a native app installed on terminal 10, for example, a web app may be used, a cloud app may be used, or an app realized by a combination of at least two of a native app, a web app, and a cloud app may be used. Here, "a combination of at least two" means, for example, a combination of a native app and a web app (a so-called hybrid app), a combination of a native app and a cloud app, etc.

[0094] For example, in an example where a web application is used instead of the application 38, the various processes of the above embodiment are realized by a cloud print server (not shown). In this modification, the CPU of the cloud print server is an example of a "computer," and the web application is an example of a "computer program." The display unit 14 of the terminal 10 is an example of a "display device." The cloud print server operates according to the web application on the server. The cloud print server transmits corresponding screen data (e.g., web page data, etc.) to the terminal 10, thereby displaying the various screens (SC1 to SC6, SC11, SC12, SC20 to SC28, etc.) described in the above embodiment on the display unit 14 of the terminal 10 via a web browser or WebView. The cloud print server receives information indicating an operation accepted on the various screens (e.g., an operation ID), various data for which a selection has been accepted, etc. from the terminal 10, and executes processing according to the operation. The operation ID is, for example, information indicating that an operation (i.e., selection) of an object such as object OB33 has been accepted. The various data for which selection has been accepted include, for example, the selected device ID "dv1", the target file specified on the file selection screen SC2, data indicating the items selected on the print setting screen SC4, etc. That is, the cloud print server accepts operations on the various screens through the reception of the above-mentioned operation ID, various data, etc. When the cloud print server receives from the terminal 10 an operation ID indicating that the operation of the print object OB33 has been accepted, it transmits print data to the MFP 100 identified by the device ID "dv1". Note that in this modified example, the device ID "dv1" may include, in addition to the model name, a token for the cloud print server to communicate with the MFP 100, authentication information, the global IP address to which the MFP 100 belongs, etc.

[0095] Furthermore, if the cloud print server cannot acquire local information such as the device ID from a web browser or the like due to security settings, a hybrid application may be created by combining a web application with a native application installed on the terminal 10. In this case, the native application can acquire local information such as the device ID stored in the memory 34 of the terminal 10 and send that information to the cloud print server. In this case, the cloud print server performs the same process as described above via the native application. The same applies when a cloud application is used instead of a web application.

[0096] (Variant 5) The communication system 2 may not include the AI ​​server 200. In this variant, for example, an AI engine that executes various processes in place of the AI ​​server 200 described above may be provided in the terminal 10. The AI ​​engine includes a trained model. In this variant, the terminal 10 includes a "natural language processing unit." In another variant, an AI engine that executes various processes in place of the AI ​​server 200 described above may be provided in the MFP 100. The AI ​​engine includes a trained model. In this variant, the MFP 100 includes a "natural language processing unit."

[0097] (Variation 6) In the above embodiment, the target object is highlighted by displaying the target object and its surrounding background in a lighter color than the background color of the main area A1. The highlighting method is not limited to the above embodiment. For example, the target object may be highlighted by displaying an icon (e.g., an arrow) pointing to the target object. In another variation, the target object may be highlighted by displaying it in a swaying manner. In another variation, the target object may be highlighted by blinking the target object. In another variation, the target object may be highlighted by making the text color of the target object different from the text color of other objects.

[0098] (Variation 7) Terminal 10 can omit the process of S10 in Fig. 6. That is, terminal 10 does not need to divide the screen into two areas. In this case, terminal 10 can display each screen that does not include guidance area A2. In this variation, the "first output control unit" can be omitted.

[0099] (Variation 8) In S20 of Fig. 6, terminal 10 may output the message by voice instead of displaying the message in guidance area A2. In this variation, the process of outputting the message by voice is an example of a process executed by the "first output control unit."

[0100] (Variant 9) The terminal 10 does not have to receive text from the AI ​​server 200 in S16 of Figure 6. In this variant, for example, the terminal 10 may generate a message to be displayed in the guidance area A2 based on the object information received from the AI ​​server 200.

[0101] (Modification 10) In the above embodiment, the terminal 10 divided the home screen SC1, etc., and arranged the main area A1 on the upper side and the guidance area A2 on the lower side. The arrangement of the divided areas A1 and A2 is not limited to the form of the above embodiment. For example, the main area A1 may be arranged on the lower side and the guidance area A2 on the upper side. Furthermore, the areas may be divided horizontally instead of vertically. That is, the terminal 10 may divide the home screen SC1, etc., horizontally, and arrange one of the main area A1 and the guidance area A2 on the left side and the other on the right side.

[0102] (Modification 11) In a situation where a new message is displayed in the guidance area A2, the guidance area A2 does not have to be automatically scrolled.

[0103] (Variation 12) The terminal 10 can omit the processes of S50 to S54 in Fig. 6. That is, even if an incorrect object is selected by the user, the terminal 10 does not need to display the warning message MB101. In this variation, the "second output control unit" can be omitted.

[0104] (Variation 13) In the above embodiment, the bot button OB19 was included in each of the screens SC1 to SC6 in Figures 2 to 4. Instead, in a variation, the bot button OB19 may be included only in, for example, the home screen SC1, and may not be included in each of the screens SC2 to SC6.

[0105] (Variation 14) In the above embodiment, the dialogs D1, D2, and D11 were not highlighted. In a variation, the objects to be selected may be highlighted in the dialogs D1, D2, and D11. In this case, for example, when there are two options, "long edge binding" and "short edge binding," as in the dialog D1, the two objects may be highlighted.

[0106] (Variant 15) In the above embodiment, the processing of each step in Figure 6 is realized by software (e.g., OS 36, application 38, programs 136, 236), but at least one of these processes may be realized by hardware such as a logic circuit.

[0107] The technical elements described in this specification or drawings may exhibit technical utility either alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technologies illustrated in this specification or drawings may achieve multiple objectives simultaneously, and achieving one of those objectives alone is technically useful.

[0108] In the scope of the claims at the time of filing, even if each claim depends on only some of the claims, it is not limited to the fact that each claim can depend on only those some of the claims. To the extent that there is no technical contradiction, each claim can also depend on other claims that were not dependent at the time of filing. In other words, the technology of each claim can be combined in various ways as follows: (Item 1) A computer program for causing an image processing device to execute image processing, The computer program controls a computer to operate in the following manner: a first display control unit that sequentially displays M screens (where M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device on a display device, each of the M views includes one or more objects; When a first object corresponding to the first process among the one or more objects is selected on each of the M screens, the first process is executed. the first display control unit; a first receiving unit that receives an input of a first natural sentence related to the first process from a user; a first supply unit that supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit when input of the first natural sentence is accepted; a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens (M1 is an integer between 2 and M) out of the M screens, first identification information that identifies the first object among the one or more objects included in the screen; a second display control unit that, when the first object information is acquired from the natural language processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, and the first object identified by the first identification information is highlighted on each of the M1 screens; and A computer program that functions as a (Item 2) The computer program further causes the computer to: The computer program described in item 1 functions as a first output control unit that causes an output device to output a message prompting a user to select the first object that is highlighted on each of the M1 screens. (Item 3) 3. The computer program according to claim 2, wherein the first acquisition unit acquires the first object information and the message from the natural language processing unit. (Item 4) the second display control unit sequentially displays the M screens in a first area within the display device; Item 2 or 3. The computer program according to item 2 or 3, wherein the first output control unit displays the message in a second area within the display device, which is the output device, and the second area is different from the first area. (Item 5) Item 5. The computer program of item 4, wherein the first output control unit sequentially displays M1 messages corresponding to the M1 screens in the second area each time the M1 screens are sequentially displayed in the first area. (Item 6) The computer program further causes the computer to: A computer program according to any one of items 1 to 5, which functions as a second output control unit that causes an output device to output warning information when an object different from the first object is selected on a first screen of the M1 screens. (Item 7) The computer program further causes the computer to: a third display control unit that sequentially displays, on the display device, N screens (where N is an integer equal to or greater than 2) corresponding to a second process related to the image processing device and different from the first process, each of the N screens includes one or more objects; the third display control unit that executes the second process when a second object corresponding to the second process among the one or more objects is selected on each of the N screens; a second receiving unit that receives input of a second natural sentence related to the second process in a state in which any one of the M screens including the M1 screens according to the first object information is displayed on the display device; a second supply unit that supplies second natural sentence data representing the second natural sentence to the natural sentence processing unit when the input of the second natural sentence is accepted; a second acquisition unit that acquires, from the natural language processing unit, second object information based on the second natural language data, the second object information being different from the first object information, wherein the second object information includes, for each of N1 screens (N1 is an integer between 2 and N) out of the N screens corresponding to the second processing, second identification information that identifies the second object corresponding to the second processing among the one or more objects included on the screen; a fourth display control unit that, when the second object information is acquired from the natural language processing unit, causes the display device to sequentially display N1 screens according to the second object information, and the second object identified by the second identification information is highlighted on each of the N1 screens; and 7. The computer program according to any one of items 1 to 6, (Item 8) each of a plurality of screens that can be displayed by the computer program includes a predetermined object; The computer program further causes the computer to: A computer program according to any one of items 1 to 7, which functions as a fifth display control unit that causes the display device to display an input area for accepting input of natural language when the specified object is selected. (Item 9) Item 8. A computer program according to item 8, wherein when the specified object included in a second screen among the plurality of screens is selected and the first natural sentence is input into the input area, each of the M screens is a screen to be displayed after the second screen. (Item 10) the image processing device includes a print execution unit; 10. The computer program according to claim 1, wherein the first process is a process of causing the print execution unit to print an image. [Explanation of symbols]

[0109] 2: Communication system, 4: LAN, 6: Internet, 10: Terminal, 12, 112: Operation unit, 14, 114: Display unit, 16, 116, 216: Communication I / F, 30, 130, 230: Control unit, 32, 132: CPU, 34, 134, 234: Memory, 36: OS program, 38: Application program, 100: MFP, 118: Print execution unit, 120: Scan execution unit, 200: AI server, 232: Processor

Claims

1. A computer program for causing an image processing device to execute image processing, The computer program controls a computer to operate in the following manner: a first display control unit that sequentially displays M screens (where M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device on a display device, each of the M screens includes one or more objects; When a first object corresponding to the first process among the one or more objects is selected on each of the M screens, the first process is executed. the first display control unit; a first receiving unit that receives an input of a first natural sentence related to the first process from a user; a first supply unit that supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is accepted; a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens (M1 is an integer between 2 and M) out of the M screens, first identification information that identifies the first object among the one or more objects included in the screen; a second display control unit that, when the first object information is acquired from the natural language processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, and the first object identified by the first identification information is highlighted on each of the M1 screens; and A computer program that functions as a

2. The computer program further causes the computer to: The computer program according to claim 1 , which functions as a first output control unit that causes an output device to output a message prompting a user to select the first object that is highlighted on each of the M1 screens.

3. The computer program according to claim 2 , wherein the first acquisition unit acquires the first object information and the message from the natural language processing unit.

4. the second display control unit sequentially displays the M screens in a first area within the display device; The computer program according to claim 2 , wherein the first output control unit displays the message in a second area within the display device that is the output device, the second area being different from the first area.

5. 5. The computer program according to claim 4, wherein the first output control unit sequentially displays the M1 messages corresponding to the M1 screens in the second area each time the M1 screens are sequentially displayed in the first area.

6. The computer program further causes the computer to:

2. The computer program of claim 1, further comprising: a computer program configured to function as a second output control unit that outputs warning information to an output device when an object other than the first object is selected on a first screen among the M1 screens.

7. The computer program further causes the computer to: a third display control unit that sequentially displays, on the display device, N screens (where N is an integer equal to or greater than 2) corresponding to a second process related to the image processing device and different from the first process; each of the N screens includes one or more objects; the third display control unit that executes the second process when a second object corresponding to the second process among the one or more objects is selected on each of the N screens; a second receiving unit that receives input of a second natural sentence related to the second process in a state in which any of the M screens including the M1 screens according to the first object information is displayed on the display device; a second supply unit that supplies second natural sentence data representing the second natural sentence to the natural sentence processing unit when the input of the second natural sentence is accepted; a second acquisition unit that acquires, from the natural language processing unit, second object information based on the second natural language data, the second object information being different from the first object information, and the second object information including, for each of N1 screens (N1 is an integer between 2 and N) out of the N screens corresponding to the second processing, second identification information that identifies the second object corresponding to the second processing among the one or more objects included on the screen; a fourth display control unit that, when the second object information is acquired from the natural language processing unit, causes the display device to sequentially display N1 screens according to the second object information, and the second object identified by the second identification information is highlighted on each of the N1 screens; and The computer program of claim 1 ,

8. each of a plurality of screens that can be displayed by the computer program includes a predetermined object; The computer program further causes the computer to: The computer program according to claim 1 , further comprising a fifth display control unit that causes the display device to display an input area for accepting input of natural language when the predetermined object is selected.

9. 9. The computer program according to claim 8, wherein when the predetermined object included in a second screen of the plurality of screens is selected and the first natural sentence is input into the input area, each of the M screens is a screen to be displayed after the second screen.

10. the image processing device includes a print execution unit; The computer program product according to claim 1 , wherein the first process is a process of causing the print execution unit to print an image.

11. An apparatus for causing an image processing device to perform image processing, a first display control unit that sequentially displays M screens (where M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device on a display device, each of the M screens includes one or more objects; When a first object corresponding to the first process among the one or more objects is selected on each of the M screens, the first process is executed. the first display control unit; a first receiving unit that receives an input of a first natural sentence related to the first process from a user; a first supply unit that supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is accepted; a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens (M1 is an integer between 2 and M) out of the M screens, first identification information that identifies the first object among the one or more objects included in the screen; a second display control unit that, when the first object information is acquired from the natural language processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, and the first object identified by the first identification information is highlighted on each of the M1 screens; and An apparatus comprising:

12. A method for causing an image processing device to perform image processing, comprising: a first display control step of sequentially displaying M screens (where M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device on a display device, each of the M screens includes one or more objects; When a first object corresponding to the first process among the one or more objects is selected on each of the M screens, the first process is executed. the first display control step; a first receiving step of receiving an input of a first natural sentence related to the first process from a user; a first supply step of supplying first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is accepted; a first acquisition step of acquiring first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens (M1 is an integer between 2 and M) out of the M screens, first identification information that identifies the first object among the one or more objects included in the screen; a second display control step of, when the first object information is acquired from the natural language processing unit, sequentially displaying the M screens including the M1 screens according to the first object information on the display device, wherein the first object identified by the first identification information is highlighted on each of the M1 screens; A method comprising:

Images