Computer program, device, and method for causing image processing device to execute image processing

Abstract

The present invention discloses a technology for improving user convenience. This computer program causes a computer to function as: a first reception unit that receives an input of a first natural sentence; a first supply unit that, when the input of the first natural sentence is received, supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit; a first acquisition unit that, from the natural sentence processing unit, acquires first object information that is based on the first natural sentence data and that includes first identification information for identifying a first object for each of M1-number of screens among M-number of screens; and a second display control unit that, when the first object information is acquired from the natural sentence processing unit, causes a display device to sequentially display the M-number of screens including the M1-number of screens according to the first object information, the first object being highlighted in each of the M1-number of screens.

Description

Computer program, device, and method for causing an image processing device to perform image processing

[0001] The present specification relates to a technique in a situation where processing relating to image processing should be performed.

[0002] Japanese Patent Laid-Open No. 2006-129999 discloses a technique for causing a printer to execute printing using an application installed on a terminal.

[0003] Japanese Patent Application Laid-Open No. 2022-085588

[0004] This specification discloses a technique for improving user convenience.

[0005] This specification discloses a computer program for causing an image processing device to execute image processing. The computer program includes a computer including the following components: a first display control unit configured to sequentially display M screens (M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device on a display device, each of the M screens including one or more objects, and configured to execute the first process when a first object corresponding to the first process is selected from the one or more objects on each of the M screens; a first accepting unit configured to accept input of a first natural sentence related to the first process from a user; a first supply unit configured to supply first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is accepted; and a second supply unit configured to supply first natural sentence data representing the first natural sentence to a natural sentence processing unit from the natural sentence processing unit. The first acquisition unit may function as a first acquisition unit that acquires first object information based on first natural language data, wherein the first object information includes, for each of M1 screens out of the M screens (M1 is an integer greater than or equal to 2 and less than or equal to M), first identification information that identifies the first object among the one or more objects included in the screen; and a second display control unit that, when the first object information is acquired from the natural language processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, wherein the first object identified by the first identification information is highlighted on each of the M1 screens.

[0006] According to the above configuration, when the computer accepts input of a first natural sentence, it supplies the first natural sentence data to the natural sentence processing unit and acquires first object information from the natural sentence processing unit. The computer then sequentially displays M screens, including M1 screens, according to the first object information. On each of the M1 screens, the first object identified by the first identification information is highlighted. This allows the user to easily recognize that they should select the first object on each of the M1 screens. This improves user convenience.

[0007] A computer-readable recording medium storing the above computer program, an apparatus implemented by the above computer program, and a method executed by the apparatus are also novel and useful.

[0008] 1 is a configuration diagram of a communication system. FIG. 1 is a screen transition diagram when a multi-function device is made to execute printing. FIG. 2 is a screen transition diagram continuing from FIG. 3. FIG. 3 is a screen transition diagram when a multi-function device is made to execute scanning. FIG. 14 is a flowchart of processing executed by an application. FIG. 15 is a screen transition diagram continuing from FIG. 7. FIG. 16 is a screen transition diagram continuing from FIG. 8. FIG. 17 is a screen transition diagram continuing from FIG. 9. FIG. 18 is a screen transition diagram continuing from FIG. 11. FIG. 19 is a screen transition diagram continuing from FIG. 22. FIG. 20 is a screen transition diagram continuing from FIG. 23.

[0009] (First Example) (Configuration of Communication System; FIG. 1) As shown in FIG. 1, a communication system 2 includes a terminal 10, a multi-function peripheral (MFP) 100, and an artificial intelligence (AI) server 200. The terminal 10 and the MFP 100 are connected to a local area network (LAN) 4. The terminal 10 and the MFP 100 can communicate with each other via the LAN 4. The LAN 4 may be a wired LAN or a wireless LAN. The AI ​​server 200 is installed on the Internet 6. The devices 10, 100 and the AI ​​server 200 can communicate with each other via the LAN 4 and the Internet 6.

[0010] (Configuration of terminal 10) The terminal 10 is a portable terminal device such as a mobile phone, smartphone, PDA, or tablet PC. In a modified example, the terminal 10 may be a stationary PC, laptop PC, or the like. The terminal 10 includes an operation unit 12, a display unit 14, a communication interface 16, and a control unit 30. Each unit 12 to 30 is connected to a bus line. Hereinafter, the interface will be referred to as "I / F."

[0011] The operation unit 12 is a user interface that allows a user to input various information into the terminal 10. The operation unit 12 includes, for example, a touch panel for displaying software keys (operation objects), hardware keys, or both. The hardware keys include, for example, buttons or switches. The display unit 14 is a display or panel for displaying various information. The panel may or may not be a touch panel. The panel is, for example, a liquid crystal panel or an organic EL panel. The communication I / F 16 is connected to the LAN 4.

[0012] The control unit 30 includes a CPU 32 and a memory 34. The memory 34 includes a main storage device and an auxiliary storage device. For example, the main storage device includes RAM and cache memory. For example, the auxiliary storage device may be a ROM, flash memory, solid-state drive (SSD), hard disk drive (HDD), or a combination thereof. An OS program 36 and application programs 38 are stored in the auxiliary storage device of the memory 34. Hereinafter, the OS program and application programs will be referred to as "OS" and "apps," respectively. The OS 36 controls the basic operations of the terminal 10. For example, the apps 38 enable the MFP to perform multiple functions, such as printing and scanning. The apps 38 are downloaded from a server on the Internet 6 and installed on the terminal 10. In other words, the apps 38 are so-called native apps. The CPU 32 performs various processes according to programs loaded from the auxiliary storage device to the main storage device. In the following description, various processes that are implemented by the CPU 32 in accordance with a program may be described as being performed primarily by the OS 36 or the application 38 .

[0013] The application 38 can cause the MFP 100 to perform multiple functions such as a print function and a scan function. In particular, the application 38 of this embodiment can cooperate with the AI ​​server 200 to guide the user through operations to be performed when causing the MFP 100 to perform a print function or the like.

[0014] (Configuration of MFP 100) MFP 100 is a peripheral device capable of performing multiple functions such as printing, scanning, and facsimile functions, for example, a peripheral device of terminal 10. In a modified example, MFP 100 may be a single-function device such as a printer capable of performing only printing functions, or a scanner capable of performing only scanning functions. MFP 100 is assigned a device ID "dv1" that is information for identifying MFP 100. The device ID is, for example, a model name, an IP address, or a combination thereof. In this specification, the device ID is the model name. The model name is model identification information for the device. The model name may be a higher-level model name that encompasses multiple model names. For example, the model name may be ABC Series, which encompasses model names ABC1 and ABC2. For example, it may be a serial number, etc. The MFP 100 includes an operation unit 112, a display unit 114, a communication I / F 116, a print execution unit 118, a scan execution unit 120, and a control unit 130. Each of the units 112 to 130 is connected to a bus line.

[0015] The operation unit 112 is a user interface that allows a user to input various information to the MFP 100. The operation unit 112 includes, for example, a touch panel for displaying software keys, hardware keys, or both. Hardware keys include, for example, buttons or switches. The display unit 114 is a display or panel for displaying various information. The panel may or may not be a touch panel. The panel may also be, for example, a liquid crystal panel or an organic EL panel. The communication I / F 116 is connected to the LAN 4. The print execution unit 118 includes an electrophotographic, inkjet, or thermal print engine. An electrophotographic print engine includes a photosensitive member and an exposure device that emits light to expose the photosensitive member. An inkjet print engine includes a print head that ejects ink droplets. A thermal print engine includes a print head that generates heat using a heater. The scan execution unit 120 includes a scanner engine having an image sensor such as a Charge-Coupled Device (CCD) image sensor or a Contact Image Sensor (CIS) image sensor.

[0016] The control unit 130 includes a CPU 132 and a memory 134. The memory 134 includes a main storage device and an auxiliary storage device. As an example, the main storage device includes a RAM and a cache memory. As an example, the auxiliary storage device may be a ROM, a flash memory, an SSD, an HDD, or a combination thereof. A program 136 is stored in the auxiliary storage device of the memory 134. The CPU 132 performs various processes in accordance with the program loaded from the auxiliary storage device to the main storage device.

[0017] (Configuration of AI Server 200) The AI ​​server 200 generates information (specifically, text and guidance information) for guiding user operations. The servers described herein (e.g., the AI ​​server 200, the cloud print server (not shown)) are installed on the Internet 6 by the vendor of the MFP 100 (hereinafter simply referred to as the "vendor"). While the AI ​​server 200 will be described in particular below, the same applies to other servers described herein, such as the cloud print server. The AI ​​server 200 may be a physical server, a virtual server on the cloud, a virtual machine, or the like. The AI ​​server 200 may be a single server or a collection of multiple servers. The AI ​​server 200 may be a server on an intranet or a cloud server on the Internet 6. In a modified example, the AI ​​server 200 may be installed on the Internet 6 by a business operator other than the vendor. In another modified example, the vendor may not prepare the hardware for the AI ​​server 200 on its own, but may use an environment provided by an external cloud computing service. In this case, the vendor may prepare a program (i.e., software) for the AI ​​server 200 and install it in the above-described environment to realize the AI ​​server 200. In this embodiment, as an example, a form in which the AI ​​server 200 is a physical server will be described.

[0018] The AI ​​server 200 includes a communication I / F 216 and a control unit 230. The communication I / F 216 and the control unit 230 are connected to a bus line. The communication I / F 216 is connected to the Internet 6.

[0019] The control unit 230 includes a processor 232 and a memory 234. The processor 232 includes a CPU, a graphics processing unit (GPU), a neural processing unit (NPU), or a combination thereof. The memory 234 includes a main storage device and an auxiliary storage device. For example, the main storage device includes a RAM and a cache memory. For example, the auxiliary storage device may be a ROM, a flash memory, an SSD, an HDD, or a combination thereof. A program 236 is stored in the auxiliary storage device of the memory 234. The processor 232 performs various processes in accordance with the program loaded from the auxiliary storage device to the main storage device.

[0020] A trained model is incorporated into the program 236. The AI ​​server 200 generates information for guiding a user's operation based on the trained model incorporated into the program 236. That is, the trained model has previously learned the correspondence between text and guidance information. The trained model includes a program and various parameters. The trained model includes, for example, a language model such as a large-scale language model (LLM) or a small-scale language model (SLM).

[0021] (Printing Process; FIGS. 2 to 4) Next, with reference to FIGS. 2 to 4, screens sequentially displayed on the terminal 10 when a user uses the app 38 to cause the MFP 100 to print an image will be described. In particular, FIGS. 2 to 4 describe a situation in which a user desires to print a document file (i.e., a document) in monochrome on both sides. Each screen displayed in this embodiment is displayed on the display unit 14 of the terminal 10 by the CPU 32 of the terminal 10 executing the app 38. In the following description of each screen, for ease of understanding, the CPU 32 or the app 38 will not be described as the subject of processing, but rather the terminal 10 will be described as the subject of processing. Furthermore, in the initial state of FIG. 2 (i.e., the state before the home screen SC1 is displayed), information about one or more MFPs, including the MFP 100 (e.g., the MFP's device ID, IP address, etc.) has already been registered in the memory 34 of the terminal 10 (i.e., the app 38).

[0022] When the terminal 10 receives an application launch operation from the user to launch the application 38, the terminal 10 launches the application 38 and displays the home screen SC1 of Fig. 2 on the display unit 14. The home screen SC1 includes one or more device objects OB11 and OB12, a plurality of function objects OB13 to OB18, and a bot button OB19.

[0023] Device object OB11 is an object corresponding to a registered MFP 100 and includes the device ID "dv1" of the MFP 100. As described above, device ID "dv1" is the model name of MFP 100. It should be noted that device ID "dv1" may include the IP address of MFP 100 in addition to the model name of MFP 100. Although partially hidden in FIG. 2 , device object OB12 is an object corresponding to another MFP registered to terminal 10 and includes the device ID of the other MFP. It should be noted that in a situation where only one device (i.e., MFP 100) is registered to terminal 10, device object OB12 may be an object for registering a new MFP. Function objects OB13 to OB18 are objects corresponding to a document printing function, a photo printing function, a voice printing function, a scan function, a service registration function, and a consumables purchasing function, respectively. The document printing function is a function for printing images represented by document files (e.g., Word files, PDF files, Text files, etc.). The photo printing function is a function for printing images represented by photo files (e.g., JPEG files, etc.). The audio printing function is a function for printing images represented by audio files (e.g., MP3 files, etc.). Here, an image represented by an audio file is an image represented by a document file obtained by transcribing the audio represented by the audio file. The service registration function is a function for registering the MFP with, for example, a flat-rate printing service. The consumables purchasing function is a function for ordering (i.e., purchasing) consumables used by the MFP (e.g., ink cartridges, printing paper, etc.). The bot button OB19 is a button for starting the operation guidance described below.

[0024] When the terminal 10 receives from the user at T20 the selection of a function object OB13 corresponding to document printing, it displays a file selection screen SC2 on the display unit 14. The file selection screen SC2 includes a plurality of file objects OB21 to OB23 corresponding to a plurality of document files stored in the memory 34, a cancel object OB24, and a bot button OB19. Each of the file objects OB21 to OB23 includes a file name (e.g., F1 to F3).

[0025] When the terminal 10 receives a selection of a file object OB21 corresponding to the file name F1 from the user at T20, the terminal 10 displays a print preview screen SC3 on the display unit 14. The print preview screen SC3 includes a preview image P, a device object OB31, a settings object OB32, a print object OB33, and a bot button OB19. The preview image P corresponds to the document image data of the first page of multiple pages (20 pages in this example) of document image data included in the target file corresponding to the file name F1. The character string "1 / 20" in the print preview screen SC3 indicates that the currently displayed preview image P corresponds to the document image data of the first page of the 20 pages of document image data. Although not shown in the figure, the user can display different preview images corresponding to different pages of document image data by sliding the area displaying the preview image P left or right.

[0026] The device object OB31 includes the device ID of the MFP 100 corresponding to the device object OB11, whose entirety is displayed when the function object OB13 is selected on the home screen SC1, and the paper size. That is, the device object OB31 is an object that indicates the device that will print the image corresponding to the preview image and the size of the print medium. In a modified example, the device object OB31 may not include the paper size, or may include information other than the paper size (e.g., paper type (plain paper, glossy paper, etc.)). Furthermore, the device object OB31 may be configured to allow settings such as paper size to be changed when the user selects the device object OB31. The setting object OB32 is an object for displaying a print setting screen. The print object OB33 is an object for executing image printing.

[0027] When the terminal 10 receives a selection of the setting object OB32 from the user at T30, it displays the print setting screen SC4 of Fig. 3 on the display unit 14. The print setting screen SC4 includes a plurality of item objects OB41 to OB46, a back button OB47, and a bot button OB19. Each of the item objects OB41 to OB46 includes the current setting value of the corresponding item.

[0028] The item object OB41 is an object for specifying the number of copies to be printed. The item object OB42 is an object for specifying layout settings (for example, "1 page / sheet" indicating that one page of images is to be printed on one sheet of printing medium, or "2 pages / sheet" indicating that two pages of images are to be printed on one sheet of printing medium).

[0029] The item object OB43 is an object for specifying a double-sided printing setting (i.e., one of "single-sided printing," "long-edge binding," and "short-edge binding"). "Long-edge binding" means that, in a double-sided printing medium, the up-down orientation of the image of page N printed on the first side of the printing medium matches the up-down orientation of the image of page (N+1) printed on the second side of the printing medium, which is the reverse side of the first side. "Short-edge binding" means that, in a double-sided printing medium, the up-down orientation of the image of page N printed on the first side of the printing medium is opposite to the up-down orientation of the image of page (N+1) printed on the second side of the printing medium.

[0030] The item object OB44 is an object for specifying a color / monochrome setting (i.e., either "color" or "monochrome"). In a modified example, an option other than "color" or "monochrome" (e.g., "grayscale") may be specified.

[0031] The item object OB45 is an object for specifying an image quality setting (e.g., "normal," "high quality," etc.). The item object OB46 is an object for selecting the range of pages desired to be printed from the file corresponding to the selected file name F1. The back button OB47 is a button for displaying a print preview screen (e.g., SC3).

[0032] When the terminal 10 receives from the user selection of the item object OB43 corresponding to the double-sided printing setting in T40, it displays a dialog box D1 on the display unit 14, superimposed on the print setting screen SC4. The dialog box D1 includes setting value objects OB51 to OB53 corresponding to each setting value of the double-sided printing setting and a cancel button OB54. The setting value object OB51 is an object corresponding to the option "single-sided." The setting value object OB52 is an object corresponding to the option "long-edge binding." The setting value object OB53 is an object corresponding to the option "short-edge binding." Each setting value object OB51 to OB53 includes a radio button for specifying the corresponding option. The cancel button OB54 is a button for canceling the change to the double-sided printing setting and returning to the print setting screen SC4. When the dialog box D1 is displayed, the radio button included in the setting value object OB51 corresponding to the current setting value (i.e., "single-sided") is selected in the dialog box D1.

[0033] When the terminal 10 receives the selection of the radio button included in the setting value object OB52 corresponding to the option "long edge binding" in T50, it displays the print setting screen SC5 on the display unit 14. The print setting screen SC5 is the same as the print setting screen SC4 except that the setting values ​​included in the item object OB43 are different.

[0034] When the terminal 10 receives from the user in T60 the selection of the item object OB44 corresponding to the color / monochrome setting, it displays a dialog D2 on the display unit 14, superimposed on the print setting screen SC5, as shown in FIG. 4 . The dialog D2 includes setting value objects OB71 and OB72 corresponding to each setting value of the color / monochrome setting, and a cancel button OB73. The setting value object OB71 is an object corresponding to the option "color." The setting value object OB72 is an object corresponding to the option "monochrome." Each setting value object OB71 and OB72 includes a radio button for specifying the corresponding option. The cancel button OB73 is a button for canceling the change to the color / monochrome setting and returning to the print setting screen SC5. When the dialog D2 is displayed, the radio button included in the setting value object OB71 corresponding to the current setting value (i.e., "color") is selected in the dialog D2.

[0035] When the terminal 10 accepts the selection of the radio button included in the setting value object OB72 corresponding to the option "Monochrome" in T70, it displays a print setting screen SC6 on the display unit 14. The print setting screen SC6 is similar to the print setting screen SC5 except that the setting values ​​included in the item object OB44 are different.

[0036] When terminal 10 accepts the user's selection of back button OB47 in print setting screen SC5 in T80, it re-displays print preview screen SC3 on display unit 14. Then, when terminal 10 accepts the user's selection of print object OB33 in T90, it converts image data included in the target file into print data in accordance with the specified setting values ​​and transmits the converted print data to MFP 100 via communication I / F 16. The print data has a data format interpretable by MFP 100. As a result, MFP 100 causes print execution unit 118 to print the image represented by the print data. In particular, because the specified setting values ​​include "long edge binding" and "monochrome," the user can obtain print media on which images included in the document file are printed in monochrome on both sides (in this case, 10 sheets of print media because 20 pages of images are printed on both sides of the print media).

[0037] (Scanning Process; FIG. 5) Next, referring to FIG. 5, screens sequentially displayed on the terminal 10 when a user uses the application 38 to cause the MFP 100 to scan a document will be described. In particular, FIG. 5 illustrates a situation in which a user desires to scan a document in monochrome. The user places the document to be scanned on the MFP 100. Thereafter, when the terminal 10 receives from the user at T100 in FIG. 2 a selection of a function object OB16 corresponding to scanning on the home screen SC1 (see FIG. 2), the terminal 10 displays the scan setting screen SC11 of FIG. 5 on the display unit 14. The scan setting screen SC11 includes a plurality of item objects OB111-OB113, a scan object OB114, a back button OB115, and a bot button OB19. Each of the item objects OB111-OB113 includes the current setting value of the corresponding item.

[0038] The item object OB111 is an object for specifying color / monochrome settings, and is similar to the item object OB44 of the print setting screen SC4, etc. The item object OB112 is an object for specifying a document size setting (e.g., "A4," "B5," "Letter," etc.). The item object OB113 is an object for specifying a resolution setting (e.g., "300 dpi," "600 dpi," etc.). The scan object OB114 is an object for scanning a document. The back button OB115 is a button for displaying a home screen (e.g., SC1).

[0039] When the terminal 10 receives, at T110, a selection of an item object OB111 corresponding to color / monochrome setting from the user, the terminal 10 displays a dialog D11 superimposed on the scan setting screen SC11 on the display unit 14. The dialog D11 is similar to the dialog D2 (see FIG. 4).

[0040] When the terminal 10 receives the selection of the radio button included in the setting value object OB122 corresponding to the option "monochrome" at T120, the terminal 10 displays the scan setting screen SC12 on the display unit 14. The scan setting screen SC12 is the same as the scan setting screen SC11 except that the setting values ​​included in the item object OB111 are different.

[0041] When terminal 10 accepts selection of scan object OB114 from the user at T130, it transmits a scan instruction including each specified setting value to MFP 100 via communication I / F 16. As a result, MFP 100 causes scan execution unit 120 to scan the document to be scanned that is installed in MFP 100, and generates scan data representing an image of the document to be scanned. In particular, the scan data is generated according to the setting values ​​included in the scan instruction. Then, the scan data is transmitted to terminal 10 (not shown), allowing the user to confirm the image represented by the scan data.

[0042] A user who is proficient in operating the app 38 can use the app 38 as described above to cause the MFP 100 to perform printing, etc. On the other hand, a user who is not proficient in operating the app 38 may have difficulty using the app 38 to cause the MFP 100 to perform printing, etc. For example, the user needs to operate the app 38 while referring to a web page or the like that contains instructions on how to operate the app 38, which is time-consuming for the user. The app 38 of this embodiment realizes the guidance process of FIG. 6 to improve user convenience.

[0043] (Guidance Process; FIG. 6) Next, the guidance process realized by the application 38 will be described with reference to FIG. 6. The process of FIG. 6 is triggered by receiving selection of the bot button OB19 from the user on any of the above-described screens SC1 to SC12. Hereinafter, the screen including the bot button OB19 selected as the trigger for the process of FIG. 6 will be referred to as the "guidance start screen."

[0044] In S10, the CPU 32 divides the guidance start screen into two areas. The guidance start screen is displayed in one of the divided areas (e.g., the main area A1 of the home screen SC20 in FIG. 7 ) (i.e., the objects included in the guidance start screen are arranged therein). Messages for guiding the user through operations to be performed are displayed as needed in the other area (e.g., the guidance area A2 of the home screen SC20 in FIG. 7 ). Furthermore, as shown in the home screen SC20 in FIG. 7 , the guidance area A2 includes an area R1 for receiving a character string from the user and a transmission object OB100 for transmitting the character string input in the area R1 to the AI ​​server 200.

[0045] In S12, the CPU 32 monitors whether the selection of the transmission object OB100 is accepted after the input of a character string into the input area R1 is accepted. The character string entered here corresponds to the process desired by the user (e.g., "I want to print the document file in black and white on both sides"). If the selection of the transmission object OB100 is accepted after the input of the character string is accepted, the CPU 32 determines YES in S12 and proceeds to S14.

[0046] In S14, the CPU 32 transmits the screen ID and the input character string to the AI ​​server 200 via the communication I / F 16. The screen ID is information that identifies the guidance start screen. When the AI ​​server 200 receives the screen ID and the character string from the terminal 10, it determines whether it is possible to start providing guidance to the user based on the received character string. Specifically, the AI ​​server 200 determines whether the process desired by the user can be uniquely identified from the received character string. For example, if the input character string is "I want to print a document in black and white on both sides," the AI ​​server 200 determines that the user wants to print a document file on both sides. On the other hand, if the input character string is "I want to print in black and white on both sides," the AI ​​server 200 determines that the process desired by the user cannot be uniquely identified. This is because in such a case, the AI ​​server 200 cannot determine which type of file the user wants to print: a document file, a photo file, or an audio file. When the AI ​​server 200 determines that guidance can be started, it transmits text, guidance information, and a guidance ID (not shown) to the terminal 10. The text includes a message corresponding to each of one or more screens and a completion message displayed when a series of processes is completed. The guidance information includes, for each of one or more screens, a combination of a screen ID identifying the screen and an object ID identifying an object to be highlighted on the screen (hereinafter referred to as a "target object"). Note that, as will be described in detail later, some of the one or more screens may not include an object to be highlighted. In this case, the combination information corresponding to the screen is a combination of a screen ID indicating the screen and information indicating that no object to be highlighted exists on the screen. As described above, the trained model has previously trained the correspondence between text and guidance information (i.e., combination information between a screen ID and an object ID) for the first natural sentence. For example, the trained model has trained using combinations of text and guidance information for various natural sentences assumed to be the first natural sentence as training data. Similarly, the trained model has previously trained the correspondence between text and guidance information (i.e., combination information of screen ID and object ID) for the second natural sentence.In a modified example, a trained model may implement a similar function by using Retrieval-Augmented Generation (RAG) and referencing the correspondence between text and guidance information for the first natural language sentence or the second natural language sentence. In another modified example, the present technology may be implemented without using a trained model (i.e., an AI function) and by referencing the correspondence between text and guidance information for the first natural language sentence or the second natural language sentence (e.g., a table, a database, etc.) in the same manner as a chatbot. The guidance ID is information that identifies the collection of the text and guidance information.

[0047] On the other hand, if the AI ​​server 200 determines that it is not possible to start the guidance, it sends an additional question to the terminal 10 to request information necessary to enable the guidance to start. Examples of the additional question are as follows: For example, assume a situation in which the guidance start screen is the home screen SC1 and the user inputs the string "I want to print double-sided in black and white" into the input area R1. As described above, in such a situation, the AI ​​server 200 cannot determine what type of file the user wants to print. In this case, the AI ​​server 200 sends an additional question to the terminal 10, for example, "Is the file you want to print a document file?"

[0048] In S16, the CPU 32 determines whether text and guidance information or an additional question has been received from the AI ​​server 200. If text and guidance information has been received from the AI ​​server 200 (YES in S16), the CPU 32 proceeds to S20, and if an additional question has been received from the AI ​​server 200 (NO in S16), the CPU 32 proceeds to S40.

[0049] In S20, the CPU 32 first identifies a message corresponding to the first screen from the text and identifies combination information corresponding to the first screen from the guidance information. Next, the CPU 32 determines whether the identified combination information includes an object ID. If the identified combination information includes an object ID, the CPU 32 highlights the target object identified by the object ID on the guidance start screen displayed in the main area A1. In this embodiment, as shown in FIG. 7 and other figures, the CPU 32 displays the target object in a lighter color (white in this case) than the background color of the main area A1 (gray in this case). Furthermore, the CPU 32 displays the identified message in the guidance area A2. On the other hand, if the identified combination information does not include an object ID, the CPU 32 does not highlight the target object in the main area A1. Furthermore, the CPU 32 displays the identified message in the guidance area A2.

[0050] In S22, the CPU 32 determines whether selection of the transmission object OB100 can be accepted after input of a character string into the input area R1 of the guidance area A2 has been accepted. If selection of the transmission object OB100 can be accepted after input of a character string into the input area R1 has been accepted, the CPU 32 determines YES in S22 and returns to S14. Here, a situation in which a YES determination is made in S22 is assumed to be a situation in which the user desires a process different from the process already input in S12 (see Case C described below). On the other hand, if input of a character string into the input area R1 has not been accepted, the CPU 32 determines NO in S22 and proceeds to S24.

[0051] In S24, the CPU 32 determines whether the selection of the correct object has been accepted. Specifically, the CPU 32 determines whether the selection of the highlighted target object has been accepted. If the selection of the target object has been accepted, the CPU 32 determines YES in S24 and proceeds to S30. On the other hand, if the selection of an object other than the target object has been accepted, the CPU 32 determines NO in S24 and proceeds to S50. Note that if no highlighting is performed in the main area A1, the CPU 32 determines YES in S24 regardless of which object is selected on the screen displayed in the main area A1. Note that the CPU 32 displays a screen corresponding to the selected object in the main area A1, regardless of the determination result in S24.

[0052] In S30, the CPU 32 determines whether a series of operations has been completed. Specifically, the CPU 32 determines whether all screens have been displayed in accordance with the text and guidance information received in S16. If all screens have been displayed, the CPU 32 determines YES in S30 and proceeds to S32. On the other hand, if all screens have not been displayed, the CPU 32 executes the process of S20 in accordance with the message and combination information corresponding to the next screen.

[0053] In S32, the CPU 32 displays the completion message included in the text received in S16 in the guidance area A2. When the process of S32 ends, the process of FIG.

[0054] In addition, in S40, the CPU 32 displays the follow-up question received in S16 in the guidance area R2. When the process of S40 ends, the process returns to the monitoring process of S12.

[0055] Furthermore, in S50, the CPU 32 identifies the screen ID of the screen displayed in the main area A1 in response to the selection of an incorrect object, and transmits the screen ID and a guidance ID to the AI ​​server 200 via the communication I / F 16. Upon receiving the screen ID and the guidance ID, the AI ​​server 200 transmits warning text to the terminal 10 in response to determining that the screen ID is not included in the guidance information identified by the guidance ID. The warning text includes a warning message to notify the user that an incorrect object has been selected by the user.

[0056] In S52, the CPU 32 receives the warning text from the AI ​​server 200 via the communication I / F 16.

[0057] In S54, the CPU 32 displays the warning message included in the received warning text in the guidance area A2. When the process of S54 ends, the process returns to the monitoring process of S12.

[0058] (Case A; FIGS. 7 to 10) Next, with reference to FIGS. 7 to 10, the screens displayed by the terminal 10 as a result of the execution of the guidance process of FIG. 6 will be described. When the terminal 10 accepts selection of the bot button OB19 in the home screen SC1 (see FIG. 2) at T200 of FIG. 2 (trigger of FIG. 6), it displays the home screen SC20 of FIG. 7 on the display unit 14 (S10). As shown in FIG. 7, the home screen SC20 is divided into a main area A1 and a guidance area A2.

[0059] The main area A1 of the home screen SC20 displays a screen that is the same as a portion of the home screen SC1. Although the remaining portion of the home screen SC1 is not displayed in the main area A1, the user can slide the main area A1 up or down to display the portion of the home screen SC1 that is not displayed in the main area A1. This also applies to the main areas A1 of the subsequent screens.

[0060] A message MB1, an input area R1, and a transmission object OB100 are arranged in the guidance area A2 of the home screen SC20. The message MB1 is a message that asks the user what kind of processing the user wants to execute.

[0061] In T201, the terminal 10 accepts the user's input of the character string "I want to print the document in black and white on both sides" into the input area R1, and then accepts the selection of the transmission object OB100 (YES in S12). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). In this case, the text includes nine messages MB2 to MB10 and a completion message MB11. The guidance information also includes nine screen IDs (i.e., information identifying each of the screens SC1 to SC4, the dialog D1, the screen SC5, the dialog D2, and the screens SC6 and SC3) and six object IDs (i.e., information identifying each of the objects OB13, OB32, OB43, OB44, OB47, and OB33). Of the nine screen IDs, three screen IDs that identify the screen SC22, the dialogs D1, and D2 are combined with information indicating that no object to be highlighted exists.

[0062] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies from the text a message (i.e., message MB2) corresponding to the first screen (i.e., home screen SC1) and identifies combination information corresponding to the first screen. Because the combination information includes an object ID identifying the functional object OB13, the functional object OB13 identified by the object ID included in the combination information is highlighted in the home screen SC1 displayed in the main area A1 of the home screen SC21 (S20). Furthermore, in the guidance area A2 of the home screen SC21, a message MU1 indicating the entered character string and a message MB2 corresponding to the first screen are further displayed, compared to the guidance area A2 of the home screen SC20 (S20). By viewing the message MB2, the user can recognize the operation to be performed by the user. In particular, in the guidance area A2, new messages are displayed in order below the older messages, and the new messages are automatically scrolled so that the user can check them. This allows the user to easily check the most recent message. Although the old message MB1 is not displayed in the guidance area A2 of the home screen SC21, the old message MB1 can be displayed by the user sliding the guidance area A2 up or down. This also applies to the guidance area A2 of the subsequent screens.

[0063] The processing of T210 is the same as the processing of T10 in Fig. 2. In this case, the terminal 10 displays the file selection screen SC22 on the display unit 14. A part of the screen of the file selection screen SC2 is displayed in the main area A1 of the file selection screen SC22. In T210, the highlighted functional object OB13 (i.e., the correct object) is selected (NO in S22 and YES in S24 in Fig. 6), and not all screens have been displayed according to the received text and guidance information (NO in S30). Therefore, the terminal 10 determines not to highlight the next screen (i.e., the file selection screen SC2) in the main area A1 of the file selection screen SC22, according to the message and combination information corresponding to the next screen, and displays a message MB3 in the guidance area A2 of the file selection screen SC22.

[0064] The subsequent processing from T220 to T290 in Figures 7 to 10 is the same as the processing from T20 to T90 in Figures 2 to 4. Furthermore, in each of screens SC23 to SC27 in Figures 8 to 10, a screen similar to SC3 to SC6 and SC3 in Figures 2 to 4 is displayed in the main area A1. Furthermore, the dialogs D1 and D2 displayed in the main area A1 of each of screens SC24 and SC25, superimposed on each of screens SC24 and SC25, are similar to the dialogs D1 and D2 displayed in the main area A1 of each of screens SC23 to SC27, superimposed on each of screens SC24 and SC25, respectively. Furthermore, in the main area A1 and each of dialogs D1 and D2 of each of screens SC23 to SC27, whether or not to highlight is determined based on the corresponding combination information included in the previously received guidance information. Furthermore, a corresponding message included in the previously received text is newly displayed in the guidance area A2 of each of screens SC23 to SC27.

[0065] 10 , when the terminal 10 accepts selection of the highlighted print object OB33 from the user, it transmits the print data to the MFP 100 and determines that the series of operations is complete (i.e., all screens have been displayed) (YES in S30 in FIG. 6 ), and displays the home screen SC28 on the display unit 14. The main area A1 of the home screen SC28 is the same as the main area A1 of the home screen SC20. The guidance area A2 of the home screen SC28 displays a completion message MB11 included in the received text (S32).

[0066] Thus, with the above configuration, when the terminal 10 receives the input of the character string "I want to print the document in black and white on both sides" (YES in S12 of FIG. 6), it supplies the character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). The terminal 10 then sequentially displays each of the screens SC21 to SC28 of FIGS. 7 to 10 in accordance with the text and guidance information. As shown in FIGS. 7 to 10, an object identified by the object ID included in the guidance information (i.e., the target object) is highlighted on at least a portion of each of the sequentially displayed screens. This allows the user to easily recognize that they should select the highlighted object. This improves user convenience.

[0067] Also, consider a comparative example in which a chatbot is used to sequentially display print operation guidance to a terminal user via chat. In this comparative example, the user can print an image by sequentially inputting print settings and the like on the chatbot's screen in response to requests (i.e., questions) from the chatbot. However, in this comparative example, an interaction with the chatbot is required each time printing is performed. In contrast, in the configuration of this embodiment, the object to be operated is highlighted in the main area A1 where the normal screen shown in Figures 2 to 4 is located. This allows the user to become familiar with operations on the normal screen. As a result, the user may be able to have the MFP 100 perform printing without requiring guidance the next time the user prints using the MFP 100.

[0068] (Case B; FIG. 11) Next, case B in which an incorrect object is selected will be described with reference to FIG. 11. The home screen SC21 in FIG. 11 is the same as the home screen SC21 in FIG.

[0069] In T310, the terminal 10 accepts selection of a functional object OB16 corresponding to scanning from the user. As shown in FIG. 11 , the functional object OB16 is not highlighted in the main area A1 of the home screen SC21. That is, the functional object OB16 is an incorrect object (NO in S24 of FIG. 6 ). In response to the selection of the functional object OB16, the terminal 10 displays a scan setting screen SC31 on the display unit 14. The scan setting screen SC11 (see FIG. 5 ) is displayed in the main area A1 of the scan setting screen SC31.

[0070] Furthermore, in response to the selection of an incorrect object, the terminal 10 transmits the screen ID of the scan setting screen SC11 displayed in the main area A1 of the scan setting screen SC31 to the AI ​​server 200 (S50) and receives warning text including a warning message MB101 from the AI ​​server 200 (S52). The terminal 10 then displays the warning message MB101 in the guidance area A2 of the scan setting screen SC31 (S54). By viewing the warning message MB101, the user can recognize that an incorrect object has been selected. Note that the user can continue to perform operations on the scan setting screen SC31 to cause the MFP 100 to scan the document.

[0071] In T320, the terminal 10 accepts the user's input of the character string "Return to guide" into the input area R1, and then accepts the selection of the transmission object OB100 (YES in S12 after S54). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). The text of this case includes a message MB102 in addition to each message included in the text of case A. Furthermore, the guidance information includes combination information that combines a screen ID that identifies the scan setting screen SC11 and an object ID that identifies the back button OB115, in addition to the combination information included in the guidance information of case A.

[0072] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies a message corresponding to the first screen (i.e., the scan setting screen SC11) and identifies combination information corresponding to the first screen. Since the combination information includes an object ID that identifies the back button OB115, the back button OB115 is highlighted in the main area A1 of the scan setting screen SC32. Furthermore, in the guidance area A2 of the scan setting screen SC32, a message MU101 indicating the entered character string and a message MB102 corresponding to the first screen included in the received text are displayed.

[0073] When the terminal 10 receives a selection of the highlighted back button OB115 from the user in T330, it determines that the correct object has been selected (YES in S24 of FIG. 6 ) and displays a screen similar to the home screen SC21 of FIG. 7 on the display unit 14. Note that the screen displayed here differs from the home screen SC21 of FIG. 7 in the message displayed in the guidance area A2. In this way, if the user selects an incorrect object, the terminal 10 notifies the user of this and allows the user to return to the guidance. This allows the processing desired by the user to be executed appropriately.

[0074] (Case C; Figs. 12 and 13) Next, case C in which the user changes the process desired will be described with reference to Figs. 12 and 13. The home screen SC21 in Fig. 12 is the same as the home screen SC21 in Fig. 7.

[0075] In T410, the terminal 10 accepts the user's input of the character string "I want to scan in monochrome" into the input area R1, and then accepts the selection of the transmission object OB100 (YES in S22). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). In this case, the text includes four messages MB201 to MB204 and a completion message MB205. The guidance information also includes four screen IDs (i.e., information identifying the screens SC1, SC11, dialog D11, and SC12) and three object IDs (i.e., information identifying the objects OB16, OB111, and OB114). Of the four screen IDs, the screen ID identifying the dialog D11 is combined with information indicating that no object to be highlighted exists.

[0076] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies a message corresponding to the first screen (i.e., the home screen SC1) and also identifies combination information corresponding to the first screen. Since the combination information includes an object ID that identifies the functional object OB16, the functional object OB16 is highlighted in the main area A1 of the home screen SC41. Furthermore, in the guidance area A2 of the home screen SC41, a message MU201 indicating the input character string and a message MB201 corresponding to the first screen included in the received text are displayed.

[0077] The processing of T420 is the same as the processing of T100 in FIG. 2 . In this case, the terminal 10 displays the scan setting screen SC42 on the display unit 14. The scan setting screen SC11 is displayed in the main area A1 of the scan setting screen SC42. In T420, the highlighted functional object OB16 (i.e., the correct object) is selected (NO in S22 and YES in S24 in FIG. 6 ), and not all screens have been displayed according to the received text and guidance information (NO in S30). Therefore, the terminal 10 determines to highlight the item object OB111 in the main area A1 of the scan setting screen SC42 according to the message and combination information corresponding to the next screen (i.e., the scan setting screen SC11), and displays a message MB202 in the guidance area A2 of the scan setting screen SC42.

[0078] The subsequent processing at T430 in FIG. 12 and the processing at T440 and T450 in FIG. 13 are similar to the processing at T110 to T130 in FIG. 5. Furthermore, in each of screens SC42 and SC43 in FIG. 13, a screen similar to SC11 and SC12 in FIG. 5 is displayed in the main area A1. Furthermore, the dialog D11 displayed in the main area A1 of scan setting screen SC42, superimposed on the screen SC42, is similar to the dialog D11 displayed in the main area A1 of scan setting screen SC42. Furthermore, in the main area A1 and dialog D11 of each of screens SC42 and SC43, whether or not to highlight is determined based on the corresponding combination information included in the previously received guidance information. Furthermore, a corresponding message included in the previously received text is newly displayed in the guidance area A2 of each of screens SC42 and SC43.

[0079] When the terminal 10 receives the selection of the highlighted scan object OB114 from the user at T450 in FIG. 13 , it sends a scan instruction to the MFP 100 and determines that the series of operations is complete (i.e., all screens have been displayed) (YES in S30 in FIG. 6 ), displaying the home screen SC44 on the display unit 14. The main area A1 of the home screen SC44 is the same as the main area A1 of the home screen SC20 (see FIG. 7 ). The guidance area A2 of the home screen SC44 displays a completion message MB205 included in the received text (S32). In this way, even if the user's desired processing is changed midway, the terminal 10 can appropriately guide the user through the operations required to execute the changed processing.

[0080] (Case D; FIG. 14) Next, case D will be described with reference to FIG. 14. Case D differs from case A in the screen on which the bot button OB19 is selected (i.e., the guidance start screen).

[0081] When the terminal 10 receives a selection of the bot button OB19 included in the print preview screen SC3 from the user at T500 in Fig. 2 (the trigger in Fig. 6), the terminal 10 displays the print preview screen SC50 in Fig. 14 on the display unit 14 (S10). As shown in Fig. 14, the print preview screen SC50 is divided into a main area A1 and a guide area A2.

[0082] The main area A1 of the print preview screen SC50 displays a screen that is the same as a part of the print preview screen SC3. The message MB301 displayed in the guide area A2 of the print preview screen SC50 is the same as the message MB1 of the case A (see FIG. 7).

[0083] In T510, the terminal 10 accepts the user's input of the character string "I want to print the document in black and white on both sides" into the input area R1, and then accepts the selection of the transmission object OB100 (YES in S12). In this case, the terminal 10 transmits the input character string to the AI ​​server 200 (S14) and receives text and guidance information from the AI ​​server 200 (YES in S16). In this case, the text includes seven messages MB302, MB5 to MB10, and a completion message MB11. The guidance information also includes seven screen IDs (i.e., information identifying each of the screens SC3, SC4, dialog D1, screen SC5, dialog D2, and screens SC6 and SC3) and five object IDs (i.e., information identifying each of the objects OB32, OB43, OB44, OB47, and OB33). Of the seven screen IDs, the two screen IDs that identify the dialogs D1 and D2 are combined with information indicating that there are no objects to be highlighted. If the AI ​​server 200 receives input of the character string "I want to print double-sided in black and white" from the user at the stage T510, the AI ​​server 200 determines that it is possible to start guidance. This is because the document file corresponding to the file name F1 has already been selected at the stage T510.

[0084] When the terminal 10 receives the text and guidance information from the AI ​​server 200, it identifies from the text a message (i.e., message MB302) corresponding to the first screen (i.e., print preview screen SC3) and identifies combination information corresponding to the first screen. Since the combination information includes an object ID that identifies the setting object OB32, the setting object OB32 is highlighted in the main area A1 of the print preview screen SC51. Furthermore, in the guidance area A2 of the print preview screen SC51, compared to the guidance area A2 of the print preview screen SC50, a message MU301 indicating the entered character string and a message MB302 corresponding to the first screen included in the received text are further displayed. The subsequent processing is the same as in Case A.

[0085] As shown in FIGS. 2 to 4, each of the screens SC1 to SC6 in FIGS. 2 to 4 includes a bot button OB19. Therefore, operation guidance can be started from any of the screens SC1 to SC6 in FIGS. 2 to 4. For example, as in case D, operation guidance can be started from the print preview screen SC3. Therefore, the user can select the bot button OB19 on the screen that is displayed when the user becomes unsure of the operation to perform. Therefore, compared to the comparative example in which only the home screen SC1 includes the bot button OB19, the user does not need to return to the home screen SC1 to start guidance when the user becomes unsure of the operation to perform. This improves user convenience.

[0086] (Correspondence) The MFP 100 is an example of an "image processing device." The CPU 32 and the application 38 are examples of a "computer" and a "computer program," respectively. The AI ​​server 200 is an example of a "natural language processing unit." The display unit 14 of the terminal 10 is an example of a "display device" and an "output device." The process of performing black-and-white double-sided printing of a document is an example of a "first process." In Case A, the home screen SC1 is an example of a "second screen." In Case A, the screens SC1 to SC6 and the dialogs D1 and D2 in Figures 2 to 4 are examples of "M screens." In Case A, the screens SC1 to SC6 are examples of "M1 screens." In Case D, the print preview screen SC3 is an example of a "second screen." In Case D, the screens SC3 to SC6 and the dialogs D1 and D2 in Figures 2 to 4 are examples of "M screens." In Case D, the print preview screen SC3 is an example of a "second screen." In Case D, the screens SC3 to SC6 and the dialogs D1 and D2 in Figures 2 to 4 are examples of "M screens." In Case D, the print preview screen SC3 is an example of a "second screen." A functional object OB13 on the home screen SC1 is an example of a "first object." The character string "I want to print the document in black and white on both sides" is an example of a "first natural sentence." The guidance information received from the AI ​​server 200 after the processing of T201 in FIG. 7 is an example of a "first object information." In particular, the object ID included in the guidance information is an example of a "first identification information." The main area A1 and the guidance area A2 are examples of a "first area" and a "second area," respectively. The home screen SC21 is an example of a "first screen." The functional object OB16 selected in T310 in FIG. 11 is an example of a "different object." The warning message MB101 is an example of "warning information." The bot button OB19 is an example of a "predetermined object."

[0087] The process of scanning in monochrome is an example of a "second process." The home screen SC1 in FIG. 2 and the screen SC11, dialog D11, and screen SC12 in FIG. 5 are examples of "N screens." The home screen SC1 in FIG. 2 and the screens SC11 and SC12 in FIG. 5 are examples of "N1 screens." The function object OB16 on the home screen SC1 is an example of a "second object." The natural sentence "I would like to scan in monochrome" is an example of a "second natural sentence." The guidance information received from the AI ​​server 200 after the process of T410 in FIG. 12 is an example of "second object information." In particular, the object ID included in the guidance information is an example of "second identification information."

[0088] The process of displaying each screen in FIGS. 2 to 4 is an example of a process executed by a "first display control unit." The process of S12 in FIG. 6 is an example of a process executed by a "first receiving unit." The process of S14 in FIG. 6 is an example of a process executed by a "first (and second) supply unit." The process of S16 in FIG. 6 is an example of a process executed by a "first (and second) acquisition unit." The processes of S20 to S30 in FIG. 6 and the process of displaying each screen in FIGS. 7 to 10 are examples of a process executed by a "second display control unit." In particular, the process of S20 in FIG. 6 is an example of a process executed by a "first output control unit." The process of S54 in FIG. 6 is an example of a process executed by a "second output control unit." The process of displaying the home screen SC1 in FIG. 2 and each screen in FIG. 5 is an example of a process executed by a "third display control unit." The process of S22 in FIG. 6 is an example of a process executed by a "second receiving unit." The processes of S20 to S30 in Fig. 6 and the processes of displaying the screens in Fig. 12 to 13 are examples of processes executed by a "fourth display control unit." The process of S10 in Fig. 6 is an example of processes executed by a "fifth display control unit."

[0089] Although specific examples of the technology disclosed in this specification have been described in detail above, these are merely examples and do not limit the scope of the claims. The technology described in the claims includes various modifications and variations of the specific examples exemplified above. Modifications of the above examples are listed below.

[0090] (Variation 1) In the above embodiment, the terminal 10 has been described as providing guidance on operations when causing the MFP 100 to execute printing or scanning. The technology of this embodiment is also useful for providing guidance on operations when executing processes other than printing or scanning. Here, "processes other than printing or scanning" refer to processes related to the MFP 100, such as a process for registering the MFP 100 with a subscription service or a process for establishing a wireless connection between the MFP 100 and an external device. Generally speaking, the "first process" may be any "process related to an image processing device."

[0091] (Modification 2) Terminal 10 may accept voice input from the user instead of inputting a character string into input area R1 of guidance area A2.

[0092] (Variation 3) In the above embodiment, the application 38 is described as a native application installed on the terminal 10. In a variation, the application 38 may be an application installed on, for example, the MFP 100. In this case, the MFP 100 may display the various screens of the above embodiment on the display unit 114 of the MFP 100. In this variation, the CPU 132 of the MFP 100 is an example of a "computer," and the display unit 114 of the MFP 100 is an example of a "display device."

[0093] (Variation 4) Instead of app 38, which is a native app installed on terminal 10, for example, a web app may be used, a cloud app may be used, or an app realized by a combination of at least two of a native app, a web app, and a cloud app may be used. Here, "a combination of at least two" means, for example, a combination of a native app and a web app (a so-called hybrid app), a combination of a native app and a cloud app, etc.

[0094] For example, in an example in which a web application is used instead of the application 38, the various processes of the above embodiment are realized by a cloud print server (not shown). In this modification, the CPU of the cloud print server is an example of a "computer," and the web application is an example of a "computer program." The display unit 14 of the terminal 10 is an example of a "display device." The cloud print server operates in accordance with the web application on the server. The cloud print server transmits corresponding screen data (e.g., web page data, etc.) to the terminal 10, thereby displaying the various screens (SC1 to SC6, SC11, SC12, SC20 to SC28, etc.) described in the above embodiment on the display unit 14 of the terminal 10 via a web browser or WebView. The cloud print server receives information indicating an operation accepted on the various screens (e.g., an operation ID), various data for which a selection has been accepted, etc. from the terminal 10, and executes processing corresponding to the operation. The operation ID is, for example, information indicating that an operation (i.e., selection) of an object such as object OB33 has been accepted. The various data for which selection has been accepted include, for example, the selected device ID "dv1," the target file specified on the file selection screen SC2, data indicating the items selected on the print setting screen SC4, etc. That is, the cloud print server accepts operations on the various screens through the reception of the above-mentioned operation ID, various data, etc. When the cloud print server receives from the terminal 10 an operation ID indicating that the operation of the print object OB33 has been accepted, it transmits print data to the MFP 100 identified by the device ID "dv1." Note that in this modified example, the device ID "dv1" may include, in addition to the model name, a token for the cloud print server to communicate with the MFP 100, authentication information, the global IP address to which the MFP 100 belongs, etc.

[0095] Furthermore, if the cloud print server cannot acquire local information such as the device ID from a web browser or the like due to security settings, a hybrid application may be created by combining a web application with a native application installed on the terminal 10. In this case, the native application can acquire local information such as the device ID stored in the memory 34 of the terminal 10 and send that information to the cloud print server. In this case, the cloud print server performs the same process as described above via the native application. The same applies when a cloud application is used instead of a web application.

[0096] (Variant 5) The communication system 2 may not include the AI ​​server 200. In this variant, for example, an AI engine that executes various processes instead of the AI ​​server 200 described above may be provided in the terminal 10. The AI ​​engine includes a trained model. In this variant, the terminal 10 includes a "natural language processing unit." In another variant, an AI engine that executes various processes instead of the AI ​​server 200 described above may be provided in the MFP 100. The AI ​​engine includes a trained model. In this variant, the MFP 100 includes a "natural language processing unit."

[0097] (Variation 6) In the above embodiment, the target object is highlighted by displaying the target object and its surrounding background in a lighter color than the background color of the main area A1. The highlighting method is not limited to the above embodiment. For example, the target object may be highlighted by displaying an icon (e.g., an arrow) pointing to the target object. In another variation, the target object may be highlighted by displaying it in a swaying manner. In another variation, the target object may be highlighted by blinking the target object. In another variation, the target object may be highlighted by changing the text color of the target object from the text colors of other objects.

[0098] (Variation 7) The terminal 10 can omit the process of S10 in Fig. 6. That is, the terminal 10 does not need to divide the screen into two areas. In this case, the terminal 10 can display each screen that does not include the guidance area A2. In this variation, the "first output control unit" can be omitted.

[0099] (Variation 8) In S20 of Fig. 6, the terminal 10 may output the message by voice instead of displaying the message in the guidance area A2. In this variation, the process of outputting the message by voice is an example of a process executed by the "first output control unit."

[0100] (Variant 9) The terminal 10 may not receive text from the AI ​​server 200 in S16 of Fig. 6. In this variant, for example, the terminal 10 may generate a message to be displayed in the guidance area A2 based on the object information received from the AI ​​server 200.

[0101] (Variation 10) In the above embodiment, the terminal 10 divided the home screen SC1 or the like, with the main area A1 arranged on the upper side and the guide area A2 arranged on the lower side. The arrangement of the divided areas A1 and A2 is not limited to the form of the above embodiment. For example, the main area A1 may be arranged on the lower side and the guide area A2 on the upper side. Furthermore, the areas may be divided horizontally instead of vertically. That is, the terminal 10 may divide the home screen SC1 or the like horizontally, with one of the main area A1 and the guide area A2 arranged on the left side and the other on the right side.

[0102] (Modification 11) When a new message is displayed in the guidance area A2, the guidance area A2 does not have to be automatically scrolled.

[0103] (Variation 12) The terminal 10 can omit the processes of S50 to S54 in Fig. 6. That is, even if an incorrect object is selected by the user, the terminal 10 does not need to display the warning message MB101. In this variation, the "second output control unit" can be omitted.

[0104] (Variation 13) In the above embodiment, the bot button OB19 was included on each of the screens SC1 to SC6 in Figures 2 to 4. Alternatively, in a variation, the bot button OB19 may be included only on the home screen SC1, for example, and not on each of the screens SC2 to SC6.

[0105] (Variation 14) In the above embodiment, the dialogs D1, D2, and D11 were not highlighted. In a variation, the objects to be selected may be highlighted in the dialogs D1, D2, and D11. In this case, for example, when there are two options, "long edge binding" and "short edge binding," as in the dialog D1, the two objects may be highlighted.

[0106] (Variant 15) In the above embodiment, the processing of each step in Figure 6 is realized by software (e.g., OS 36, application 38, programs 136, 236), but at least one of these processes may be realized by hardware such as a logic circuit.

[0107] The technical elements described in this specification or drawings may exhibit technical utility either alone or in various combinations, and are not limited to the combinations described in the claims at the time of filing. Furthermore, the technologies illustrated in this specification or drawings may achieve multiple objectives simultaneously, and achieving one of those objectives alone is technically useful.

[0108] In the scope of the claims at the time of filing, even if each claim depends on only some of the claims, it is not limited to the fact that each claim can depend on only those some of the claims. To the extent that there is no technical contradiction, each claim can also depend on other claims that were not dependent at the time of filing. In other words, the technology of each claim can be combined in various ways as follows: (Item 1) A computer program for causing an image processing device to execute image processing, the computer program comprising: a first display control unit that causes a display device to sequentially display M screens (M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device, each of the M screens including one or more objects, and the first process being executed when a first object corresponding to the first process is selected from the one or more objects on each of the M screens; a first accepting unit that accepts input of a first natural sentence related to the first process from a user; and a first supply unit that, when the input of the first natural sentence is accepted, supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit. a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens out of the M screens (M1 is an integer greater than or equal to 2 and less than or equal to M), first identification information that identifies the first object among the one or more objects included in the screen; and a second display control unit that, when the first object information is acquired from the natural sentence processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, the first object identified by the first identification information being highlighted on each of the M1 screens.(Item 2) The computer program according to Item 1, further causing the computer to function as a first output control unit that causes an output device to output a message prompting a user to select the first object highlighted on each of the M1 screens. (Item 3) The computer program according to Item 2, wherein the first acquisition unit acquires the first object information and the message from the natural language processing unit. (Item 4) The computer program according to Item 2 or 3, wherein the second display control unit sequentially displays the M screens in a first area within the display device, and the first output control unit displays the message in a second area within the display device that is the output device, the second area being different from the first area. (Item 5) The computer program according to Item 4, wherein the first output control unit sequentially displays the M1 messages corresponding to the M1 screens in the second area each time the M1 screens are sequentially displayed in the first area. (Item 6) The computer program according to any one of Items 1 to 5, further causing the computer to function as a second output control unit that outputs warning information to an output device when an object different from the first object is selected on a first screen of the M1 screens.(Item 7) The computer program further includes a third display control unit that causes the display device to sequentially display N screens (N is an integer equal to or greater than 2) corresponding to a second process related to the image processing device, the second process being different from the first process, each of the N screens including one or more objects, and the third display control unit that executes the second process when a second object corresponding to the second process is selected from the one or more objects on each of the N screens; a second accepting unit that accepts input of a second natural sentence related to the second process when any of the M screens including the M1 screens according to the first object information is displayed on the display device; and a second supply unit that supplies second natural sentence data representing the second natural sentence to the natural sentence processing unit when the input of the second natural sentence is accepted. 7. The computer program according to any one of items 1 to 6, wherein the computer program functions as: a second acquisition unit that acquires from the natural language processing unit second object information based on the second natural language data, the second object information being different from the first object information, the second acquisition unit including, for each of N1 screens (N1 is an integer greater than or equal to 2 and less than or equal to N) out of the N screens corresponding to the second processing, second identification information that identifies the second object corresponding to the second processing among the one or more objects included on the screen; and a fourth display control unit that, when the second object information is acquired from the natural language processing unit, causes the display device to sequentially display N1 screens according to the second object information, the second object identified by the second identification information being highlighted on each of the N1 screens.(Item 8) The computer program of any one of items 1 to 7, wherein each of a plurality of screens displayable by the computer program includes a predetermined object, and the computer program further causes the computer to function as a fifth display control unit that, when the predetermined object is selected, causes the display device to display an input area for accepting input of natural language. (Item 9) The computer program of item 8, wherein, when the predetermined object included in a second screen of the plurality of screens is selected and the first natural language sentence is input in the input area, each of the M screens is a screen to be displayed after the second screen. (Item 10) The computer program of any one of items 1 to 9, wherein the image processing device includes a print execution unit, and the first process is a process of causing the print execution unit to print an image.

[0109] 2: Communication system, 4: LAN, 6: Internet, 10: Terminal, 12, 112: Operation unit, 14, 114: Display unit, 16, 116, 216: Communication I / F, 30, 130, 230: Control unit, 32, 132: CPU, 34, 134, 234: Memory, 36: OS program, 38: Application program, 100: MFP, 118: Print execution unit, 120: Scan execution unit, 200: AI server, 232: Processor

Claims

1. A computer program for causing an image processing device to execute image processing, the computer program comprising a computer comprising the following units: a first display control unit for sequentially displaying M screens (M is an integer of 2 or more) corresponding to a first process related to the image processing device on a display device, each of the M screens including one or more objects, the first process being executed when a first object corresponding to the first process is selected from the one or more objects on each of the M screens; a first accepting unit for accepting input of a first natural sentence related to the first process from a user; and a first supply unit for supplying first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is accepted. a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens out of the M screens (M1 is an integer greater than or equal to 2 and less than or equal to M), first identification information that identifies the first object among the one or more objects included in the screen; and a second display control unit that, when the first object information is acquired from the natural sentence processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, the first object identified by the first identification information being highlighted on each of the M1 screens.

2. The computer program according to claim 1, further causing the computer to function as a first output control unit that causes an output device to output a message prompting a user to select the first object that is highlighted on each of the M1 screens.

3. The computer program according to claim 2, wherein the first acquisition unit acquires the first object information and the message from the natural language processing unit.

4. The computer program of claim 2, wherein the second display control unit sequentially displays the M screens in a first area within the display device, and the first output control unit displays the message in a second area within the display device, which is the output device, that is different from the first area.

5. The computer program of claim 4, wherein the first output control unit sequentially displays M1 messages corresponding to the M1 screens in the second area each time the M1 screens are sequentially displayed in the first area.

6. The computer program according to claim 1, further causing the computer to function as a second output control unit that outputs warning information to an output device when an object different from the first object is selected on a first screen of the M1 screens.

7. The computer program further comprises a third display control unit that causes the display device to sequentially display N screens (N is an integer equal to or greater than 2) corresponding to a second process related to the image processing device, the second process being different from the first process, each of the N screens including one or more objects, and the third display control unit that executes the second process when a second object corresponding to the second process is selected from the one or more objects on each of the N screens; a second accepting unit that accepts input of a second natural sentence related to the second process when any of the M screens including the M1 screens according to the first object information is displayed on the display device; and a second supply unit that supplies second natural sentence data representing the second natural sentence to the natural sentence processing unit when the input of the second natural sentence is accepted.

2. The computer program product of claim 1, further comprising: a second acquisition unit that acquires from the natural language processing unit second object information based on the second natural language data, the second object information being different from the first object information, the second acquisition unit including second identification information that identifies the second object corresponding to the second processing among the one or more objects included on each of N1 screens (N1 is an integer greater than or equal to 2 and less than or equal to N) out of the N screens corresponding to the second processing; and a fourth display control unit that, when the second object information is acquired from the natural language processing unit, causes the display device to sequentially display N1 screens according to the second object information, the second object identified by the second identification information being highlighted on each of the N1 screens.

8. The computer program of claim 1, wherein each of a plurality of screens that can be displayed by the computer program includes a predetermined object, and the computer program further causes the computer to function as a fifth display control unit that, when the predetermined object is selected, causes the display device to display an input area for accepting input of natural language.

9. A computer program as described in claim 8, wherein when the specified object included in a second screen of the plurality of screens is selected and the first natural sentence is input into the input area, each of the M screens is a screen to be displayed after the second screen.

10. The computer program according to claim 1, wherein the image processing device includes a print execution unit, and the first process is a process of causing the print execution unit to print an image.

11. A device for causing an image processing device to execute image processing, comprising: a first display control unit that causes a display device to sequentially display M screens (M is an integer equal to or greater than 2) corresponding to a first process related to the image processing device, each of the M screens including one or more objects, and the first process being executed when a first object corresponding to the first process is selected from the one or more objects on each of the M screens; a first accepting unit that accepts input of a first natural sentence related to the first process from a user; and a first supply unit that, when the input of the first natural sentence is accepted, supplies first natural sentence data representing the first natural sentence to a natural sentence processing unit. a first acquisition unit that acquires first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens out of the M screens (M1 is an integer greater than or equal to 2 and less than or equal to M), first identification information that identifies the first object among the one or more objects included in the screen; and a second display control unit that, when the first object information is acquired from the natural sentence processing unit, causes the display device to sequentially display the M screens including the M1 screens according to the first object information, the second display control unit highlighting the first object identified by the first identification information on each of the M1 screens.

12. A method for causing an image processing device to execute image processing, comprising: a first display control step of sequentially displaying M screens (M is an integer of 2 or more) corresponding to a first process related to the image processing device on a display device, each of the M screens including one or more objects, and the first process being executed when a first object corresponding to the first process is selected from the one or more objects on each of the M screens; a first receiving step of receiving an input of a first natural sentence related to the first process from a user; and a first supply step of supplying first natural sentence data representing the first natural sentence to a natural sentence processing unit when the input of the first natural sentence is received. a first acquisition step of acquiring first object information based on the first natural sentence data from the natural sentence processing unit, the first object information including, for each of M1 screens out of the M screens (M1 is an integer between 2 and M), first identification information that identifies the first object among the one or more objects included in the screen; and a second display control step of, when the first object information is acquired from the natural sentence processing unit, sequentially displaying, on the display device, the M screens including the M1 screens according to the first object information, the first object identified by the first identification information, in each of the M1 screens.

Images