Image processing device, image processing method, and program

Abstract

The purpose is to reduce the effort required to input instructions when having an AI service execute a process. [Solution] In order to solve the above problem, the image processing device of the present invention is characterized by having a receiving means for receiving a button selection from a user, an acquisition means for reading a document and acquiring image data in accordance with the reception means' acceptance of the selection, and a transmission means for transmitting an instruction statement registered by the user in association with the button, which instruction statement is used by an AI service for processing using the image data, and the image data to the AI ​​service.

Description

Technical Field

[0001] The present invention relates to image a processing device, image a processing method, and a program.

Background Art

[0002] In recent years, generative AI (Artificial Intelligence) that can automatically generate creative content such as images, text, and voice has been rapidly spreading. Along with this, various services (AI services) using generative AI are being provided. In Patent Document 1, an image generation device is disclosed that creates an instruction sentence including text corresponding to tags selected by a user, inputs the instruction sentence into generative AI, and outputs an image generated using the generative AI.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] An image obtained by scanning a manuscript may be processed by an AI service, or a new image may be output based on the image obtained by scanning. Specifically, for example, an image obtained by scanning a manuscript and an instruction sentence may be sent to an AI service to execute a process of summarizing the text included in the image. The instruction sentence when executing the summarization process of the text in the image by the AI service is, for example, "Summarize the content described in the image within 400 characters", and when performing the summarization process for each of 10 images, the image to be sent changes each time but the instruction sentence does not change.

[0005] However, in the past, even when having an AI service perform the same process, users had to manually enter instructions each time, which was time-consuming.

[0006] This invention has been made in view of the above problems, and aims to reduce the effort required to input instructions when having an AI service perform processing. [Means for solving the problem]

[0007] The image processing apparatus of the present invention includes a receiving means for receiving a button selection from a user, and, in accordance with the receiving means receiving the selection, reading the document and acquiring image data. First means of acquisition, A second acquisition means for acquiring information from the acquired image data, A natural language instruction registered by the user in association with the button, which is used by the generation AI service for processing using the image data. First The instruction text and the aforementioned image data are used together. As a result of selecting the aforementioned button A transmission means for sending to the aforementioned AI generation service, A generation means that generates a second instruction sentence different from the first instruction sentence based on the acquired information, which is used for processing performed by the generation AI service using the transmitted image data, It has, When the second instruction is generated, the transmission means transmits the second instruction and the acquired image data to the generation AI service without transmitting the first instruction. The transmitted image data and , the first instruction or the second instruction transmitted The process based on this is performed by the generation AI service. [Effects of the Invention]

[0008] According to the present invention, the effort required to input instructions when processing images using an AI service is reduced. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of the information processing system 100 in the first embodiment. [Figure 2] This diagram shows an example of the hardware configuration of each device that makes up the information processing system 100. [Figure 3] Sequence diagram of the processing performed by the information processing system 100 [Figure 4] Flowchart for registering one-touch buttons that support the scan extension function [Figure 5] Flowchart showing an example of a process for executing a process on a document image [Figure 6] Sequence diagram of the process performed by the information processing system 100 in the second embodiment [Figure 7] Sequence diagram of the process performed by the information processing system 100 in the third embodiment [Figure 8] Flowchart showing an example of error notification processing for an instruction sentence in the second embodiment [Figure 9] Flowchart showing an example of instruction sentence setting control in the third embodiment [Figure 10] Diagram showing an example of a setting screen for setting a scan extension function [Figure 11] Diagram showing an example of a data flow and screen for executing a process on a document image [Figure 12] Flowchart showing an example of instruction sentence setting control in the fourth embodiment

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. Note that the components described in the following embodiments are examples and are not intended to limit the scope of the technology of the present disclosure. For example, each part constituting the present invention can be replaced with any configuration that can exhibit the same function. Also, any component may be added.

[0011] <Embodiment 1> FIG. 1 is a block diagram showing an example of the network configuration of the information processing system 100 of the present invention.

[0012] As shown in FIG. 1, the information processing system 100 includes a computer 101 which is a terminal device, an image processing device 102 configured to be able to read a document such as paper, and a generation AI server 103. The computer 101 and the image processing device 102 are arranged, for example, within an office and are communicably connected to each other via an in-house network 104. The in-house network 104 is connected to an external Internet 105 via a router (not shown).

[0013] The generation AI server 103 is communicably connected to the computer 101 and the image processing device 102 via the Internet 105 and the in-house network 104. The generation AI server 103 is a server managed by an operator providing an AI service, and the AI service is provided by the generation AI server 103.

[0014] Note that the generation AI server 103 may be one that can be used in combination with a plug-in that realizes an additional function developed by an operator providing an AI service utilizing the generation AI service. Also, the in-house network 104 may be a wired connection or a wireless connection. In the information processing system 100, the generation AI server 103 may also be arranged within the office, and the computer 101, the image processing device 102, and the generation AI server 103 may be connected via the in-house network 104 and be communicable with each other.

[0015] FIG. 2 is a diagram showing an example of the hardware configuration of the computer 101, the image processing device 102, and the generation AI server 103 included in the information processing system 100.

[0016] Figure 2(a) shows an example of the hardware configuration of computer 101. Computer 101 has a CPU (Central Processing Unit) 201, ROM (Read Only Memory) 202, RAM (Random Access Memory) 204, and storage 205. Computer 101 also has an input device 206, a display device 207, and an external interface 208. Each part is connected to the others via a data bus 203 so that they can communicate with each other.

[0017] The CPU 201 is a control unit that controls the overall operation of the computer 101. The CPU 201 starts up the computer 101 system by executing a startup program stored in the ROM 202, and then executes a control program stored in the storage 205 to realize various functions such as displaying document images and inputting instructions to the generation AI.

[0018] ROM202 is a storage unit such as non-volatile memory that stores the startup program for starting the computer 101. Data bus 203 is a communication unit for sending and receiving data between devices that make up the computer 101. RAM204 is a storage unit such as volatile memory that is used as work memory when the CPU 201 executes the control program. Storage 205 is a storage unit such as an HDD (Hard Disk Drive) that stores various information, such as the aforementioned control program and image data of document images.

[0019] The input device 206 is an operating unit such as a mouse or keyboard that receives operation input from a user operating the computer 101. The display device 207 is a display unit such as a liquid crystal display that displays the settings screen of the image processing device 102 and the input screen of the generation AI server 103 to the user. The external interface 208 is an interface that connects the computer 101 and the network 104, and receives image data of document images from the image processing device 102 and sends instruction messages (prompts) to the generation AI server 103.

[0020] Figure 2(b) shows an example of the hardware configuration of the image processing device 102. The image processing device 102 has a reading function and can use, for example, a scanner, an MFP (Multi-Function Printer / Peripheral) or other multifunction device, an image forming device, or an image processing device. The image processing device 102 has a CPU 231, ROM 232, RAM 234, printer device 235, scanner device 236, document transport device 237, storage 238, input device 239, display device 240, and external interface 241. Each part is connected to the others via a data bus 233 so that they can communicate with each other.

[0021] The CPU 231 is a control unit that controls the overall operation of the image processing device 102. The CPU 231 starts the image processing device 102 system by executing a startup program stored in the ROM 232, and realizes the scanning, printing, faxing, and other functions of the image processing device 102 by executing a startup program stored in the storage 238. The ROM 232 is a storage unit such as non-volatile memory that stores the startup program for starting the image processing device 102. The data bus 233 is a communication unit for sending and receiving data between devices that make up the image processing device 102. The RAM 234 is a storage unit such as volatile memory that is used as work memory when the CPU 231 executes control programs.

[0022] The printer device 235 is an image output device that prints and outputs document images onto a recording medium such as paper.

[0023] The scanner device 236 is an image input device that optically reads a recording medium (original) such as paper on which text, diagrams, etc. are printed. The data (image data) obtained by the scanner device 236 reading the original is acquired as a document image.

[0024] The document transport device 237 is implemented as an ADF (Auto Document Feeder) or the like, and detects documents placed on the document glass and transports the detected documents one by one to the scanner device 236.

[0025] Storage 238 is a storage unit such as an HDD, which stores the aforementioned control program and image data.

[0026] The input device 239 is an operating unit such as a touch panel or hard keys, which receives operation input from a user using the image processing device 102.

[0027] The display device 240 is a display unit such as a liquid crystal display, which displays and outputs the settings screen of the image processing device 102 to the user.

[0028] The external interface 241 is an interface that connects the image processing device 102 and the network 104, and transmits image data to the computer 101 and transmits image data and instructions (prompts) to the generation AI server 103.

[0029] Figure 2(c) shows the hardware configuration of the generation AI server 103. The generation AI server 103 has a CPU 261, ROM 262, RAM 264, and a GPU (Graphics Processing Unit) 265. The generation AI server 103 also has storage 266, an input device 267, a display device 268, and an external interface 269, and each part is connected to the others via a data bus 263 so that they can communicate with each other.

[0030] The CPU 261 is a control unit that controls the overall operation of the generation AI server 103. The CPU 261 starts the generation AI server 103 system by executing a startup program stored in the ROM 262, and then executes a control program stored in the storage 266. The control program executed here utilizes a large language model (LLM) capable of taking multimodal data, including at least images and text, as input, and outputs the result of a conversion according to instructions given in text.

[0031] ROM262 is a storage unit such as non-volatile memory, and stores a startup program for starting the generation AI server 103.

[0032] The data bus 263 is a communication unit for sending and receiving data between devices that make up the generation AI server 103.

[0033] RAM264 is a storage unit such as volatile memory, and is used as work memory when the CPU261 executes a control program.

[0034] The GPU265 is a processing unit composed of image processing processors. For example, the GPU265 performs calculations to transform input image and text data using a Large-Scale Language Model (LLM) according to control commands given by the CPU231.

[0035] Storage 266 is a storage unit such as an HDD, which stores various information such as the aforementioned control program, large-scale language model, image data, and instruction statements (prompts).

[0036] The input device 267 is an operating unit such as a mouse and keyboard, which receives operation input to the generation AI server 103 from a user utilizing the generation AI server 103.

[0037] The display device 268 is a display unit such as a liquid crystal display, which displays the settings screen of the generation AI server 103 to the user using the generation AI server 103.

[0038] The external interface 269 is an interface that enables communication between the generation AI server 103 and the network 104, and receives image data and instructions from the image processing device 102 and sends output results from LLM to the computer 101.

[0039] Figure 3 shows the sequence of processes performed by the information processing system 100. The symbol "S" in the description of each process indicates a step in the sequence, and the same applies to subsequent flowcharts and sequence diagrams. For ease of explanation, user operations will also be described using steps.

[0040] Figure 3(a) illustrates the process of registering a one-touch button corresponding to the scan extension function. This process begins when the user starts the one-touch button registration mode, for example, by pressing a button not shown. The method for configuring the image processing device 102 to extend its scan function using the computer 101 in Figure 3(a) will be explained later using Figure 4. Furthermore, the operation input on the setting screen related to activating the one-touch button in steps S315 to S317 will be explained later using Figure 10.

[0041] In S301, a user of the information processing system 100 places a document such as paper onto the document transport device 237 of the image processing device 102 and instructs the scanning of the document by pressing the scan execution button using the input device 239.

[0042] In S302, the CPU 231 of the image processing device 102 controls the scanner device 236 to read the document placed on the document transport device 237 in S301 and acquire image data of the document based on the document. The CPU 231 performs image processing such as OCR (Optical Character Recognition) processing and handwriting detection on the acquired image data. OCR is an optical character recognition function that extracts text data from parts that can be recognized as text and creates a text-searchable PDF / XPS / OOXML file.

[0043] In S303, the CPU 231 of the image processing device 102 controls the external interface 241 to send the image data of the document image acquired in S302 to the computer 101. The computer 101 receives this data and stores it in the storage 205 as data that can be used by the user in subsequent steps.

[0044] In S304, when computer 101 receives image data of a document image, CPU 201 displays a screen (not shown) on display device 207 for the user to input user information. The user inputs user information for using the AI ​​service provided by generation AI server 103 on the screen displayed on display device 207. Here, user information is used to control access to logs that record input / output data when using the AI ​​service provided by generation AI server 103, and to control billing based on the user's usage of generation AI server 103.

[0045] In S305, the CPU 201 of computer 101 controls the external interface 208 and transmits the user information entered by the user in S304 to the generation AI server 103. The generation AI server 103 receives this information, and its CPU 261 performs user authentication for using the AI ​​service provided by the generation AI server 103 based on the user information received from computer 101.

[0046] In S306, the CPU 261 of the generation AI server 103 controls the external interface 269 to send the result of the user authentication performed in S305 to the computer 101. The following steps will be explained assuming that the user authentication performed by the generation AI server 103 has been completed.

[0047] In S307, when computer 101 receives the user authentication result from generation AI server 103, CPU 201 displays a screen (not shown) on display device 207 for the user to input instructions (prompts). The user inputs instructions (prompts) on the screen displayed on display device 207 to specify what they want to output using the AI ​​service for the image data acquired by image processing device 102 in S302. In other words, CPU 201 receives instructions from the user.

[0048] In S308, the CPU 201 of the computer 101 controls the external interface 208 and sends the image data received from the image processing device 102 in S303 and the instruction text (prompt) entered by the user in S307 to the generation AI server 103. The generation AI server 103 receives this. The instruction text (prompt) is a string of characters that defines the conditions for the processing to be executed by the generation AI server 103.

[0049] In S309, the CPU 261 of the generation AI server 103 controls various parts and performs processing on the image data in accordance with the instruction message (prompt) received from the computer 101 in S308.

[0050] In S310, the CPU 261 of the generation AI server 103 controls the external interface 269 and sends the result of the processing performed in S309 (output result) to the computer 101. The computer 101 receives this.

[0051] In S311, the CPU 201 of computer 101 displays the processing results received from the AI ​​server 103 in S310 on the display device 207. The user checks the output results displayed on the display device 207 and confirms that the desired output results have been obtained for the content instructed in S307. If the output results desired by the user have not been obtained, the user appropriately changes the content of the instruction (prompt) entered in S307 and repeats S307 to S311 until the desired output results are obtained.

[0052] In S312, the user instructs the computer 101 to apply the action to the one-touch button (described later in Figure 4).

[0053] In S313, following instructions from the user in S312, the CPU 201 of the computer 101 controls the external interface 208 to send the instruction text (prompt) entered by the user in S307 to the image processing device 102. The image processing device 102 receives this and the CPU 231 saves it to the storage 238 as a template for the instruction text (prompt) to be used as a one-touch button. In other words, the CPU 231 registers the instruction text (prompt) entered by the user in the instruction text display frame 1080 for the generated AI in S307 as an instruction text template.

[0054] In S314, the user inputs user information from the image processing device 102 to use the AI ​​service provided by the generation AI server 103, and the CPU 231 of the image processing device 102 accepts this information. Here, by using the same user information for user authentication in S304 and user authentication in S314, it becomes possible to use the AI ​​service provided by the generation AI server 103 from both the computer 101 and the image processing device 102. As a result, for example, even if an instruction (prompt) and image data are sent from the image processing device 102 to the generation AI server 103, the output result can be used from the computer 101. The user information is used, for example, to control access to logs that record input / output data when using the AI ​​service provided by the generation AI server 103, and to control billing based on the user's usage of the generation AI server 103.

[0055] In S315, the CPU 231 of the image processing device 102 controls the external interface 241 and transmits the user information entered by the user in S314 to the generation AI server 103. The generation AI server 103 receives this information, and the CPU 261 of the generation AI server 103 performs user authentication from the image processing device 102 to use the AI ​​service provided by the generation AI server 103, based on the user information received from the image processing device 102.

[0056] In S316, the CPU 261 of the generation AI server 103 controls the external interface 269 to send the user authentication result performed in S315 to the image processing device 102.

[0057] In S317, the user sets customizable parameters in the instruction text used by the one-touch button to access the AI ​​service provided by the AI ​​server 103 generated from the image processing device 102. Specifically, the user sets parameters 1061 to 1064, as exemplified in Figure 10(b), from the instruction text (prompt) entered in S307.

[0058] In S318, the user activates a one-touch button to use the AI ​​service provided by the AI ​​server 103 generated by the image processing device 102. Specifically, the user selects "Enabled" in the activation setting 1021 corresponding to the scan extension function 1010 illustrated in Figure 10(a).

[0059] In steps S314-S318, the setting information activated as a one-touch button after user authentication is stored in the storage 238 of the image processing device 102 as information associated with the user. Specifically, the user information for using the AI ​​service is associated with the default setting parameters 1061-1064 (application format, number of pages, page layout, language) used when using the scan extension function 1010 with a one-touch button, and these are stored in association. In other words, the one-touch button corresponding to the scan extension function is registered.

[0060] Figure 3(b) illustrates the process of executing a scan and processing the acquired image data using the one-touch button registered in Figure 3(a). The method of processing image data using the AI ​​service provided by the AI ​​server 103 from the image processing device 102 will be explained later using Figure 5. Furthermore, the operation input on the input screen related to input and output to the AI ​​server 103 will be explained later using Figure 11.

[0061] In S331, the user of the information processing system 100 selects a one-touch button registered in S318, and the CPU 231 accepts this. At this point, multiple one-touch buttons may be registered, and if multiple one-touch buttons are displayed on the display device 207, the user selects the one-touch button corresponding to the desired process. The one-touch button selected by the user simultaneously issues an instruction to perform a document scan and an instruction to use the AI ​​service provided by the generation AI server 103. In other words, once the one-touch button is selected by the user, the instructions from S332 onward are automatically executed without accepting further user instructions, and the user can obtain output from the AI ​​service. Specifically, when the user presses a one-touch button, the image processing device 102 performs the scanning of the document, image processing on the image data acquired by the scan, and transmission of the image data and instruction text to the generation AI server 103. For scanning documents, similar to S301, the document, such as paper, is placed on the document transport device 237 of the image processing device 102, and the scanning is initiated by pressing a one-touch button using the input device 239. A detailed explanation of S331 will be provided later with reference to Figure 11.

[0062] In S332, similar to S302, the CPU 231 of the image processing device 102 controls the scanner device 236 to read the original document and acquire image data of the document image based on the original document. The CPU 231 then performs image processing such as OCR and handwriting detection on the acquired image data of the document image.

[0063] In S333, the CPU 231 of the image processing device 102 controls the external interface 241 and sends the image data of the document image acquired in S332 and the instruction text (prompt) corresponding to the one-touch button selected in S331 to the generation AI server 103. The generation AI server 103 receives this.

[0064] In S334, similar to S309, the CPU 261 of the generation AI server 103 controls each part and executes processing on the image data in accordance with the instruction message (prompt) received from the image processing device 102 in S333.

[0065] In S335, the CPU 261 of the generation AI server 103 controls the external interface 269 and sends the result of the processing performed in S334 to the computer 101. The computer 101 receives this.

[0066] In S336, the CPU 201 of computer 101 displays the processing results (output results) received from the AI ​​server 103 in S335 on the display device 207. The user checks the output results displayed on the display device 207 and confirms that the desired output results have been obtained for the content selected with the one-touch button in S331.

[0067] The reason why the AI ​​service can be logged into from both the computer 101 and the image processing device 102 using the same user information is to allow the AI ​​service to be used commonly by both the computer 101 and the image processing device 102. Specifically, the results of processing by the generation AI server 103 based on the instruction text and image data sent from the image processing device 102 can be checked from the computer 101.

[0068] In this invention, the CPU 261 of the generation AI server 103 authenticates (logs in to) users from the computer 101 and the image processing device 102, and stores them in storage 266 in association with user information. As a result, the generation AI server 103 transmits the results of the processing performed based on the image data and instructions received from the image processing device 102 to the computer 101 based on the information stored in storage 266. In other words, the generation AI server 103 transmits the results of the processing performed by the generation AI server 103 to the computer 101 corresponding to the image processing device 102 that sent the image data and instructions.

[0069] Figure 4 is a flowchart illustrating the process of registering a one-touch button corresponding to the scan extension function described in Figure 3(a). The series of processes shown in the flowchart of Figure 4 are performed by the CPU 201 of the computer 101 and the CPU 231 of the image processing device 102 loading the program code stored in their respective ROMs or storage into RAM and executing the program. This process is initiated when the user starts the one-touch button registration mode, for example, by pressing a button not shown.

[0070] In S401, the CPU 201 controls the external interface 208 and receives image data of a document image acquired by scanning a paper document or other original with the image processing device 102 from the image processing device 102.

[0071] In S402, the CPU 201 receives user information (S304) from the computer 101 via the input device 206, which is used to utilize the AI ​​service provided by the AI ​​server 103.

[0072] In S403, the CPU 201 causes the generation AI server 103 to perform user authentication for use by the computer 101, as described in S305 and S306. Specifically, the CPU 201 controls the external interface 208 to transmit the user information received in S402 to the generation AI server 103.

[0073] In S404, the user inputs a prompt via the input device 206 to instruct the generation AI server 103 on what to process with respect to the image data received in S401, and the CPU 201 accepts this prompt. A specific example of a prompt is to instruct the system to translate or summarize the text obtained by OCR of the strings contained in the document image. Another example is to instruct the system to extract pixels with handwritten markings in the document image and correct the text obtained by OCR of the strings contained in the document image based on the correction instructions given by the handwritten markings.

[0074] In S405, the CPU 201 controls the external interface 208 to receive the processing results (output results) from the generation AI server 103 in response to the instruction text (prompt) entered by the user in S402. For example, it can obtain translated or summarized text from text obtained by OCR of strings contained in a document image. Alternatively, it can obtain text that has been corrected based on correction instructions handwritten on the document image from text obtained by OCR of strings contained in a document image.

[0075] In S406, the CPU 201 receives user input via the input device 206 and determines whether the instruction (prompt) entered in S404 is appropriate. Specifically, the CPU 201 displays the output result received in S405 on the display device 207. The user, upon seeing this, determines whether the output result is appropriate (as expected) and inputs whether the obtained output result is appropriate or not via the input device 206, which the CPU 201 accepts. If the CPU 201 receives information indicating that it is as expected by the user, it determines that the instruction entered in S404 is appropriate. If it receives information indicating that it is not as expected by the user, it determines that the instruction entered in S404 is inappropriate. In other words, if the result is not as expected by the user for the instruction (NO in S406), the content of the instruction (prompt) entered in S404 is changed and S402 to S403 are repeated. If the result is as expected by the user (YES in S406), the process proceeds to S407.

[0076] In S407, CPU201 registers a template of instruction sentences (prompts) according to the user's instructions. Specifically, CPU201 displays the settings screen 1050 for the detailed settings of the scan extension function on the display device 207 and receives the instruction sentences (prompts) entered by the user into the instruction sentence display frame 1080 for the generated AI. CPU201 registers fixed phrases within the instruction sentence as templates and stores them in storage 205. Note that CPU201 may also register the entire instruction sentence as a template. For example, to realize the translation document creation 1011 in Figure 10(a), the following instruction sentence is registered as a template: "Translate into the language of [] with the application format of [], with the page layout of [], so that it fits within the number of pages of []" ([]: parameter).

[0077] In S408, the CPU 231 of the image processing device 102 authenticates the user of the generation AI server 103, as described in S314 and S315. Specifically, the CPU 231 receives user information from the image processing device 102 via the input device 239 to use the AI ​​service provided by the generation AI server 103, controls the external interface 241, and transmits it to the generation AI server 103.

[0078] In S409, CPU201 modifies some keywords in the instruction text (prompt) according to the user's instructions and sets parameters 1061-1064. Specifically, for example, the application format parameter 1061, page number parameter 1062, page layout parameter 1063, and language parameter 1064 are set as parameters used in the instruction text to realize "translation document creation" as exemplified in S405. The user sets the default parameters 1061-1064 in the detailed settings screen 1050 of the scan extension function to register keywords to be used in part of the instruction text (prompt), and CPU231 accepts this. Furthermore, CPU231 accepts a selection (range) of values ​​that can be selected as parameters 1061-1064, which are pre-set as keywords to be used in part of the instruction text (prompt), as exemplified by parameters 1071-1074.

[0079] In S410, CPU201 creates one-touch buttons corresponding to the instruction template and customizable parameter range (including default values) set in S407-S409. CPU201 also places these buttons in the scan extension function menu screen 1110 in a user-selectable format. The information set in S407-S409 is associated with the one-touch buttons and saved to storage 238, creating the one-touch buttons. Note that all instruction texts or parameters registered to the one-touch buttons may be fixed. In that case, the same instruction text will be sent each time the one-touch button is selected.

[0080] Furthermore, when the activation setting 1021 is set to "Enabled," the scan extension function 1010 is activated, the one-touch buttons become functional, and when a button is selected, the corresponding process is executed. Additionally, the information associated with the one-touch buttons may be changeable from the detailed setting 1022.

[0081] Alternatively, the registration of the one-touch button can be done by accessing the image processing device 102 from the computer 101 via the network 104 and entering the information on the remote UI screen, thereby reflecting the settings in the image processing device 102.

[0082] Note that the processing in S408 is not limited to this timing, and the order of other processes may be reversed. The reason why the AI ​​service is logged in from both the computer 101 and the image processing device 102 using the same user information is to allow the AI ​​service to be used commonly by both the computer 101 and the image processing device 102. Specifically, the results of the processing by the generation AI server 103 based on the instruction text and image data sent from the image processing device 102 can be checked from the computer 101.

[0083] In this invention, the CPU 261 of the generation AI server 103 authenticates (logs in to) users from the computer 101 and the image processing device 102, and stores them in storage 266 in association with user information. As a result, the generation AI server 103 transmits the results of the processing performed based on the image data and instructions received from the image processing device 102 to the computer 101 based on the information stored in storage 266. In other words, the generation AI server 103 transmits the results of the processing performed by the generation AI server 103 to the computer 101 corresponding to the image processing device 102 that sent the image data and instructions.

[0084] Alternatively, the computer 101 may access the image processing device 102, and the computer 101 may indirectly execute this process based on instructions from the user.

[0085] Figure 5 is a flowchart illustrating an example of the process for performing image data processing as described in Figure 3(b). The series of processes shown in the flowchart of Figure 5 are described assuming that the CPU 231 of the image processing device 102 loads program code stored in ROM 232 or storage 238 into RAM 234 and executes it. This process is started when a registered one-touch button is selected by the user.

[0086] In S501, the CPU 231 retrieves the instruction template corresponding to the one-touch button, which was registered in S407, from the storage 205. Specifically, for example, if the user selects the translation document creation one-touch button 1121 in Figure 11(b), the CPU 231 retrieves the instruction corresponding to this button, which is shown in the instruction display frame 1080 for the generation AI in Figure 10(b). If the user selects the summary document creation 1012 or the handwritten correction reflection 1013 one-touch button, the instruction saved corresponding to the button is displayed in the instruction display frame 1080 for the generation AI, and the CPU 231 retrieves it in this step.

[0087] Note that what is obtained in S501 does not have to be a template of the instruction text corresponding to the one-touch button, but rather the default values ​​of each parameter that makes up the instruction text, in which case S502 is not necessary.

[0088] In S502, CPU231 retrieves the parameters used in the instruction template set in S407. For example, if the translation document creation one-touch button 1121 in Figure 11(b) is selected by the user, the default values ​​of parameters 1071 to 1074 used in the instruction associated with this button are retrieved.

[0089] In S503, the CPU 231 scans the paper document 1101 shown in Figure 11(a) using the scanner device 236 and acquires image data 1102 of the document image.

[0090] In S504, the CPU 231 performs scanned image processing on the image data 1102 acquired in S503. Specifically, for example, it performs OCR on the image data 1102 to obtain the language and number of characters of the text contained in the document image based on the image data 1102, or it detects handwritten pixel areas contained in the document image based on the image data 1102.

[0091] In S505, the CPU 231 obtains the instruction template acquired in S501 and the parameters to be used in the instruction acquired in S502, and generates an instruction 1103 to instruct the processing to be performed on the image data acquired in S503. An instruction 1103 is generated as shown in the instruction display frame 1080 for the generation AI in Figure 10(b), and parameters 1071 to 1074 are reflected in the instruction 1103.

[0092] In S506, the CPU 231 controls the external interface 241 to send the image data 1102 acquired in S503 and the instruction statement 1103 generated in S505 to the generation AI server 103 that provides the AI ​​service. The output of the generation AI server 103 for the image data 1102 and instruction statement 1103 sent here is sent from the generation AI server 103 to the computer 101, and from the computer 101 to the image processing device 102. Specifically, for example, as shown in Figure 11(c), the output result 1171 of the AI ​​service is obtained by inputting the image data 1102 (attached image 1162) and the instruction statement 1103 (instruction statement 1161) as a pair and performing instruction input 1160. Note that the image processing device 102 may directly receive the output of the generation AI server 103 from the generation AI server 103, or the image processing device 102 may not receive the output of the generation AI server 103.

[0093] Figure 10 shows an example of a settings screen for configuring the scan extension function. The scan extension function settings screen 1000 and the scan extension function detailed settings screen 1050 are examples of setting reception screens displayed on the display device 240 of the image processing device 102 by the CPU 231. These screens may also be displayed on the display device 207 of the computer 101 by the CPU 201.

[0094] Figure 10(a) shows an example of a screen for configuring the scan extension function. On the scan extension function configuration screen 1000, by setting the function to be converted into a one-touch button to "Enabled" using the operation input 1030, which shows an example of a user selection operation, the image processing device 102 makes the scan extension function's one-touch button available. Conversely, by setting the function to "Disabled," the image processing device 102 makes the scan extension function's one-touch button unavailable.

[0095] Specifically, for example, as a scan extension function 1010, settings 1020 for translation document creation 1011, summary document creation 1012, and handwritten correction reflection 1013 include an enable setting 1021 that sets "enabled" or "disabled," and detailed settings 1022 for when enabled. If detailed setting 1022 is selected, the user is redirected to the scan extension function's detailed settings screen 1050.

[0096] The scan extension settings screen 1000 has, for example, a back button 1040 and an OK button 1041. When the back button 1040 is pressed, the user returns to the previous screen without reflecting the changes made on the scan extension settings screen 1000. When the OK button 1041 is pressed, the settings configured at that time are saved.

[0097] If the activation setting 1021 is set to "Enabled," the one-touch button corresponding to the enabled scan extension function 1010 will be displayed on the scan extension function menu screen 1110. On the other hand, if the activation setting 1021 is set to "Disabled," the one-touch button corresponding to the disabled scan extension function 1010 will not be displayed on the scan extension function menu screen 1110. Even if it is set to "Disabled," the corresponding one-touch button may still be displayed on the scan extension function menu screen 1110, and it may be displayed with a different color or shape than the one-touch button corresponding to the enabled function. Furthermore, even if it is displayed in the same way as the one-touch button corresponding to the enabled function, it may be configured to warn with an error message or sound at the user's selection, or it may be configured not to respond even if selected.

[0098] Figure 10(b) shows an example of a screen for entering detailed settings for the scan extension function. The scan extension function detailed settings screen 1050 shown in Figure 10(b) is displayed by transitioning to a screen in response to the operation input 1030 for the detailed settings 1022 for translation document creation 1011 in Figure 10(a). In other words, the scan function detailed settings screen 1050 is displayed on the display device 240 when the user selects the detailed settings 1022 on the scan extension function settings screen 1000. Note that Figure 10(b) is an example of a screen displayed when the detailed settings 1022 corresponding to translation document creation 1011 is selected. If the detailed settings 1022 corresponding to summary document creation 1012 or handwritten correction reflection 1013 are selected, the setting screen corresponding to each function will be displayed.

[0099] The detailed settings screen 1050 for the scan function has a display frame 1080 for instructions to the generating AI, and allows setting default values ​​1071 to 1074 for parameters 1016 to 1064. By setting default values ​​1071 to 1074 corresponding to parameters 1061 to 1064, the user can pre-register standard settings for when to have the generating AI server 103 perform processing using a one-touch button.

[0100] The application format parameter 1061 is used to specify the application and file extension for output. The example "MS Word" specifies that the output will be in Microsoft's Word software (with the .word extension). Other options include "Text" (with the .txt extension) and "Pages" (software provided by Apple).

[0101] The page count parameter 1062 is used to specify the number of pages in the output. The example "8 pages" specifies that the output should fit within 8 pages. Note that the page count can be a maximum or minimum value.

[0102] Page layout parameter 1063 is used to specify the page layout of the output. In addition to the example "two columns," you can specify other layouts that are possible with the specified application, such as "three columns" or "narrow first column."

[0103] The language parameter 1064 specifies the language of the output. In addition to the example "Japanese," various languages ​​corresponding to the processing of the generation AI server 103 can be selected. Alternatively, a "Automatic" option may be provided to retrieve the language settings of the computer 101 and the image processing device 102 and specify the appropriate language.

[0104] Note that the parameters that users can set are not limited to these. Also, these user-configurable parameters change depending on the scan extension function 1010. For example, when creating a summary document 1012, the configuration may allow specifying the number of characters, and when reflecting handwritten corrections 1013, the configuration may allow specifying the color of the corrected part. The settings screen 1050 for detailed settings of the scan function allows users to set all parameters. Furthermore, the parameters that users can set can be customized by the user.

[0105] The instruction display frame 1080 for the generating AI can be modified or changed by user input instructions 1031. Specifically, when the user inputs an instruction 1031, the text displayed in the instruction display frame 1080 for the generating AI becomes editable, and the user can edit the text using an input device 239 or the like. For example, as an instruction for creating a translated document 1011, the user can edit, delete, or add instructions such as, "Translate into the language of [] with the application format of [], with the page layout of [], so that it fits within the number of pages of []."

[0106] The settings screen 1050 for the advanced settings of the scan extension function has, for example, a back button 1090 and an OK button 1091. When the back button 1090 is pressed, the user returns to the previous screen without reflecting the changes made on the settings screen 1050 for the advanced settings of the scan extension function. When the OK button 1091 is pressed, the settings configured at that time are saved.

[0107] Figure 11 shows an example of a data flow and screen for performing processing on image data.

[0108] Figure 11(a) is a diagram showing the data flow for processing image data. The CPU 231 of the image processing device 102 of the present invention scans a paper document 1101 placed on the document tray of the document transport device 237 by the user, and sends the generated (acquired) image data 1102 and instruction text 1103 to the generation AI server 103. The output results from the generation AI server 103 can be received by the computer 101. Here, as shown in S305, S306, S314, and S315 of Figure 3, since the user to use the AI ​​service provided by the generation AI server 103 is authenticated from each device, the user's data can be accessed regardless of the device being operated. Specifically, the user can access the output of the generation AI server 103 by operating the computer 101 or by operating the image processing device 102.

[0109] Figure 11(b) shows an example of an operation screen displayed on the display device 240 of the image processing device 102. The user can instruct the execution of the corresponding process by selecting a one-touch button on the scan extension function menu screen 1110 displayed on the touch panel screen of the image processing device 102. The CPU 231 receives user instruction input 1131 via the input device 239 (touch panel).

[0110] The scan extension menu screen 1110 displays the one-touch buttons 1121 for creating translated documents, 1122 for creating summarized documents, and 1123 for reflecting handwritten corrections, corresponding to the scan extensions 1011-1013 registered by the user. The user can input execution commands by selecting the one-touch button displayed on the scan extension menu screen 1110. If a scan extension is not registered, its one-touch button will not be displayed on the scan extension menu screen 1110. Also, one-touch buttons corresponding to scan extensions set to "disabled" in the scan extension settings screen 1000 will not be displayed on the scan extension menu screen 1110. It is not mandatory to hide the one-touch buttons when they are not registered or when the scan extension is set to "disabled"; the system may change the color or shape of the one-touch buttons to make them recognizable to the user. Furthermore, even if they are displayed in the same way as the one-touch buttons corresponding to functions set to "enabled," the system may provide an error message or sound warning based on the user's selection, or it may be configured not to respond even if selected.

[0111] Furthermore, users of the image processing device 102 who use the generation AI server 103 will use the services provided by the generation AI server 103 as users who have been authenticated in S314 and S315. Here, the user information 1111 of users using the generation AI server 103 may be linked to the user information of the logged-in user of the image processing device 102 and stored and managed in the storage 238. The CPU 231 of the image processing device 102 detects that the logout button 1112 has been pressed and terminates the reception of instruction input 1131 by the logged-in user.

[0112] The AI ​​service screen 1150 in Figure 11(c) is an example of a screen displayed on the display device 207 when using the AI ​​service provided by the AI ​​generation server 103 from the computer 101. The AI ​​service screen 1150 may also be displayed on the display device 240 of the image processing device 102 by the CPU 231. The AI ​​service screen 1150 provided by the AI ​​generation server 103 displays the response (output result 1170) from the AI ​​generation to the user's instruction input 1160. The user's instruction input 1160 is an instruction (prompt) generated by the CPU 231 of the image processing device 102 in S505 and sent to the AI ​​generation server 103 in S506.

[0113] The AI ​​service screen 1150 in Figure 11(c) shows how the user's instruction input, instruction text 1161 (instruction text 1103), is automatically entered, image data 1102 is automatically attached to instruction text 1103 (attached image 1162), and then the message is sent.

[0114] Furthermore, users of computer 101 who use the AI ​​service provided by the generation AI server 103 use the AI ​​service as users who have been authenticated in S304 to S306. Here, the user information 1151 of users using the generation AI server 103 may be linked to the user information of the logged-in user of computer 101 and stored and managed in storage 205. The CPU 201 of computer 101 detects that the logout button 1112 has been pressed and terminates the reception of instruction input 1131 by the logged-in user of the user information 1151.

[0115] Figure 11(b) shows an example of an operation screen displaying a list of one-touch buttons for registered scan extension functions. However, any configuration using a different screen is acceptable as long as the CPU 231 is configured to accept (receive) operation input in line with the essence of the present invention. For example, an operation screen extending the conventional SEND function for sending scanned images via email may be used to accept user input by sending to a user-authenticated generation AI server 103. Alternatively, the one-touch buttons may be displayed on the menu screen or settings screen for the copy process executed by the image processing device 102, or the functions provided on the image processing device 102 may be customizable physical buttons. Furthermore, the one-touch buttons may be displayed superimposed on the screen by a specific operation, or they may be invoked by voice input.

[0116] As described above, according to the present invention, by transmitting image data obtained by scanning a document and text (instructions) based on the user's instructions to an AI service, the AI ​​service outputs the results processed in accordance with the user's intentions, and the user can receive these results.

[0117] Note that "generative AI (AI services)" is a general term for services that utilize AI. Examples include text generation AI services such as ChatGPT (artificial intelligence chatbot) provided by OpenAI and Copilot provided by Microsoft. Also, examples include image generation AI services such as Stable Diffsion and Midjourney.

[0118] Generative AI, on the other hand, is an AI that can generate various types of content. It learns data patterns and relationships (deep learning) and aims to generate new content.

[0119] In this invention, when the translation document creation process and the summary document creation process are performed by an AI service, the image processing device 102 may send the text obtained as a result of OCR on the image data to the generation AI server 103 instead of the image data.

[0120] <Embodiment 2> Embodiment 1 describes a method of acquiring image data of a document from a paper original by inputting instructions to a one-touch button registered in the image processing device 102, generating instruction text based on set parameters, and transmitting these to the generation AI server 103. Embodiment 2 describes a method of detecting and notifying errors occurring in the generation AI server 103 based on the results of image processing performed by the image processing device 102 before transmitting them to the generation AI server 103.

[0121] Figure 6 shows the sequence of the information processing system 100 in Embodiment 2. Note that steps in the sequence shown in Figure 6 that are common to Figure 3(b) are the same as in Embodiment 1, and therefore their explanation is omitted.

[0122] Sections S601 to S602 are the same as S331 to S332, so their explanation will be omitted.

[0123] In S603, the CPU 231 of the image processing device 102 detects whether the instruction text corresponding to the one-touch button selected in S601 and the image data of the document image acquired in S602 meet the data requirements for using the AI ​​service. In other words, it checks whether an error is likely to occur when the AI ​​service processes the data. For example, if the translated document creation one-touch button 1121 is selected, the CPU 231 compares the language of the text contained in the document image of the OCR result with the target language set in the instruction text, and if they are the same language, it detects that an error is likely to occur where the translation cannot be performed. Similarly, if the summary document creation one-touch button 1122 is selected, the CPU 231 compares the number of characters in the text contained in the document image with the number of characters after summarization set in the instruction text, and if the former is less than the latter, it detects that an error will occur where the summary cannot be performed. Furthermore, if the handwritten correction reflection one-touch button 1123 is selected, it detects that an error will occur where the handwritten correction cannot be reflected if pixels with handwritten text cannot be extracted from the document image as a result of image processing.

[0124] In S604, the CPU 231 of the image processing device 102 notifies the user of the error detected in S603. Specifically, it notifies the user that there is a possibility of an error occurring during the generation AI input in the translation document creation, summary document creation, and handwritten correction reflection functions as shown in S603. Specifically, the CPU 231 displays a message on the display device 240 informing the user that there is a possibility of an error occurring during the generation AI input.

[0125] In S605, the user modifies the settings to correct part of the instruction text corresponding to the one-touch button, taking into account the error notified in S604, and the CPU231 accepts this change. Specifically, for example, the user changes the target language in the translation document creation function. Also, in the summary document creation function, the user changes the character count information corresponding to the number of pages after summarization. Furthermore, in the handwriting correction reflection function, the user reconfirms whether the document image acquired in S602 contains handwriting and changes it to a document image that includes handwriting.

[0126] Sections S606 to S609 are the same as S333 to S336, so their explanation will be omitted.

[0127] Figure 8 is a flowchart illustrating the error notification flow for instruction statements in Embodiment 2. Note that steps in the flowchart shown in Figure 8 that are common to Figure 5 are the same as in Embodiment 1 and therefore their explanation is omitted. The series of processes shown in the flowchart of Figure 8 are performed by the CPU 231 of the image processing device 102 loading the program code stored in the ROM 232 or storage 238 into the RAM 234 and executing the program. This process is initiated when a registered one-touch button is selected by the user.

[0128] Sections S801-S804 are the same as S501-S504, so their explanation will be omitted.

[0129] In S805, CPU231 detects whether the instruction text corresponding to the one-touch button selected in S601 and the image data of the document image acquired in S602 meet the data requirements for using the AI ​​service. In other words, it determines whether an error is likely to occur when the AI ​​service processes the data. For example, if the translated document creation one-touch button 1121 is selected, it compares the language of the text contained in the document image of the OCR result with the target language set in the instruction text, and if they are the same language, it detects that an error is likely to occur where the translation cannot be performed. Similarly, if the summary document creation one-touch button 1122 is selected, it compares the number of characters in the text contained in the document image with the number of characters after summarization set in the instruction text, and if the former is less than the latter, it detects that an error will occur where the summary cannot be performed. Furthermore, if the handwriting correction reflection one-touch button 1123 is selected, it detects that an error will occur where the handwriting correction cannot be reflected if pixels with handwritten text in the document image cannot be extracted as a result of image processing.

[0130] If CPU231 determines that an error is likely to occur when using the generated AI (YES in S805), it proceeds to S806. If it does not determine that an error is likely to occur when using the generated AI (NO in S805), this process terminates. In other words, CPU231 decides to notify if it determines that an error is likely to occur when using the generated AI, and proceeds to S806.

[0131] In S806, the CPU 231 notifies the user of the error detected in S603. Specifically, it notifies the user that there is a possibility of an error occurring during AI generation input in the translation document creation, summary document creation, and handwritten correction reflection functions as described in S603. Specifically, the CPU 231 displays a message on the display device 240 informing the user that there is a possibility of an error occurring during AI generation input.

[0132] As described above, according to this embodiment, before the image processing device 102 transmits image data and instruction text to the generation AI server 103, it is possible to detect and notify errors occurring in the generation AI server 103 based on the results of image processing performed by the image processing device 102.

[0133] <Embodiment 3> Embodiment 3 describes a method for controlling or recommending parameters for generating an instruction to be sent to the generation AI server 103, based on the results of image processing performed by the image processing device 102, before the image processing device 102 transmits the image data and the instruction. Here, control means that the CPU 231 of the image processing device 102 changes the parameters or limits the parameters that the user can select. Recommendation means displaying a prompt to the user to change the parameters or displaying the suggested parameters higher in the list of parameter candidates.

[0134] Figure 7 shows the sequence of the information processing system 100 in Embodiment 3. Note that steps in the sequence shown in Figure 7 that are common to Figure 3(b) are the same as in Embodiment 1, and therefore their explanation is omitted.

[0135] Sections S701 to S702 are the same as S331 to S332, so their explanation will be omitted.

[0136] In S703, the CPU 231 of the image processing device 102 obtains recommended settings for the instruction text to be used by the one-touch button selected in S701, based on the image processing results of the image data acquired by S702. Specifically, for example, in the case of the translation document creation function, it obtains a recommended setting range for the language parameter based on the language setting set in the image processing device 102. For example, if the main setting of the image processing device 102 is Japanese, "Japanese" is obtained. In the case of the summary document creation function, it obtains a recommended setting range for the page number parameter based on the number of characters in the text contained in the document image obtained by OCR of the image data. For example, if the text contained in the document image is 3000 characters, "3 pages" is obtained. Furthermore, in the handwriting correction reflection function, it obtains recommended settings for the processing target range, such as handwritten instructions and handwritten circles, and pixels (noise) that were mistakenly detected as handwriting, based on the handwritten objects and pixel count of the handwriting detection results.

[0137] The recommended setting range can be pre-configured by the user, and the S703 process is executed by referring to the recommended setting range stored in storage 238. Note that the recommended setting range may differ for each scan extension function.

[0138] In S704, the CPU 231 of the image processing device 102 controls the range of selectable parameters for the instruction text used by the one-touch button selected in S701, or recommends recommended settings, based on the settings acquired in S703. Specifically, for example, in the translation document creation function, it controls the language parameter based on the language setting of the image processing device 102 acquired in S703. In the summary document creation function, it controls the selection so that the number of pages parameter is less than or equal to the net number of pages of the OCR result string. In the handwriting correction reflection function, it controls the processing range for handwritten instructions, handwritten circles, etc., or pixels (noise) that were mistakenly detected as handwriting.

[0139] CPU231 controls the user to limit the range of available instruction options based on the image processing results, but it may also display recommended settings to the user, allowing the user to make their own decisions regarding the settings.

[0140] In S705, the user modifies the settings to include a portion of the instruction text corresponding to the one-touch button, taking into account the settings range controlled by S704 or the recommended settings, and CPU231 accepts this change. Specifically, for example, the user can change the target language in the translation document creation function, change the number of pages after summarization to a value smaller than the net number of pages in the OCR result string in the summary document creation function, or modify the instruction text in the handwriting correction reflection function so that handwritten circles are interpreted as the processing range. Note that user changes to settings are not mandatory, and if the user does not need to change the settings, they do not need to change to the controlled or recommended settings.

[0141] Sections S706-S709 are the same as S333-S336, so their explanation will be omitted.

[0142] Figure 9 is a flowchart illustrating the flow of instruction setting control in Embodiment 3. Note that steps in the flowchart shown in Figure 9 that are common to Figure 5 are the same as in Embodiment 1 and therefore their explanation is omitted. The series of processes shown in the flowchart of Figure 9 are performed by the CPU 231 of the image processing device 102 loading the program code stored in the ROM 232 or storage 238 into the RAM 234 and executing the program. This process is initiated when a registered one-touch button is selected by the user.

[0143] Sections S901-S904 are the same as S501-S504, so their explanation will be omitted.

[0144] In S905, the CPU 231 obtains recommended settings for the instruction text to be used by the one-touch button selected in S701, based on the image processing results of the image data acquired in S702. Specifically, for example, in the case of the translation document creation function, it obtains a recommended setting range for the language parameter based on the language setting set in the image processing device 102. For example, if the main setting of the image processing device 102 is Japanese, "Japanese" is obtained. In the case of the summary document creation function, it obtains a recommended setting range for the page number parameter based on the number of characters in the text contained in the document image obtained by OCR of the image data. In addition, in the handwriting correction reflection function, it obtains recommended settings for the processing target range, such as handwritten instructions and handwritten circles, and pixels (noise) that were mistakenly detected as handwriting, based on the handwritten objects and pixel count of the handwriting detection results. If there is a mismatch with the conditions when using the generation AI (YES in S905), it proceeds to S906; if there is no mismatch with the conditions when using the generation AI (NO in S905), it terminates.

[0145] In S906, the CPU 231 controls the range of selectable parameters for the instruction text used by the one-touch button selected in S701, or recommends suitable settings, based on the settings acquired in S905. Specifically, for example, in the translation document creation function, it controls the language parameter based on the language setting of the image processing device 102 acquired in S703. In the summary document creation function, it controls the selection so that the number of pages parameter is less than or equal to the net number of pages of the OCR result string. In the handwriting correction reflection function, it controls the processing range for handwritten instructions, handwritten circles, etc., based on the handwriting detection results as image processing results, or for pixels (noise) that were mistakenly detected as handwriting.

[0146] CPU231 controls the user to limit the range of available instruction options based on the image processing results, but it may also display recommended settings to the user, allowing the user to make their own decisions regarding the settings.

[0147] As described above, according to this embodiment, the image processing device 102 can control or recommend the instruction text to be sent to the generation AI server 103 based on the results of the image processing performed before sending the image data and instruction text to the generation AI server 103.

[0148] <Embodiment 4> Embodiment 4 describes a method in which the image processing device 102 recommends parameters for generating instructions to be sent to the generation AI server 103 using user information (e.g., attributes, usage history) before sending image data and instructions to the generation AI server 103. In other words, the image processing device 102 in this embodiment acquires user information.

[0149] Figure 12 is a flowchart illustrating the flow of instruction setting control in Embodiment 4. Note that steps in the flowchart shown in Figure 12 that are common to Figure 9 are the same as in Embodiment 3, and therefore their explanation is omitted. The series of processes shown in the flowchart of Figure 12 are performed by the CPU 231 of the image processing device 102 loading the program code stored in the ROM 232 or storage 238 into the RAM 234 and executing the program. This process is initiated when a registered one-touch button is selected by the user.

[0150] In S1201, CPU231 acquires user information, including user attributes and usage history. User attributes include, for example, information about the language the user primarily uses, and user usage history includes information about the application software used most recently or frequently used as the processing destination for image data by the scanning extension function. CPU231 acquires this information.

[0151] Sections S1202 to S1205 are the same as S901 to S904, so their explanation will be omitted.

[0152] In S1206, CPU231 uses the user information obtained in S1201 to obtain recommended settings. Specifically, for example, it compares the language of the text contained in the document image obtained by OCR of the image data with the language the user primarily uses. It also compares the application set in the application format parameter 1061 with the application obtained from the user's usage history. If there is a mismatch with the user information (YES in S1206), the process proceeds to S1207; if there is no mismatch with the user information (NO in S1206), the process terminates.

[0153] In S1207, CPU231 displays recommendation settings to the user according to the result of S1206. Specifically, for example, it displays recommendation settings that instruct the user to translate the instruction into the language they primarily use. Alternatively, for example, it displays recommendation settings that instruct the user to convert the instruction into an application derived from their usage history (most recently used, frequently used).

[0154] As described above, according to this embodiment, before the image processing device 102 transmits image data and instructions to the generation AI server 103, it can recommend (recommend) instructions to be sent to the generation AI server 103 based on the results of image processing and user information.

[0155] <Variation> In a modified example, the information processing system 100 is described as consisting of an image processing device 102 and a generation AI server 103. That is, the user scans the document with the image processing device 102, and the processing does not go through the computer 101, but the final output result from the generation AI server 103 is confirmed on the image processing device 102. The sequence of processing in this configuration will be explained by focusing on the main entity performing the processing, as described in Figure 3, for processes S331 to S336.

[0156] In S331, the user using the information processing system 100 selects the one-touch button registered in S318.

[0157] In S332, the CPU 231 of the image processing device 102 controls the scanner device 236 to read the original document and acquire image data of the document based on the original document. The CPU 231 then performs image processing such as OCR and handwriting detection on the acquired image data.

[0158] In S333, the CPU 231 of the image processing device 102 controls the external interface 241 and sends the image data acquired in S332 and the instruction text (prompt) corresponding to the one-touch button selected in S331 to the generation AI server 103. The generation AI server 103 receives this.

[0159] In S334, the CPU 261 of the generation AI server 103 controls various parts and performs processing on the image data in accordance with the instruction message (prompt) received from the image processing device 102 in S333.

[0160] In S335, the CPU 261 of the generation AI server 103 controls the external interface 269 and sends the result of the processing performed in S334 (output result) to the image processing device 102. The image processing device 102 receives this.

[0161] In S336, the CPU 231 of the image processing device 102 displays the output results received from the AI ​​generation server 103 in S335 on the display device 240. The user checks the output results displayed on the display device 240 and confirms that the desired output results have been obtained for the content selected with the one-touch button in S331.

[0162] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0163] 100 Information Processing Systems 101 Computer 102 Image Processing Device 103 Generation AI Server

Claims

1. A means of receiving button selections from the user, In accordance with the reception means receiving the selection, a first acquisition means reads the document and acquires image data, A second acquisition means for acquiring information from the acquired image data, A first instruction sentence, which is a natural language instruction sentence registered by the user in association with the button and used by the generation AI service for processing using the image data, and a transmission means for transmitting the image data to the generation AI service when the button is selected. The system includes a generation means that generates a second instruction sentence, which is different from the first instruction sentence, based on the acquired information, and which is used for processing performed by the generation AI service using the transmitted image data. When the second instruction is generated, the transmission means transmits the second instruction and the acquired image data to the generation AI service without transmitting the first instruction. The AI ​​generation service performs processing based on the transmitted image data and the transmitted first instruction or second instruction. An image processing apparatus characterized by the following:

2. The system further includes a setting reception means for receiving settings from the user for the processing to be executed by the generation AI service, The transmission means transmits the instruction text registered in association with the button based on the setting received by the setting receiving means, and the image data to the generation AI service. The image processing apparatus according to feature 1.

3. The system further includes an instruction receiving means for receiving the aforementioned instruction from the user, The transmission means transmits the instruction text, which has been registered in association with the button upon receipt by the instruction text receiving means, and the image data to the generation AI service. The image processing apparatus according to feature 1.

4. The image processing apparatus according to claim 1, characterized in that the second acquisition means acquires information based on character information obtained by performing image processing on the acquired image data.

5. The image processing apparatus according to claim 4, further comprising means for receiving settings from a user for the processing to be performed by the generation AI service.

6. The aforementioned information is the number of characters contained in the image shown by the image data, The above setting is for the number of characters, The image processing apparatus according to claim 5, further comprising a first determination means for determining whether to notify the user based on the number of characters based on the settings received by the aforementioned means and the number of characters acquired by the second acquisition means.

7. It further has a first notification means, The first determination means decides to issue the notification if the number of characters based on the settings received by the means exceeds the number of characters acquired by the second acquisition means. The first notification means notifies based on the decision. The image processing apparatus according to claim 6.

8. The aforementioned information is information indicating the language of the characters contained in the image shown by the image data, The aforementioned setting is a language setting. The image processing apparatus according to claim 5, further comprising a second determination means for determining whether to notify the user based on the language setting received by the aforementioned means and the language information acquired by the second acquisition means.

9. It further has a second means of notification, The second determination means decides to issue the notification if the language indicated by the language setting received by the means is different from the language indicated by the language information acquired by the second acquisition means. The second notification means notifies based on the decision. The image processing apparatus according to feature 8.

10. The means restricts the settings accepted from the user based on the information acquired by the second acquisition means. The image processing apparatus according to feature 5.

11. The image processing device further includes a first information acquisition means for acquiring user information of a user who logs in to the image processing device, The means limits the settings to be accepted from the user based on the user information of the user acquired by the first information acquisition means. The image processing apparatus according to feature 5.

12. The image processing apparatus according to claim 11, characterized in that the user information refers to the language used by the user.

13. The image processing apparatus according to claim 11, characterized in that the user information refers to a history of functions used by the user.

14. The image processing device further includes a second information acquisition means for acquiring user information of a user who logs in to the image processing device, The means recommends the content of the settings to be made by the user based on the user information of the user acquired by the second information acquisition means. The image processing apparatus according to feature 5.

15. The image processing apparatus according to claim 14, characterized in that the user information refers to the language and function usage history used by the user.

16. Display means and Receiving means for receiving processing results from the aforementioned AI generation service, It further possesses, The display means displays the processing result received by the receiving means. The image processing apparatus according to feature 1.

17. The image processing apparatus according to claim 16, characterized in that the display means displays the natural language instruction sentence input to the generation AI service while the received processing result is displayed.

18. The first acquisition means reads the document and acquires image data in accordance with the fact that the receiving means has received the selection. The transmission means transmits to the generation AI service the instruction statement registered by the user in association with the button, which is used for processing performed by the generation AI service using the image data, and the image data. The image processing apparatus according to feature 1.

19. The image processing apparatus according to claim 1, characterized in that the first acquisition means acquires image data by reading the document placed on the document table by transporting the document.

20. The image processing apparatus according to claim 1, characterized in that the image processing apparatus is a scanner.

21. The image processing apparatus according to claim 1, wherein the instruction statement in natural language registered by the user in association with the button, and which is used for the processing performed by the generation AI service using the image data, is an instruction statement that specifies the format of the processing result output by the generation AI service.

22. A program characterized by causing a computer to execute each of the means described in any one of claims 1 to 21.

23. A reception process that accepts button selections from the user, In accordance with the acceptance of the selection in the acceptance process, a first acquisition process is performed in which the document is read and image data is obtained, A second acquisition step of acquiring information from the acquired image data, A first instruction, which is a natural language instruction registered by the user in association with the button and used by the generating AI service for processing using the image data, and a transmission step of sending the image data to the generating AI service when the button is selected. The process includes generating a second instruction sentence, which is different from the first instruction sentence, based on the acquired information, and which is used by the generation AI service to perform processing using the transmitted image data. When the second instruction is generated, the transmission step transmits the second instruction and the acquired image data to the generation AI service without transmitting the first instruction. The AI ​​generation service performs processing based on the transmitted image data and the transmitted first instruction or second instruction. An image processing method characterized by the following:

Images