Image processing apparatus, image processing method, and storage medium

Abstract

To reduce time and effort required for inputting an instruction sentence when making a AI service execute processing.SOLUTION: According to an aspect of the present invention, an image processing apparatus includes a receiving unit configured to receive a selection of a button from a user, an obtaining unit configured to obtain an image by reading a document in response to the receiving unit receiving the selection, and a transmitting unit configured to transmit, to a AI service, the image and an instruction registered by the user in association with the button, the instruction being used for a process performed by the AI service using the image.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, generative AI (Artificial Intelligence), which can automatically generate creative content such as images, text, and audio, has rapidly become popular. Accordingly, a variety of services (AI services) using generative AI have been provided. Patent Document 1 discloses an image generation device that creates an instruction statement including text corresponding to a tag selected by a user, inputs the instruction statement to a generative AI, and outputs an image generated using the generative AI. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7398723 Summary of the Invention [Problem to be solved by the invention]

[0004] An AI service may process the image obtained by scanning a document, or output something new based on the scanned image. Specifically, for example, an image obtained by scanning a document and instructions may be sent to an AI service to perform a process to summarize the text contained in the image. When an AI service is asked to perform a process to summarize the text contained in an image, the instructions may be, for example, "Summarize the content of the image in 400 characters or less." If the AI ​​service is asked to perform a summary process on 10 images, the images sent will change each time, but the instructions will remain the same.

[0005] However, in the past, even when having an AI service perform the same processing, users had to input instructions each time, which was time-consuming.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to reduce the effort required to input instructions when having an AI service execute processing. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized by having a receiving means for receiving a button selection from a user, an acquisition means for reading a document and acquiring image data in accordance with the reception of the selection by the receiving means, and a transmission means for transmitting an instruction statement registered by the user in association with the button, which instruction statement is used by an AI service for processing using the image data, and the image data to the AI ​​service. [Effects of the Invention]

[0008] According to the present invention, the effort required to input instructions when processing an image using an AI service is reduced. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 shows an example of the configuration of an information processing system 100 according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the hardware configuration of each device constituting an information processing system 100. [Figure 3] Sequence diagram of processing performed by the information processing system 100 [Figure 4] Flowchart for registering one-touch buttons that support the scan expansion function [Figure 5] 1 is a flowchart showing an example of a process for executing a process on a document image; [Figure 6] FIG. 10 is a sequence diagram of a process performed by the information processing system 100 according to the second embodiment. [Figure 7] 10 is a sequence diagram of a process performed by the information processing system 100 according to the third embodiment. [Figure 8]10 is a flowchart illustrating an example of a directive statement error notification process according to the second embodiment. [Figure 9] 10 is a flowchart showing an example of setting control of directives in the third embodiment. [Figure 10] Figure showing an example of the setting screen for setting the scan extension function [Figure 11] FIG. 10 is a diagram showing an example of a data flow and a screen for executing processing on a document image. [Figure 12] 10 is a flowchart showing an example of setting control of directives in the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components described in the following embodiments are merely examples and are not intended to limit the scope of the technology of the present disclosure. For example, each part constituting the present invention can be replaced with any other component that can perform the same function. Also, any other component may be added.

[0011] <Embodiment 1> FIG. 1 is a block diagram showing an example of a network configuration of an information processing system 100 according to the present invention.

[0012] As shown in Fig. 1, the information processing system 100 includes a computer 101, which is a terminal device, an image processing device 102 configured to be able to read documents such as paper, and a generation AI server 103. The computer 101 and the image processing device 102 are located, for example, in an office and are connected to each other so that they can communicate with each other via an internal network 104. The internal network 104 is connected to the external Internet 105 via a router (not shown).

[0013] The generation AI server 103 is connected to the computer 101 and the image processing device 102 so that they can communicate with each other via the Internet 105 and the in-house network 104. The generation AI server 103 is a server managed by a business that provides AI services, and the AI ​​services are provided by the generation AI server 103.

[0014] The generation AI server 103 may be capable of being used in combination with plug-ins that provide additional functions developed by businesses that provide AI services utilizing the generation AI service. The in-house network 104 may be a wired or wireless connection. In the information processing system 100, the generation AI server 103 may also be located within an office, and the computer 101 and image processing device 102 may be connected to the generation AI server 103 via the in-house network 104 so that they can communicate with each other.

[0015] FIG. 2 is a diagram showing an example of the hardware configuration of the computer 101, the image processing device 102, and the generation AI server 103 included in the information processing system 100.

[0016] 2(a) is a diagram showing an example of the hardware configuration of a computer 101. The computer 101 has a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 204, and a storage 205. The computer 101 also has an input device 206, a display device 207, and an external interface 208. Each component is connected to each other via a data bus 203 so as to be able to communicate with each other.

[0017] The CPU 201 is a control unit that controls the overall operation of the computer 101. The CPU 201 starts up the system of the computer 101 by executing a startup program stored in the ROM 202, and realizes various functions such as displaying document images and inputting instructions to the generation AI by executing a control program stored in the storage 205.

[0018] The ROM 202 is a storage unit such as a nonvolatile memory, and stores a startup program that starts up the computer 101. The data bus 203 is a communication unit for transmitting and receiving data between devices that make up the computer 101. The RAM 204 is a storage unit such as a volatile memory, and is used as a work memory when the CPU 201 executes a control program. The storage 205 is a storage unit such as an HDD (Hard Disk Drive), and stores various information such as the control program and image data of document images.

[0019] The input device 206 is an operation unit such as a mouse or keyboard, and accepts operation input from a user operating the computer 101. The display device 207 is a display unit such as an LCD display, and displays the setting screen of the image processing device 102 and the input screen of the generation AI server 103 to the user. The external interface 208 is an interface that connects the computer 101 and the network 104, and receives image data of document images from the image processing device 102 and sends instructions (prompts) to the generation AI server 103.

[0020] 2(b) is a diagram showing an example of the hardware configuration of the image processing device 102. The image processing device 102 has a reading function, and may be, for example, a scanner, a multifunction peripheral such as an MFP (Multi-Function Printer / Peripheral), an image forming device, or an image processing device. The image processing device 102 has a CPU 231, a ROM 232, a RAM 234, a printer device 235, a scanner device 236, a document transport device 237, a storage 238, an input device 239, a display device 240, and an external interface 241. Each unit is connected to each other via a data bus 233 so as to be able to communicate with each other.

[0021] The CPU 231 is a control unit that controls the overall operation of the image processing device 102. The CPU 231 starts up the system of the image processing device 102 by executing a startup program stored in the ROM 232, and realizes functions of the image processing device 102, such as scanning, printing, and faxing, by executing a startup program stored in the storage 238. The ROM 232 is a storage unit such as a non-volatile memory, and stores the startup program that starts up the image processing device 102. The data bus 233 is a communication unit for transmitting and receiving data between devices that make up the image processing device 102. The RAM 234 is a storage unit such as a volatile memory, and is used as a work memory when the CPU 231 executes the control program.

[0022] The printer device 235 is an image output device that prints and outputs a document image on a recording medium such as paper.

[0023] The scanner device 236 is an image input device that optically reads a recording medium (original) such as paper on which characters, diagrams, etc. are printed. The data (image data) obtained by the scanner device 236 reading the original is acquired as a document image.

[0024] The document transport device 237 is realized by an ADF (Auto Document Feeder) or the like, detects documents placed on a document table, and transports the detected documents one by one to the scanner device 236 .

[0025] The storage 238 is a storage unit such as an HDD, and stores the control programs and image data described above.

[0026] The input device 239 is an operation unit such as a touch panel or hard keys, and receives operation inputs from a user who uses the image processing apparatus 102 .

[0027] The display device 240 is a display unit such as a liquid crystal display, and displays and outputs the setting screen of the image processing device 102 to the user.

[0028] The external interface 241 is an interface that connects the image processing device 102 and the network 104, and transmits image data to the computer 101 and transmits image data and instructions (prompts) to the generation AI server 103.

[0029] 2(c) is a diagram showing the hardware configuration of the generation AI server 103. The generation AI server 103 has a CPU 261, a ROM 262, a RAM 264, and a GPU (Graphics Processing Unit) 265. The generation AI server 103 further has a storage 266, an input device 267, a display device 268, and an external interface 269, and each part is connected to each other via a data bus 263 so that they can communicate with each other.

[0030] The CPU 261 is a control unit that controls the overall operation of the generation AI server 103. The CPU 261 starts up the system of the generation AI server 103 by executing a startup program stored in the ROM 262, and executes a control program stored in the storage 266. The control program executed here utilizes a large language model (LLM) that can input multimodal data of at least images and text, and outputs the results of conversion according to instructions given in text.

[0031] The ROM 262 is a storage unit such as a nonvolatile memory, and stores a startup program that starts the generation AI server 103.

[0032] The data bus 263 is a communication unit for transmitting and receiving data between the devices that make up the generation AI server 103.

[0033] The RAM 264 is a storage unit such as a volatile memory, and is used as a work memory when the CPU 261 executes the control program.

[0034] The GPU 265 is a computing unit configured with an image processing processor. For example, the GPU 265 performs calculations to convert input image or text data using a large-scale language model (LLM) in accordance with control commands provided by the CPU 231.

[0035] The storage 266 is a storage unit such as an HDD, and stores various information such as the control programs, large-scale language models, image data, and prompts.

[0036] The input device 267 is an operation unit such as a mouse and keyboard, and accepts operation input to the generation AI server 103 from a user who uses the generation AI server 103 .

[0037] The display device 268 is a display unit such as a liquid crystal display, and displays the setting screen of the generation AI server 103 to the user who uses the generation AI server 103.

[0038] The external interface 269 is an interface that connects the generation AI server 103 and the network 104 so that they can communicate with each other, and receives image data and instructions from the image processing device 102 and transmits output results from the LLM to the computer 101.

[0039] 3 is a diagram showing a sequence of processes performed by the information processing system 100. The symbol "S" in the explanation of each process indicates a step in the sequence, and this also applies to subsequent flowcharts and sequence diagrams. For convenience of explanation, user operations will also be explained using steps.

[0040] Fig. 3(a) is a diagram illustrating the flow of registering a one-touch button corresponding to the scan expansion function. This process starts when the one-touch button registration mode is started by the user pressing a button (not shown), for example. Note that a method for performing settings to expand the scan function of the image processing device 102 using the computer 101 in Fig. 3(a) will be described later with reference to Fig. 4. Also, the operation input on the setting screen related to enabling the one-touch button in S315 to S317 will be described later with reference to Fig. 10.

[0041] In S301, a user of the information processing system 100 places an original such as paper on the original conveying device 237 of the image processing apparatus 102 and presses a scan execution button using the input device 239 to instruct scanning of the original.

[0042] In S302, the CPU 231 of the image processing device 102 controls the scanner device 236 to read the document placed on the document conveying device 237 by the user in S301, and acquires image data of a document image based on the document. The CPU 231 performs image processing such as OCR (Optical Character Recognition) processing and handwriting detection on the acquired image data. OCR is an optical character recognition function that extracts text data from portions that can be recognized as text and creates text-searchable PDF / XPS / OOXML.

[0043] In S303, the CPU 231 of the image processing device 102 controls the external interface 241 to send image data of the document image acquired in S302 to the computer 101. The computer 101 receives this data and stores it in the storage 205 as data that can be used by the user in subsequent steps.

[0044] In S304, when the computer 101 receives image data of the document image, the CPU 201 displays a screen (not shown) on the display device 207 for the user to input user information. The user inputs user information for using the AI ​​service provided by the generation AI server 103 into the screen displayed on the display device 207. Here, the user information is used to control access to the log that records input and output data when using the AI ​​service provided by the generation AI server 103, and to control charges associated with the user's usage of the generation AI server 103.

[0045] In S305, the CPU 201 of the computer 101 controls the external interface 208 to send the user information entered by the user in S304 to the generation AI server 103. The generation AI server 103 receives this, and the CPU 261 of the generation AI server 103 performs user authentication to use the AI ​​service provided by the generation AI server 103 based on the user information received from the computer 101.

[0046] In S306, the CPU 261 of the generation AI server 103 controls the external interface 269 to send the results of the user authentication performed in S305 to the computer 101. The following steps will be explained assuming that the user authentication performed by the generation AI server 103 has been completed.

[0047] In S307, when the computer 101 receives the result of user authentication from the generation AI server 103, the CPU 201 displays a screen (not shown) on the display device 207 for the user to input an instruction (prompt). The user inputs an instruction (prompt) to the screen displayed on the display device 207 to specify the content to be output using the AI ​​service for the image data acquired by the image processing device 102 in S302. In other words, the CPU 201 receives an instruction from the user.

[0048] In S308, the CPU 201 of the computer 101 controls the external interface 208 to transmit the image data received from the image processing device 102 in S303 and the instruction (prompt) input by the user in S307 to the generation AI server 103. The generation AI server 103 receives this. The instruction (prompt) is a character string that specifies the conditions for the processing to be executed by the generation AI server 103.

[0049] In S309, the CPU 261 of the generation AI server 103 controls each unit to execute processing on the image data in accordance with the prompt received from the computer 101 in S308.

[0050] In S310, the CPU 261 of the generation AI server 103 controls the external interface 269 to transmit the results (output results) of the processing performed in S309 to the computer 101. The computer 101 receives this.

[0051] In S311, the CPU 201 of the computer 101 displays the processing results received from the generation AI server 103 in S310 on the display device 207. The user checks the output results displayed on the display device 207 and confirms that the desired output result has been obtained for the content of the instructions entered in S307. If the user's desired output result has not been obtained, the user changes the content of the instruction (prompt) entered in S307 as appropriate, and repeats S307 to S311 until the desired output result is obtained.

[0052] In S312, the user instructs the computer 101 to reflect the changes to the one-touch buttons (described later with reference to FIG. 4).

[0053] In S313, in accordance with the instruction from the user in S312, the CPU 201 of the computer 101 controls the external interface 208 to transmit the instruction (prompt) input by the user in S307 to the image processing device 102. The image processing device 102 receives this, and the CPU 231 stores it in the storage 238 as a template for an instruction (prompt) to be used as a one-touch button. In other words, the CPU 231 registers the instruction (prompt) input by the user in S307 into the instruction display frame 1080 for the generated AI as a template for the instruction.

[0054] In S314, the user inputs user information for using the AI ​​service provided by the generation AI server 103 from the image processing device 102, and the CPU 231 of the image processing device 102 accepts this. Here, by using common user information for the user authentication in S304 and the user authentication in S314, the AI ​​service provided by the generation AI server 103 can be used in common from the computer 101 and the image processing device 102. As a result, even if, for example, an instruction text (prompt) and image data are sent from the image processing device 102 to the generation AI server 103, the output result can be used from the computer 101. The user information is used, for example, to control access to logs that record input and output data when using the AI ​​service provided by the generation AI server 103, and to control charges associated with the user's usage of the generation AI server 103.

[0055] In S315, the CPU 231 of the image processing device 102 controls the external interface 241 to transmit the user information input by the user in S314 to the generation AI server 103. The generation AI server 103 receives this, and the CPU 261 of the generation AI server 103 performs user authentication based on the user information received from the image processing device 102 to enable the image processing device 102 to use the AI ​​service provided by the generation AI server 103.

[0056] In S316, the CPU 261 of the generation AI server 103 controls the external interface 269 to transmit the result of the user authentication performed in S315 to the image processing device 102.

[0057] In S317, the user sets customizable parameters in the instruction sentence used with the one-touch button for using the AI ​​service provided by the generation AI server 103 from the image processing device 102. Specifically, the user sets parameters 1061 to 1064 shown as an example in Figure 10(b) in the instruction sentence (prompt) input in S307.

[0058] In S318, the user activates a one-touch button to use the AI ​​service provided by the generation AI server 103 from the image processing device 102. Specifically, the user selects "enable" in the activation setting 1021 corresponding to the scan extension function 1010 illustrated in FIG. 10(a).

[0059] In steps S314 to S318, the setting information enabled as a one-touch button after user authentication is stored in the storage 238 of the image processing device 102 as information associated with the user. Specifically, the user information using the AI ​​service is associated with default setting parameters 1061 to 1064 (application format, number of pages, page layout, language) used when using the scan expansion function 1010 with the one-touch button, and stored. In other words, the one-touch button corresponding to the scan expansion function is registered.

[0060] Figure 3(b) is a diagram explaining the flow of executing a scan using the one-touch button registered in Figure 3(a) and processing the acquired image data. Note that a method of processing image data from the image processing device 102 using the AI ​​service provided by the generation AI server 103 will be explained later using Figure 5. Also, an explanation of the operation input on the input screen related to input / output to the generation AI server 103 will be explained later using Figure 11.

[0061] In S331, the user of the information processing system 100 selects the one-touch button registered in S318, and the CPU 231 accepts this. Note that multiple one-touch buttons may be registered at this point. If multiple one-touch buttons are displayed on the display device 207, the user selects the one-touch button corresponding to the desired process. The one-touch button selected by the user simultaneously instructs the document to be scanned and to use the AI ​​service provided by the generation AI server 103. In other words, when the user selects the one-touch button, the instructions from S332 onward are automatically executed without accepting any further user instructions, and the user can obtain an output produced by the AI ​​service. Specifically, when the user presses the one-touch button, the image processing device 102 scans the document, processes the image data obtained by the scan, and transmits the image data and instructions to the generation AI server 103. To scan a document, similarly to S301, a document such as paper is placed on the document conveying device 237 of the image processing device 102, and a command to scan the document is issued by pressing a one-touch button using the input device 239. Note that S331 will be described in detail later with reference to FIG. 11.

[0062] In S332, similarly to S302, the CPU 231 of the image processing device 102 controls the scanner device 236 to read the original document and acquire image data of the document image based on the original document. The CPU 231 then performs image processing such as OCR and handwriting detection on the acquired image data of the document image.

[0063] In S333, the CPU 231 of the image processing device 102 controls the external interface 241 to transmit the image data of the document image acquired in S332 and the instruction text (prompt) corresponding to the one-touch button selected in S331 to the generation AI server 103. The generation AI server 103 receives this.

[0064] In S334, similarly to S309, the CPU 261 of the generation AI server 103 controls each unit to execute processing on the image data in accordance with the prompt received from the image processing device 102 in S333.

[0065] In S335, the CPU 261 of the generation AI server 103 controls the external interface 269 to transmit the results of the processing performed in S334 to the computer 101. The computer 101 receives them.

[0066] In S336, the CPU 201 of the computer 101 displays the processing results (output results) received from the generation AI server 103 in S335 on the display device 207. The user checks the output results displayed on the display device 207 and confirms that the desired output results have been obtained for the content selected with the one-touch button in S331.

[0067] The reason why the same user information is used to log in to the AI ​​service from the image processing device 102 in addition to the computer 101 is so that the AI ​​service can be used in common from both the computer 101 and the image processing device 102. Specifically, the results of processing by the generation AI server 103 based on the instruction text and image data sent from the image processing device 102 can be confirmed from the computer 101.

[0068] In the present invention, the CPU 261 of the generation AI server 103 performs user authentication (login) from the computer 101 and image processing device 102, and stores these in storage 266 in association with user information. As a result, the generation AI server 103 transmits the results of processing performed based on the image data and instruction text received from the image processing device 102 to the computer 101 based on the information stored in storage 266. In other words, the generation AI server 103 transmits the results of processing performed by the generation AI server 103 to the computer 101 corresponding to the image processing device 102 that transmitted the image data and instruction text.

[0069] Fig. 4 is a flowchart illustrating the flow of registering a one-touch button corresponding to the scan expansion function described in Fig. 3(a). The series of processes shown in the flowchart in Fig. 4 are performed by the CPU 201 of the computer 101 and the CPU 231 of the image processing device 102 loading program code stored in each ROM or storage into RAM and executing the program. This process is also started when the one-touch button registration mode is started by the user pressing a button (not shown), for example.

[0070] In step S401, the CPU 201 controls the external interface 208 to receive, from the image processing apparatus 102, image data of a document image acquired by the image processing apparatus 102 reading an original such as paper.

[0071] In S402, the CPU 201 receives user information for using the AI ​​service provided by the generation AI server 103 from the computer 101 via the input device 206, as input by the user (S304).

[0072] In S403, the CPU 201 causes the generation AI server 103 to perform user authentication to enable the computer 101 to use the generation AI server 103, as described in S305 and S306. Specifically, the CPU 201 controls the external interface 208 to send the user information received in S402 to the generation AI server 103.

[0073] In S404, the user inputs a prompt via the input device 206 to specify the processing to be performed on the image data received in S401 using the generation AI server 103, and the CPU 201 accepts this. Specific examples of prompts include instructions to translate or summarize text obtained by OCR of character strings contained in a document image. Another example is an instruction to extract handwritten pixels from a document image and correct the text obtained by OCR of character strings contained in the document image based on handwritten correction instructions.

[0074] In S405, the CPU 201 controls the external interface 208 to receive the processing result (output result) from the generation AI server 103 in response to the instruction (prompt) entered by the user in S402. For example, a translated or summarized text is obtained from the text obtained by OCR of the character strings contained in the document image. Alternatively, a corrected text is obtained from the text obtained by OCR of the character strings contained in the document image based on the correction instructions handwritten on the document image.

[0075] In S406, the CPU 201 receives a user input via the input device 206, thereby determining whether the instruction (prompt) input in S404 is appropriate. Specifically, the CPU 201 displays the output result received in S405 on the display device 207. The user, upon viewing this, determines whether the output result is appropriate (whether it is as expected), and inputs whether the obtained output result is appropriate via the input device 206, which the CPU 201 receives. If the CPU 201 receives information indicating that the output result is as expected by the user, it determines that the instruction input in S404 is appropriate, and if the CPU 201 receives information indicating that the output result is not as expected by the user, it determines that the instruction input in S404 is inappropriate. In other words, if the result is not as expected by the user in response to the instruction (NO in S406), S402 to S403 are repeated while changing the content of the instruction (prompt) input in S404, and if the result is as expected by the user (YES in S406), the process proceeds to S407.

[0076] In S407, the CPU 201 registers a template of a prompt in accordance with a user instruction. Specifically, the CPU 201 displays a setting screen 1050 for detailed settings of the scan extension function on the display device 207, and accepts a prompt input by the user into a prompt display frame 1080 for instructions to the generation AI. The CPU 201 registers a fixed phrase that can be fixed in the prompt as a template, and stores it in the storage 205. At this time, the CPU 201 may register the entire prompt as a template. For example, as a prompt for realizing the translation document creation 1011 in FIG. 10(a), "Please translate into the language [], in the application format [], with the page layout [], so that it fits within the number of pages []" is registered as a template ([]: parameter).

[0077] In S408, the CPU 231 of the image processing device 102 authenticates the user of the generation AI server 103, as described in S314 and S315. Specifically, the CPU 231 accepts user information for using the AI ​​service provided by the generation AI server 103 from the image processing device 102 via the input device 239, and controls the external interface 241 to transmit the information to the generation AI server 103.

[0078] In S409, the CPU 201 changes some of the keywords in the instruction (prompt) in accordance with a user instruction and sets parameters 1061 to 1064. Specifically, for example, an application format parameter 1061, a page count parameter 1062, a page layout parameter 1063, and a language parameter 1064 are set as parameters to be used in the instruction to realize the "translation document creation" exemplified in S405. In order to register keywords to be used in part of the instruction (prompt), the user sets parameters 1061 to 1064 to be used as defaults on the setting screen 1050 for setting detailed settings of the scan extension function, and the CPU 231 accepts these. Furthermore, the CPU 231 accepts options (ranges) of values ​​selectable for the parameters 1061 to 1064, which are set in advance as keywords to be used in part of the instruction (prompt), as exemplified by parameters 1071 to 1074.

[0079] In S410, the CPU 201 creates a one-touch button that corresponds to the instruction template set in S407 to S409 and the range of customizable parameters (including default values). The CPU 201 also arranges the button in a form that is selectable by the user on the scan extension function menu screen 1110. The information set in S407 to S409 is stored in the storage 238 in association with the one-touch button, and the one-touch button is created. Note that the instruction text or parameters registered in the one-touch button may all be fixed. In this case, the same instruction text is sent each time the one-touch button is selected.

[0080] Note that when the enable setting 1021 is set to "enabled," the scan extension function 1010 is enabled, the one-touch buttons become functional, and when a button is selected, the process corresponding to the button is executed. Also, the information associated with the one-touch button may be changeable from the advanced setting 1022.

[0081] The one-touch button may be registered by accessing the image processing device 102 from the computer 101 via the network 104 and inputting information on a remote UI screen, so that the settings are reflected in the image processing device 102.

[0082] Note that the timing of the processing of S408 is not limited to this, and the order of the processing may be interchanged with other processing. The reason why the same user information is used to log in to the AI ​​service from the image processing device 102 in addition to the computer 101 is to allow the AI ​​service to be used in common from the computer 101 and the image processing device 102. Specifically, the results of processing by the generation AI server 103 based on the instruction text and image data sent from the image processing device 102 can be confirmed from the computer 101.

[0083] In the present invention, the CPU 261 of the generation AI server 103 performs user authentication (login) from the computer 101 and image processing device 102, and stores these in storage 266 in association with user information. As a result, the generation AI server 103 transmits the results of processing performed based on the image data and instruction text received from the image processing device 102 to the computer 101 based on the information stored in storage 266. In other words, the generation AI server 103 transmits the results of processing performed by the generation AI server 103 to the computer 101 corresponding to the image processing device 102 that transmitted the image data and instruction text.

[0084] Note that the image processing device 102 may be accessed from the computer 101, and the computer 101 may indirectly execute this processing in response to a user instruction to the computer 101.

[0085] Fig. 5 is a flowchart showing an example of the process for executing the process on image data described in Fig. 3(b). The series of processes shown in the flowchart in Fig. 5 will be described assuming that the CPU 231 of the image processing device 102 loads program code stored in the ROM 232 or storage 238 into the RAM 234 and executes it. This process is started when a registered one-touch button is selected by the user.

[0086] In S501, the CPU 231 retrieves from the storage 205 a template of an instruction corresponding to the one-touch button registered in S407. Specifically, for example, when the user selects the translation document creation one-touch button 1121 in Fig. 11(b), the instruction corresponding to this button, shown in the instruction to generation AI display frame 1080 in Fig. 10(b), is retrieved. Note that when the summary document creation 1012 or handwritten correction reflection 1013 one-touch button is selected by the user, the instruction saved corresponding to the button is displayed in the instruction to generation AI display frame 1080, and the CPU 231 retrieves it in this step.

[0087] It should be noted that what is acquired in S501 may not be a template of an instruction statement corresponding to a one-touch button, but may be default values ​​of each parameter constituting the instruction statement, in which case S502 is not necessary.

[0088] In S502, the CPU 231 acquires parameters used in the instruction of the instruction template set in S407. For example, when the translation document creation one-touch button 1121 in Fig. 11(b) is selected by the user, the CPU 231 acquires default values ​​of the parameters 1071 to 1074 used in the instruction associated with this button.

[0089] In S503, the CPU 231 scans the paper document 1101 in FIG. 11(a) using the scanner device 236, and acquires image data 1102 of the document image.

[0090] In S504, the CPU 231 executes scan image processing on the image data 1102 acquired in S503. Specifically, for example, by executing OCR on the image data 1102, the language and number of characters of text included in the document image based on the image data 1102 are acquired, and handwritten pixel areas included in the document image based on the image data 1102 are detected.

[0091] In S505, the CPU 231 acquires the instruction statement template acquired in S501 and the parameters used in the instruction statement acquired in S502, and generates an instruction statement 1103 for instructing the processing to be performed on the image data acquired in S503. The instruction statement 1103 is generated as shown in an instruction statement display frame 1080 for the generation AI in Fig. 10(b), and parameters 1071 to 1074 are reflected in the instruction statement 1103.

[0092] In S506, the CPU 231 controls the external interface 241 to transmit the image data 1102 acquired in S503 and the instruction statement 1103 generated in S505 to the generation AI server 103 that provides the AI ​​service. The output of the generation AI server 103 in response to the transmitted image data 1102 and instruction statement 1103 is transmitted from the generation AI server 103 to the computer 101, and then transmitted from the computer 101 to the image processing device 102. Specifically, as shown in FIG. 11(c), for example, the image data 1102 (attached image 1162) and the instruction statement 1103 (instruction statement 1161) are input as a pair to perform instruction input 1160, thereby obtaining the output result 1171 of the AI ​​service. Note that the image processing device 102 may be configured to receive the output of the generation AI server 103 directly from the generation AI server 103, or the image processing device 102 may be configured not to receive the output of the generation AI server 103.

[0093] 10 is a diagram showing an example of a setting screen for setting the scan extension function. A setting screen 1000 for the scan extension function and a setting screen 1050 for detailed settings of the scan extension function are examples of setting acceptance screens displayed on the display device 240 of the image processing device 102 by the CPU 231. Note that these screens may also be displayed on the display device 207 of the computer 101 by the CPU 201.

[0094] 10A is a diagram showing an example of a screen for setting the extended scan function. On the extended scan function setting screen 1000, a function to be made available as a one-touch button is set to "enabled" by an operation input 1030 showing an example of a user's selection operation, thereby setting the one-touch button for the extended scan function to a state in which it can be used from the image processing device 102. Conversely, setting the function to "disabled" sets the one-touch button for the extended scan function to a state in which it cannot be used from the image processing device 102.

[0095] Specifically, for example, as the scan extension function 1010, an activation setting 1021 for setting "enabled" or "disabled" as settings 1020 for translation document creation 1011, summary document creation 1012, and handwritten correction reflection 1013, and a detailed activation setting 1022 are set. When the detailed setting 1022 is selected, the screen transitions to a setting screen 1050 for the detailed settings of the scan extension function.

[0096] The scan extension function setting screen 1000 has, for example, a back button 1040 and an OK button 1041. When the back button 1040 is pressed, the screen returns to the previous screen without reflecting any changes made on the scan extension function setting screen 1000. When the OK button 1041 is pressed, the settings made at that time are saved.

[0097] When the enable setting 1021 is set to "enabled," a one-touch button corresponding to the scan extension function 1010 set to "enabled" is displayed on the scan extension function menu screen 1110. On the other hand, when the enable setting 1021 is set to "disabled," a one-touch button corresponding to the scan extension function 1010 set to "disabled" is not displayed on the scan extension function menu screen 1110. Even when set to "disabled," the corresponding one-touch button may be displayed on the scan extension function menu screen 1110, or may be displayed in a different color or shape from the one-touch button corresponding to the function set to "enabled." Furthermore, even when displayed in the same manner as the one-touch button corresponding to the function set to "enabled," the one-touch button may be configured to warn the user with an error message or sound upon selection by the user, or may not respond even when selected.

[0098] Fig. 10(b) is a diagram showing an example of a screen for inputting detailed settings of the scan extension function. The setting screen 1050 for detailed settings of the scan extension function shown in Fig. 10(b) is displayed by screen transition in response to an operation input 1030 for detailed settings 1022 of translation document creation 1011 in Fig. 10(a). That is, the setting screen 1050 for detailed settings of the scan function is displayed on the display device 240 when the user selects detailed settings 1022 on the setting screen 1000 of the scan extension function. Note that Fig. 10(b) is an example of a screen displayed when detailed settings 1022 corresponding to translation document creation 1011 is selected; when detailed settings 1022 corresponding to summary document creation 1012 or handwritten correction reflection 1013 are selected, a setting screen corresponding to each function is displayed.

[0099] The setting screen 1050 for detailed settings of the scan function has an instruction display frame 1080 for the generated AI, and also allows default values ​​1071 to 1074 to be set for parameters 1016 to 1064. By setting the default values ​​1071 to 1074 corresponding to the parameters 1061 to 1064, the user can pre-register standard settings for when the generated AI server 103 performs processing using a one-touch button.

[0100] The application format parameter 1061 is a parameter for specifying the application and extension for output. The example "MS Word" specifies that output will be made using the Word software (.word extension) provided by Microsoft. Other options include "Text" which specifies the ".txt" extension, and "Pages" (software provided by Apple).

[0101] The page count parameter 1062 is a parameter for specifying the number of pages at the time of output. The example "8 pages" specifies that the output will fit within 8 pages. Note that the number of pages may be an item that specifies not only a maximum value but also a minimum value.

[0102] The page layout parameter 1063 is a parameter for specifying the page layout of the output. In addition to the example of "two columns", it is possible to specify a layout that can be realized in the specified application, such as "three columns" or "narrow the first column".

[0103] The language parameter 1064 is a parameter that specifies the language of the output. In addition to the example of "Japanese", various languages ​​that correspond to the processing of the generation AI server 103 can be selected. Also, "automatic" or the like may be prepared, and the language setting of the computer 101 or image processing device 102 may be acquired and the language that matches this may be specified.

[0104] Note that these parameters that the user can set are not limited to these. Furthermore, these parameters that the user can set vary depending on the scan extension function 1010. For example, in the case of summary document creation 1012, the number of characters may be specified, and in the case of handwritten correction reflection 1013, the color of the corrected portion may be specified. In the setting screen 1050 for detailed settings of the scan function, the user can set all kinds of parameters. Furthermore, the parameters that the user can set can be customized by the user.

[0105] The instruction display frame 1080 for the generation AI can be modified or changed by the user's input instruction 1031. Specifically, when the user inputs instruction 1031, the text displayed in the instruction display frame 1080 for the generation AI becomes editable, and the user can edit the text using the input device 239 or the like. For example, the user can edit, delete, or add an instruction to realize translation document creation 1011, such as "Please translate into the language [], with the page layout [], in the application format [], so that it fits within the number of pages []."

[0106] The setting screen 1050 for setting the advanced settings of the scan extension function has, for example, a back button 1090 and an OK button 1091. When the back button 1090 is pressed, the screen returns to the previous screen without reflecting any changes made on the setting screen 1050 for setting the advanced settings of the scan extension function. When the OK button 1091 is pressed, the settings made at that time are saved.

[0107] FIG. 11 is a diagram showing an example of a data flow and a screen for executing processing on image data.

[0108] Figure 11(a) is a diagram showing the data flow for processing image data. The CPU 231 of the image processing device 102 of the present invention scans a paper document 1101 placed by a user on the platen of the document transport device 237, and transmits the generated (acquired) image data 1102 and instruction statement 1103 to the generation AI server 103. The output results from the generation AI server 103 can be received by the computer 101. Here, as shown in S305, S306, S314, and S315 of Figure 3, since the user has been authenticated to use the AI ​​service provided by the generation AI server 103 from each device, the user's data can be referenced regardless of the device being operated. Specifically, the user can reference the output of the generation AI server 103 by operating either the computer 101 or the image processing device 102.

[0109] 11(b) is a diagram showing an example of an operation screen displayed on the display device 240 of the image processing device 102. The user can instruct the execution of a process corresponding to each one-touch button by selecting a one-touch button on a scan extension function menu screen 1110 displayed on the touch panel screen of the image processing device 102. The CPU 231 accepts an instruction input 1131 from the user via the input device 239 (touch panel).

[0110] The extended scan function menu screen 1110 displays a one-touch button for creating a translated document 1121, a one-touch button for creating a summary document 1122, and a one-touch button for reflecting handwritten corrections 1123, which correspond to the extended scan functions 1011 to 1013 registered by the user. The user can input an execution command by selecting a one-touch button displayed on the extended scan function menu screen 1110. If a one-touch button is not registered, the one-touch button for that extended scan function is not displayed on the extended scan function menu screen 1110. Furthermore, one-touch buttons corresponding to extended scan functions set to "disabled" on the extended scan function setting screen 1000 are also not displayed on the extended scan function menu screen 1110. It is not essential that the one-touch button is not displayed when it is not registered or when the extended scan function is set to "disabled." The color or shape of the one-touch button may be changed to allow the user to recognize it. Even if the one-touch button is displayed in the same way as a one-touch button corresponding to a function set to "enabled," the button may be configured to warn the user with an error message or sound upon selection, or may not respond even when selected.

[0111] A user using the generation AI server 103 from the image processing device 102 uses the services provided by the generation AI server 103 as a user who has been authenticated in S314 and S315. Here, the user information 1111 of the user using the generation AI server 103 may be linked to the user information of the logged-in user of the image processing device 102 and stored and managed in the storage 238. The CPU 231 of the image processing device 102 detects that the logout button 1112 has been pressed, and terminates reception of the instruction input 1131 from the logged-in user.

[0112] The AI ​​service screen 1150 in Figure 11(c) is a diagram showing an example of a screen displayed on the display device 207 when the AI ​​service provided by the generation AI server 103 is used from the computer 101. Note that the AI ​​service screen 1150 may be displayed on the display device 240 of the image processing device 102 by the CPU 231. The AI ​​service screen 1150 provided by the generation AI server 103 displays a response (output result 1170) from the generation AI in response to a user instruction input 1160. The user instruction input 1160 is a prompt generated by the CPU 231 of the image processing device 102 in S505 and transmitted to the generation AI server 103 in S506.

[0113] The AI ​​service screen 1150 in Figure 11(c) shows how a user's instruction input, instruction statement 1161 (instruction statement 1103), is automatically entered, and image data 1102 is automatically attached to instruction statement 1103 (attached image 1162) and sent.

[0114] Note that a user using the AI ​​service provided by the generation AI server 103 from the computer 101 uses the AI ​​service as a user who has been authenticated in S304 to S306. Here, the user information 1151 of the user using the generation AI server 103 may be linked to the user information of the logged-in user of the computer 101 and stored and managed in the storage 205. The CPU 201 of the computer 101 detects that the logout button 1112 has been pressed, and thereby terminates reception of the instruction input 1131 by the logged-in user of the user information 1151.

[0115] While FIG. 11(b) shows an example of an operation screen displaying a list of one-touch buttons for registered scan extension functions, a different screen configuration may be used as long as the CPU 231 is configured to accept (receive) operation inputs consistent with the spirit of the present invention. For example, an operation screen that extends the SEND function for sending a conventional scanned image by email may be configured to accept operation inputs from a user by sending the image to a user-authenticated generation AI server 103. Alternatively, the one-touch buttons may be displayed on a menu screen or setting screen for a copy process performed by the image processing device 102, or may be physical buttons whose functions can be customized on the image processing device 102. Furthermore, the one-touch buttons may be displayed superimposed on the screen by a specific operation, or may be invoked by voice input.

[0116] As described above, according to the present invention, by sending image data obtained by scanning a document and text (instructions) based on the user's instructions to an AI service, the AI ​​service outputs the results of processing in accordance with the user's intentions, and the user can receive these results.

[0117] Generative AI (AI services) is a general term for services that use AI. Examples include text generation AI services such as ChatGPT (artificial intelligence chatbot) provided by OpenAI and Copilot provided by Microsoft. Other examples include image generation AI services such as Stable Diffsion and Midjourney.

[0118] Generative AI is AI that can generate a variety of content, and its purpose is to learn patterns and relationships in data (deep learning) and generate new content.

[0119] In addition, in the present invention, when the translation document creation process and summary document creation process are performed by an AI service, the image processing device 102 may send the text obtained as a result of OCR of the image data to the generation AI server 103 instead of the image data.

[0120] <Embodiment 2> In the first embodiment, a method is described in which image data of a document image is acquired from a paper manuscript by inputting an instruction to a one-touch button registered in the image processing device 102, and an instruction sentence is generated based on set parameters and transmitted to the generation AI server 103. In the second embodiment, a method is described in which an error occurring in the generation AI server 103 is detected and notified based on the results of image processing performed by the image processing device 102 before transmission to the generation AI server 103.

[0121] Fig. 6 is a diagram showing a sequence of the information processing system 100 in embodiment 2. Of the steps in the sequence shown in Fig. 6, those steps common to Fig. 3(b) are the same as those in embodiment 1, and therefore description thereof will be omitted.

[0122] S601 to S602 are the same as S331 to S332, and therefore the explanation will be omitted.

[0123] In S603, the CPU 231 of the image processing device 102 detects whether the instruction corresponding to the one-touch button selected in S601 and the image data of the document image acquired in S602 meet the prerequisites for using the AI ​​service. In other words, it checks whether an error is likely to occur when the AI ​​service processes the document. For example, when the translation document creation one-touch button 1121 is selected, the CPU 231 compares the language of the text contained in the document image in the OCR results with the translation target language set in the instruction. If they are the same language, it detects that an error preventing translation is likely to occur. When the summary document creation one-touch button 1122 is selected, the CPU 231 compares the number of characters in the text contained in the document image with the number of characters after summary set in the instruction. If the former is smaller than the latter, it detects that an error preventing summary execution will occur. When the handwritten correction reflection one-touch button 1123 is selected, if the pixels handwritten in the document image cannot be extracted as a result of image processing, it detects that an error preventing handwritten correction reflection will occur.

[0124] In S604, the CPU 231 of the image processing device 102 notifies the user of the error detected in S603. That is, the CPU 231 notifies the user that an error may occur when inputting the generated AI in the functions of translation document creation, summary document creation, and handwritten correction reflection shown in S603. Specifically, the CPU 231 displays on the display device 240 a message informing the user that an error may occur when inputting the generated AI.

[0125] In S605, the user changes the settings to correct part of the instruction text corresponding to the one-touch button, taking into consideration the error content notified in S604, and the CPU 231 accepts this. Specifically, for example, the user changes the translation language in the translation document creation function. Also, in the summary document creation function, the user changes the information on the number of characters corresponding to the number of pages after summary. Also, in the handwriting correction reflection function, the user reconfirms whether the document image acquired in S602 includes handwriting, and changes the document image to include handwriting.

[0126] S606 to S609 are the same as S333 to S336, and therefore the explanation will be omitted.

[0127] Fig. 8 is a flowchart illustrating the flow of error notification of an instruction statement in the second embodiment. Of the steps in the flowchart shown in Fig. 8, those common to those in Fig. 5 are the same as those in the first embodiment, and therefore will not be described again. The series of processes shown in the flowchart in Fig. 8 are performed by the CPU 231 of the image processing device 102 loading program code stored in the ROM 232 or storage 238 into the RAM 234 and executing the program. This process is started when a registered one-touch button is selected by the user.

[0128] S801 to S804 are the same as S501 to S504, and therefore the explanation will be omitted.

[0129] In S805, the CPU 231 detects whether the instruction corresponding to the one-touch button selected in S601 and the image data of the document image acquired in S602 meet the prerequisites for using the AI ​​service. In other words, it determines whether an error is likely to occur when the AI ​​service processes the document. For example, when the translation document creation one-touch button 1121 is selected, the CPU 231 compares the language of the text contained in the document image in the OCR results with the translation target language set in the instruction. If they are the same language, it detects that an error preventing translation is likely to occur. When the summary document creation one-touch button 1122 is selected, the CPU 231 compares the number of characters in the text contained in the document image with the number of characters after the summary set in the instruction. If the former is smaller than the latter, it detects that an error preventing summary execution will occur. When the handwritten corrections reflection one-touch button 1123 is selected, if the pixels handwritten in the document image cannot be extracted as a result of image processing, it detects that an error preventing handwritten corrections from being reflected will occur.

[0130] If the CPU 231 determines that an error is likely to occur when using the generated AI (YES in S805), the process proceeds to S806, and if the CPU 231 does not determine that an error is likely to occur when using the generated AI (NO in S805), the process ends. In other words, if the CPU 231 determines that an error is likely to occur when using the generated AI, it decides to notify the error and proceeds to S806.

[0131] In S806, the CPU 231 notifies the user of the error detected in S603. That is, the CPU 231 notifies the user that an error may occur when inputting to the generated AI in the functions of translation document creation, summary document creation, and handwritten correction reflection shown in S603. Specifically, the CPU 231 displays on the display device 240 a message informing the user that an error may occur when inputting to the generated AI.

[0132] As described above, according to this embodiment, before the image processing device 102 sends image data and instructions to the generation AI server 103, errors that occur in the generation AI server 103 can be detected and notified based on the results of image processing performed by the image processing device 102.

[0133] <Embodiment 3> In the third embodiment, a method is described for controlling or recommending parameters for generating an instruction statement to be sent to the generation AI server 103 based on the results of image processing performed by the image processing device 102 before the image data and instruction statement are sent by the image processing device 102. Note that "control" here means that the CPU 231 of the image processing device 102 changes the parameters, or limits the parameters that the user can select. Also, "recommend" means that a display is displayed urging the user to change the parameters, or that the parameter is displayed at the top of the parameter candidate list.

[0134] Fig. 7 is a diagram showing a sequence of the information processing system 100 in embodiment 3. Of the steps in the sequence shown in Fig. 7, the steps common to Fig. 3(b) are the same as those in embodiment 1, and therefore description thereof will be omitted.

[0135] S701 and S702 are the same as S331 and S332, and therefore the explanation will be omitted.

[0136] In S703, the CPU 231 of the image processing device 102 acquires recommended settings for the instruction text to be used with the one-touch button selected in S701, based on the results of image processing of the image data acquired in S702. Specifically, for example, in the case of a translation document creation function, a recommended setting range for the language parameter is acquired based on the language setting configured in the image processing device 102. For example, if the main setting of the image processing device 102 is Japanese, "Japanese" is acquired. In the case of a summary document creation function, a recommended setting range for the page count parameter is acquired based on the number of characters in the text contained in the document image obtained by OCR of the image data. For example, if the text contained in the document image is 3,000 characters, "3 pages" is acquired. Furthermore, in the handwriting correction reflection function, recommended settings are acquired for the processing range of handwritten instructions, handwritten circles, etc., pixels erroneously detected as handwriting (noise), etc., based on the handwritten objects and pixel counts of the handwriting detection results.

[0137] The recommended setting range can be set in advance by the user, and the process of S703 is executed by referring to the recommended setting range stored in the storage 238. The recommended setting range may differ for each scan extension function.

[0138] In S704, the CPU 231 of the image processing device 102 controls the selectable range of parameters for the instruction sentence used by the one-touch button selected in S701, or recommends recommended settings, based on the settings acquired in S703. Specifically, for example, in the translation document creation function, the language acquired as the language parameter is controlled based on the language settings of the image processing device 102 acquired in S703. Also, in the summary document creation function, the CPU 231 controls the range of selection so that the page count parameter is equal to or less than the net page count of the character string in the OCR result. Also, in the handwriting correction reflection function, the CPU 231 controls handwritten character instructions, handwritten circled characters, etc., as either the processing range or pixels (noise) that are erroneously detected as handwriting.

[0139] The CPU 231 controls to limit the options and selection range of the instruction statements that can be set by the user based on the image processing results, but it may also display a recommendation to the user in the form of a recommended setting, and the setting itself may be selected based on the user's discretion.

[0140] In S705, the user changes the settings to modify part of the instruction text corresponding to the one-touch button, taking into account the setting range controlled in S704 or the recommended setting, and the CPU 231 accepts this. Specifically, for example, the user changes the translation target language in the translation document creation function. In the summary document creation function, the user changes the number of pages after summary to a value smaller than the net page count of the character string in the OCR result. Also, in the handwritten correction reflection function, the user changes the instruction text so that handwritten circled text can be interpreted as the processing target range. Note that the user's setting changes are not required, and if the user does not need to change the settings, they do not need to change to the controlled or recommended settings.

[0141] S706 to S709 are the same as S333 to S336, and therefore the explanation will be omitted.

[0142] Fig. 9 is a flowchart illustrating the flow of instruction statement setting control in the third embodiment. Of the steps in the flowchart shown in Fig. 9, those common to those in Fig. 5 are the same as those in the first embodiment, and therefore will not be described again. The series of processes shown in the flowchart in Fig. 9 is performed by the CPU 231 of the image processing device 102 loading program code stored in the ROM 232 or storage 238 into the RAM 234 and executing the program. This process is started when a registered one-touch button is selected by the user.

[0143] S901 to S904 are the same as S501 to S504, and therefore the explanation will be omitted.

[0144] In S905, the CPU 231 acquires recommended settings for the instruction text to be used with the one-touch button selected in S701 based on the results of image processing of the image data acquired in S702. Specifically, for example, in the case of a translation document creation function, a recommended setting range for the language parameter is acquired based on the language setting configured in the image processing device 102. For example, if the main setting of the image processing device 102 is Japanese, "Japanese" is acquired. In the case of a summary document creation function, a recommended setting range for the page count parameter is acquired based on the number of characters in the text included in the document image obtained by OCR of the image data. Furthermore, in the handwriting correction reflection function, recommended settings are acquired for the processing range of handwritten instructions, handwritten circled text, pixels erroneously detected as handwriting (noise), etc., based on the handwritten objects and pixel counts resulting from handwriting detection. If there is a mismatch with the conditions for using the generation AI (YES in S905), the process proceeds to S906. If there is no mismatch with the conditions for using the generation AI (NO in S905), the process ends.

[0145] In S906, the CPU 231 controls the range of selectable parameters for the instruction statement used by the one-touch button selected in S701 and recommends recommended settings based on the settings acquired in S905. Specifically, for example, in the translation document creation function, the language acquired as the language parameter is controlled based on the language setting of the image processing device 102 acquired in S703. In addition, in the summary document creation function, the CPU 231 controls the range of selectable pages so that the page count parameter is equal to or less than the net page count of the character string in the OCR result. In addition, in the handwriting correction reflection function, the CPU 231 controls handwritten instructions, handwritten circled characters, etc., based on the handwriting detection result as the image processing result, as either the processing range or pixels (noise) that are erroneously detected as handwriting.

[0146] The CPU 231 controls to limit the options and selection range of the instruction statements that can be set by the user based on the image processing results, but it may also display a recommendation to the user in the form of a recommended setting, and the setting itself may be selected based on the user's discretion.

[0147] As described above, according to this embodiment, before the image processing device 102 sends image data and instructions to the generation AI server 103, it is possible to control or recommend the instructions to be sent to the generation AI server 103 based on the results of the image processing performed.

[0148] <Embodiment 4> In the fourth embodiment, a method is described in which, before the image processing device 102 transmits image data and an instruction statement to the generation AI server 103, the image processing device 102 uses user information (e.g., attributes, usage history) to recommend parameters for generating an instruction statement to be transmitted to the generation AI server 103. That is, the image processing device 102 in this embodiment acquires user information.

[0149] Fig. 12 is a flowchart illustrating the flow of instruction statement setting control in the fourth embodiment. Note that, among the steps in the flowchart shown in Fig. 12, steps common to those in Fig. 9 are the same as those in the third embodiment, and therefore description thereof will be omitted. The series of processes shown in the flowchart in Fig. 12 is performed by the CPU 231 of the image processing device 102 loading program code stored in the ROM 232 or storage 238 into the RAM 234 and executing the program. Furthermore, this process is started when a registered one-touch button is selected by the user.

[0150] In S1201, the CPU 231 acquires user attributes and usage history as user information. The user attributes include, for example, information about the language the user primarily uses, and the user usage history includes information about the application software most recently used or frequently used to process image data using the scan extension function. The CPU 231 acquires these.

[0151] S1202 to S1205 are the same as S901 to S904, and therefore the explanation will be omitted.

[0152] In S1206, the CPU 231 acquires recommended settings using the user information acquired in S1201. Specifically, for example, the language of the text contained in the document image acquired by OCR of the image data is compared with the language that the user mainly uses. The application set in the application format parameter 1061 is also compared with the application acquired from the user's usage history. If there is a mismatch with the user information (YES in S1206), the process proceeds to S1207, and if there is no mismatch with the user information (NO in S1206), the process ends.

[0153] In S1207, the CPU 231 displays the recommended settings to the user according to the result of S1206. Specifically, for example, the recommended settings are displayed so that the instruction sentence instructs the user to translate into the language that the user mainly uses. Also, for example, the recommended settings are displayed so that the instruction sentence instructs the user to convert into an application obtained from the user's usage history (most recently used, frequently used).

[0154] As described above, according to this embodiment, before the image processing device 102 sends image data and instructions to the generation AI server 103, it can recommend an instruction sentence to be sent to the generation AI server 103 based on the results of image processing and user information.

[0155] <Modification> In this modified example, an example will be described in which the information processing system 100 is composed of an image processing device 102 and a generation AI server 103. That is, the user scans an original with the image processing device 102, and the processing does not go through the computer 101, and the final output result by the generation AI server 103 is confirmed on the image processing device 102. The sequence for processing with this configuration will be described by focusing on the entities performing the processing, from S331 to S336 described in Figure 3.

[0156] In S331, the user of the information processing system 100 selects the one-touch button registered in S318.

[0157] In S332, the CPU 231 of the image processing apparatus 102 controls the scanner device 236 to read the original and acquire image data of a document image based on the original. The CPU 231 executes image processing such as OCR and handwriting detection on the acquired image data.

[0158] In S333, the CPU 231 of the image processing device 102 controls the external interface 241 to transmit the image data acquired in S332 and the instruction text (prompt) corresponding to the one-touch button selected in S331 to the generation AI server 103. The generation AI server 103 receives this.

[0159] In S334, the CPU 261 of the generation AI server 103 controls each unit to execute processing on the image data in accordance with the prompt received from the image processing device 102 in S333.

[0160] In S335, the CPU 261 of the generation AI server 103 controls the external interface 269 to transmit the result (output result) of the processing performed in S334 to the image processing device 102. The image processing device 102 receives this.

[0161] In S336, the CPU 231 of the image processing device 102 displays the output result received from the generation AI server 103 in S335 on the display device 240. The user checks the output result displayed on the display device 240 and confirms that the desired output result has been obtained for the content selected with the one-touch button in S331.

[0162] <Other embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0163] 100 Information Processing Systems 101 Computer 102 Image processing device 103 Generative AI Server

Claims

1. Accepting means for accepting a button selection from a user; an acquisition means for reading a document and acquiring image data in response to the selection being accepted by the acceptance means; a transmission means for transmitting to the AI ​​service an instruction statement registered by the user in association with the button, the instruction statement being used in a process performed by the AI ​​service using the image data, and the image data; 1. An image processing device comprising:

2. Further comprising a setting receiving means for receiving from the user settings for the processing to be executed by the AI ​​service, The transmission means transmits the instruction text registered in association with the button based on the setting accepted by the setting acceptance means, and the image data to the AI ​​service.

2. The image processing device according to claim 1, wherein:

3. further comprising an instruction statement receiving means for receiving the instruction statement from a user; The transmission means transmits the instruction statement that has been received by the instruction statement receiving means and registered in association with the button, and the image data, to the AI ​​service.

2. The image processing device according to claim 1, wherein:

4. further comprising an execution means for executing image processing on the image data; 2. The image processing apparatus according to claim 1, wherein the execution means executes the image processing to acquire information.

5. The image processing device according to claim 4 , further comprising means for accepting settings for processing to be executed by the AI ​​service from a user.

6. the information is the number of characters included in the image represented by the image data, The setting is a setting of the number of characters, 6. The image processing device according to claim 5, further comprising a first determination means for determining whether to notify the user based on the number of characters based on the setting received by the means and the number of characters acquired by the execution means.

7. The device further includes a first notification means, the first determining means determines to issue the notification when the number of characters based on the setting received by the means exceeds the number of characters acquired by the executing means; The first notification means notifies based on the decision.

7. The image processing device according to claim 6,

8. the information is information indicating the language of characters included in an image indicated by the image data, The setting is a language setting, 6. The image processing device according to claim 5, further comprising a second determination means for determining whether to notify the user based on the language setting received by the means and the information indicating the language acquired by the execution means.

9. Further comprising a second notification means; the second determining means determines to issue the notification when the language setting received by the means is different from the language indicated by the information indicating the language acquired by the execution means; The second notification means notifies based on the decision.

9. The image processing device according to claim 8,

10. The means restricts settings that are accepted from the user based on the information acquired by the execution means.

6. The image processing device according to claim 5,

11. a first information acquisition unit for acquiring user information of a user who logs in to the image processing device; The means limits settings to be accepted from the user based on the user information of the user acquired by the first information acquisition means.

6. The image processing device according to claim 5,

12. 12. The image processing apparatus according to claim 11, wherein the user information is a language used by the user.

13. 12. The image processing apparatus according to claim 11, wherein the user information is a history of functions used by the user.

14. The image processing device further includes a second information acquisition unit for acquiring user information of a user who logs in to the image processing device, The means recommends settings to be made by the user based on the user information of the user acquired by the second information acquisition means.

6. The image processing device according to claim 5,

15. 15. The image processing apparatus according to claim 14, wherein the user information includes a language used by the user and a function usage history.

16. A display means; A receiving means for receiving a processing result from the AI ​​service; and The display means displays the processing result received by the receiving means.

2. The image processing device according to claim 1, wherein:

17. The acquisition means reads a document and acquires image data in accordance with the acceptance of the selection by the acceptance means, and the transmission means transmits to the AI ​​service an instruction statement registered by the user in association with the button, the instruction statement being used in processing performed by the AI ​​service using the image data, and the image data.

2. The image processing device according to claim 1, wherein:

18. 2. The image processing apparatus according to claim 1, wherein the acquisition unit acquires the image data by reading a document placed on a document table while conveying the document.

19. 19. A program causing a computer to execute the steps of any one of claims 1 to 18.

20. a receiving step of receiving a button selection from a user; an acquisition step of reading a document and acquiring image data in response to the selection being accepted by the acceptance step; a transmission step of transmitting, to the AI ​​service, an instruction statement registered by the user in association with the button, the instruction statement being used in a process performed by the AI ​​service using the image data, and the image data; An image processing method comprising:

Images