Information processing apparatus, information processing method, and program

Abstract

To achieve conversion according to a user's intention by inputting a pair of an image and a text to a generation AI on the basis of a user's instruction.SOLUTION: An image processing method comprising: receiving an instruction to cause a generation AI to convert a document image in accordance with a sample file; acquiring the document image; acquiring the sample file; generating an instruction sentence corresponding to the instruction; and transmitting the document image, the sample file, and the instruction sentence to a server of the generation AI.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a technology for converting document images using generative AI. [Background technology]

[0002] In recent years, generative AI, which can automatically generate creative content such as images, text, and audio, has rapidly become popular. Accordingly, a variety of services using generative AI have been provided. For example, an image generation device is known that creates an instruction sentence (prompt) including text (prompt element) corresponding to a tag selected by a user, inputs the instruction sentence (prompt) to the generative AI, and outputs an image generated by the generative AI (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7398723 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the image generation device in Patent Document 1 could not obtain the results of conversion by the generation AI from a paper or other original document as data with a layout and design that matched the user's intentions (e.g., an application file in Office (registered trademark) format). In other words, the user had to attach a document image obtained by scanning the original document to the generation AI, and then input instructions (prompts) in text describing the layout and design into which the user wanted the document image converted, which posed a significant burden for inputting instructions.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to use a generation AI that can input multimodal data of images and text, and to perform conversion in line with the user's intentions by inputting image and text pairs into the generation AI based on the user's instructions. [Means for solving the problem]

[0006] The information processing method of the present invention is characterized by comprising an instruction receiving step for receiving an instruction to cause a generation AI to convert an original image in accordance with a sample file, an original image acquisition step for acquiring the original image, a sample acquisition step for acquiring the sample file, an instruction statement generation step for generating an instruction statement corresponding to the instruction, and an instruction sending step for sending the original image, the sample file, and the instruction statement to the server of the generation AI. [Effects of the Invention]

[0007] According to the present invention, by inputting a document image scanned from a user's manuscript, text based on the user's instructions, and a sample image specified by the user into the generation AI, the AI ​​can convert the data into data with a layout and design that matches the user's intentions and output the results. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of the configuration of an information processing system. [Figure 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of each device that configures the information processing system. [Figure 3] FIG. 1 is a sequence diagram of an information processing system. [Figure 4] 10 is a flowchart for setting a scan extension function in an information processing system. [Figure 5] 10 is a flowchart illustrating a process for converting a document image in an information processing system. [Figure 6] FIG. 10 is a diagram showing a setting screen for setting a scan extension function. [Figure 7] 10A and 10B are diagrams illustrating a data flow for converting a document image and an example of a screen. [Figure 8] FIG. 10 is a diagram showing a setting screen for application file conversion. [Figure 9] FIG. 10 is a diagram showing a setting screen for setting details of a scan extension function. [Figure 10] 10A and 10B are diagrams illustrating a data flow for converting a document image and an example of a screen according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the components described in the following embodiments are merely examples and are not intended to limit the scope of the technology of the present disclosure. For example, each part constituting the present invention can be replaced with any other component that can perform the same function. Also, any other component may be added.

[0010] <Embodiment 1> FIG. 1 is a block diagram showing an example of a network configuration of an information processing system 100 according to the present invention.

[0011] As shown in Figure 1, the information processing system 100 includes a computer 101, which is a terminal device, a scanner (reading device) 102 configured to be able to read original documents such as paper, and a generation AI server 103. For example, the computer 101 and the scanner 102 are located in an office and are connected to each other so that they can communicate with each other via an internal network 104. The internal network 104 is connected to the external Internet 105 via a router (not shown).

[0012] The generation AI server 103 is communicably connected to the computer 101 and the scanner 102 via the Internet 105 and the in-house network 104. The generation AI server 103 is a server managed by a business that provides a generation AI service. Here, the generation AI server 103 may be capable of being used in combination with plug-ins that realize additional functions developed by a business that provides a service utilizing the generation AI service. Furthermore, the in-house network 104 may be a wired connection or a wireless connection.

[0013] FIG. 2 is a diagram showing an example of the hardware configuration of the computer 101, scanner 102, and generation AI server 103 included in the information processing system 100.

[0014] Fig. 2(a) is a diagram showing the hardware configuration of a computer 101. As shown in Fig. 2(a), the computer 101 has a CPU 201, a ROM 202, a RAM 204, and a storage 205. It also has an input device 206, a display device 207, and an external interface 208. Each unit is connected to one another via a data bus 203.

[0015] The CPU 201 is a control unit that controls the overall operation of the computer 101. The CPU 201 starts up the system of the computer 101 by executing a startup program stored in the ROM 202, and realizes various functions such as displaying document images and inputting instructions to the generation AI by executing a control program stored in the storage 205.

[0016] ROM 202 is a storage unit realized by non-volatile memory, and stores a startup program that starts up computer 101. Data bus 203 is a communication unit for transmitting and receiving data between devices that make up computer 101. RAM 204 is a storage unit realized by volatile memory, and is used as a work memory when CPU 201 executes a control program.

[0017] The storage 205 is a storage unit realized by an HDD (Hard Disk Drive) or the like, and stores the above-mentioned control programs, document images, application (for example, document creation, spreadsheets, presentations, etc.) files, and the like.

[0018] The input device 206 is an operation unit realized by a mouse, keyboard, etc., and accepts operation input from a user operating the computer 101. The display device 207 is a display unit realized by a liquid crystal display, etc., and displays the setting screen of the scanner 102, the input screen of the generation AI server 103, etc. to the user.

[0019] The external interface 208 is an interface that connects the computer 101 and the network 104 , and receives document images from the scanner 102 and sends prompts to the generation AI server 103 .

[0020] 2(b) is a diagram showing the hardware configuration of the scanner 102. The scanner 102 is not particularly limited as long as it has a reading function, and for example, an MFP (Multi-Function Printer / Peripheral) or the like can be used. As shown in FIG. 2(b), the scanner 102 has a CPU 231, a ROM 232, a RAM 234, a printer device 235, a scanner device 236, a document transport device 237, and a storage 238. It also has an input device 239, a display device 240, and an external interface 241. Each unit is connected to each other via a data bus 233.

[0021] The CPU 231 is a control unit that controls the overall operation of the scanner 102. The CPU 231 starts up the system of the scanner 102 by executing a startup program stored in the ROM 232, and realizes functions of the scanner 102, such as scanning, printing, and faxing, by executing a startup program stored in the storage 238. The CPU 231 is an example of the instruction statement generating means of the present invention.

[0022] The ROM 232 is a storage unit realized by a nonvolatile memory, and stores a startup program for starting up the scanner 102. The data bus 233 is a communication unit for transmitting and receiving data between devices that make up the scanner 102. The RAM 234 is a storage unit realized by a volatile memory, and is used as a work memory when the CPU 231 executes the control program.

[0023] The printer device 235 is an image output device that prints and outputs a document image on a storage medium such as paper.

[0024] The scanner device 236 is an image input device that optically reads a storage medium such as paper on which characters, diagrams, etc. are printed. The data obtained by the scanner device 236 is acquired as a document image. The scanner device 236 is an example of the original image acquisition means and sample acquisition means of the present invention.

[0025] The document transport device 237 is realized by an ADF (Auto Document Feeder) or the like, detects documents placed on a document table, and transports the detected documents one by one to the scanner device 236. The storage 238 is a memory unit realized by an HDD (Hard Disk Drive) or the like, and stores the control programs and document images described above.

[0026] The input device 239 is an operation unit realized by a touch panel, hard keys, or the like, and receives operation input from a user who uses the scanner 102. The input device 239 is an example of an instruction receiving means of the present invention. The display device 240 is a display unit realized by a liquid crystal display or the like, and displays and outputs a setting screen of the scanner 102, etc., to the user.

[0027] The external interface 241 is an interface that connects the scanner 102 and the network 104, and transmits document images to the computer 101 and transmits document images and prompts to the generation AI server 103. The external interface 241 is an example of the instruction transmission means of the present invention.

[0028] 2(c) is a diagram showing the hardware configuration of the generation AI server 103. The generation AI server 103 has a CPU 261, a ROM 262, a RAM 264, a GPU 265, a storage 266, an input device 267, a display device 268, and an external interface 269, and each part is connected to each other via a data bus 263.

[0029] The CPU 261 is a control unit that controls the overall operation of the generation AI server 103. The CPU 261 starts up the system of the generation AI server 103 by executing a startup program stored in the ROM 262, and executes a control program stored in the storage 266. The control program uses a large language model (LLM) that can input multimodal data of at least images and text, and outputs the results of conversion according to instructions (prompts) given in text.

[0030] The ROM 262 is a storage unit realized by non-volatile memory, and stores a startup program that starts up the generation AI server 103. The data bus 263 is a communication unit for transmitting and receiving data between devices that make up the generation AI server 103.

[0031] The RAM 264 is a storage unit realized by a volatile memory and is used as a work memory when the CPU 261 executes a control program. The GPU 265 is a calculation unit configured with an image processing processor. The GPU 265 performs calculations to convert input image or text data using a large-scale language model, for example, in accordance with control commands given from the CPU 231.

[0032] The storage 266 is a storage unit realized by an HDD (Hard Disk Drive) or the like, and stores the above-mentioned control programs, large-scale language models, document images, prompts, application files in a predetermined format, and the like.

[0033] The input device 267 is an operation unit realized by a mouse, keyboard, etc., and accepts operation input to the generation AI server 103 from a user using the generation AI server 103. The display device 268 is a display unit realized by an LCD display, etc., and displays the setting screen of the generation AI server 103 to a user using the generation AI server 103.

[0034] The external interface 269 is an interface that connects the generation AI server 103 and the network 104. The external interface 269 receives document images and prompts from the scanner 102 and transmits output results from the large-scale language model to the computer 101.

[0035] 3 is a diagram showing a sequence of the information processing system 100. The symbol "S" in the explanation of each process indicates a step in the sequence, and this also applies to the subsequent flowcharts. For convenience of explanation, user operations will also be explained using steps.

[0036] Fig. 3(a) shows a flow for explaining the process of creating a one-touch button for setting the scan extension function. Note that a method for setting the computer 101 in Fig. 3(a) to extend the scan function of the scanner 102 will be described later with reference to Fig. 4. Also, a description of the operation input on the setting screen for enabling the one-touch button in S313 to S319 will be described later with reference to Fig. 6.

[0037] In S301, the user inputs user information from the computer 101 to use the service provided by the generation AI server 103. Here, the user information is used to control access to the log that records input and output data when using the service provided by the generation AI server 103, and to control charges associated with the user's usage of the generation AI server 103.

[0038] In S302, the computer 101 accesses the generation AI server 103, and performs user authentication to use the service provided by the generation AI server 103 from the computer 101.

[0039] In S303, the generation AI server 103 notifies the computer 101 that the user authentication performed in S302 has been completed.

[0040] In S304, the user of the information processing system 100 places an original such as paper on the original conveying device 237 of the scanner 102 and presses the scan execution button using the input device 239 to instruct scanning of the original.

[0041] In S305, the scanner 102 performs image processing such as OCR and handwriting detection on the document image obtained by reading the user's original document using the scanner device 236.

[0042] In S306, the document image data acquired by the scanner 102 is sent to the computer 101 and stored as data that can be used by the user in the subsequent steps.

[0043] In S307, the user specifies a sample (document image or application file in a specified format) to be used as a reference for layout and design when converting the document image using the generation AI. The specified format may be any format, such as PDF format, a word processing application format, a spreadsheet application format, or a presentation application format. The same applies to the format of the converted file.

[0044] In S308, the user inputs a prompt to instruct the generation AI server 103 on the content to be converted for the document image received in S306.

[0045] In S309, the computer 101 transmits to the generation AI server 103 a set of the document image received in S306, the sample designated in S307, and the instruction (prompt) input in S308.

[0046] In S310, the generation AI server 103 converts the document image in accordance with the instruction text while referring to the sample.

[0047] In S311, the generation AI server 103 transmits the conversion result of S310 to the computer 101.

[0048] In S312, the user displays the conversion result received by the generation AI server 103 on the computer 101 and confirms that the desired conversion result has been obtained for the content instructed in S308. If the desired conversion result has not been obtained, the user changes the content of the instruction (prompt) input in S308 and repeats S308 to S312.

[0049] In S313, after the desired conversion result is obtained in S312 in response to the prompt input in S308, the user instructs the computer 101 to reflect the result in the one-touch button of the scanner 102.

[0050] In S314, the computer 101, in accordance with the instruction from the user in S313, transmits the instruction (prompt) confirmed in S308 to the scanner 102 and sets a template of the instruction to be used as the one-touch button. Specifically, the instruction (prompt) input in S308 is set as the instruction template 680 in FIG. 6(b).

[0051] In S315, the user inputs user information from the scanner 102 to use the services provided by the generation AI server 103. Here, by using common user information for the user authentication in S301 and the user authentication in S315, the services provided by the generation AI server 103 can be used in common from the computer 101 and the scanner 102.

[0052] Specifically, for example, instructions and document images sent from the scanner 102 can be input to the generation AI server 103, and the output results can be used from the computer 101. Here, the user information is used to control access to the log that records input and output data when using the services provided by the generation AI server 103, and to control charges associated with the user's usage of the generation AI server 103.

[0053] In S316, the scanner 102 accesses the generation AI server 103, and performs user authentication to use the services provided by the generation AI server 103 from the scanner 102.

[0054] In S317, the generation AI server 103 notifies the scanner 102 that the user authentication performed in S315 has been completed.

[0055] In S318, the user sets customizable parameters in the prompts used with the one-touch button on the scanner 102 to use the services provided by the generation AI server 103. Specifically, the user sets parameters 661 to 664 shown in Figure 6(b) in the prompts entered in S308.

[0056] In S319, the user activates a one-touch button on the scanner 102 to use the service provided by the generation AI server 103. Specifically, the user selects "Enable" in the activation setting 621 corresponding to the scan extension function 610 illustrated in FIG. 6(a).

[0057] In steps S315 to S319, the setting information enabled as a one-touch button after user authentication is stored as information associated with the user in the storage 238 of the scanner 102. Specifically, the setting information stores user information for using the generation AI server 103 and parameters 661 to 664 (sample, reference portion of the sample, original, application) used as default settings when using the scan extension function 610 with a one-touch button.

[0058] Figure 3(b) shows a flow chart explaining the process of selecting the one-touch button set in Figure 3(a) to perform document image conversion when scanning. Note that the method of converting a document image using the scanner 102 in Figure 3(b) and the service provided by the generation AI server 103 will be described later with reference to Figure 5. Also, the operational inputs on the input screen related to input and output to the generation AI server 103 in S331, S336, and S337 will be described later with reference to Figure 7.

[0059] In S331, the user of the information processing system 100 selects the one-touch button set in S313 to use the service provided by the generation AI server 103. Here, the one-touch button selected by the user simultaneously instructs the scanner 102 to scan a document and to use the service provided by the generation AI server 103. To scan a document, similar to S301, the user places a document such as paper on the document feed device 237 of the scanner 102 and presses the one-touch button using the input device 239 to instruct the scanner 102 to scan the document. Note that a detailed description of S331 will be given later using FIG. 11.

[0060] In S332, the scanner 102 performs image processing such as OCR and handwriting detection on the document image acquired by reading the user's original document using the scanner device 236, similar to S302.

[0061] In S333, the user specifies a sample (document image or application file in a specified format) to be used as a reference for layout and design when converting the document image using the generation AI. The sample specified in S333 corresponds to the sample specified in S307, but any data having the layout and design desired as a sample can be specified for the document image acquired in S332.

[0062] In S334, the scanner 102 transmits the document image acquired in S332, the sample specified in S333, and an instruction (prompt) corresponding to the one-touch button selected in S331 to the generation AI server 103. Note that the set of data transmitted from the scanner 102 to the generation AI server 103 in S334 corresponds to the set of data transmitted from the computer 101 to the generation AI server 103 in S308.

[0063] In S335, the generation AI server 103 converts the document image in accordance with the instruction text while referring to the sample, similar to S310.

[0064] In S336, the generation AI server 103 transmits the conversion result of S335 to the computer 101.

[0065] In S337, the user displays the conversion result received by the generation AI server 103 on the computer 101, and confirms that the desired conversion result has been obtained for the content selected with the one-touch button in S331.

[0066] Fig. 4 is a flowchart illustrating the flow of creating a one-touch button for setting the scan extension function, as described in Fig. 3(a). The series of processes shown in the flowchart in Fig. 4 will be described assuming that the CPU 201 of the computer 101 loads program code stored in the ROM 202 or storage 205 into the RAM 204 and executes it.

[0067] In S401, the CPU 201 acquires user information for using the service provided by the generation AI server 103 from the computer 101 as the user input in S301.

[0068] In S402, the CPU 201 authenticates the user who is accessing the generation AI server 103 from the computer 101, as described in S302 and S303.

[0069] In step S403, the CPU 201 acquires a document image obtained by scanning an original such as paper with the scanner 102.

[0070] In S404, the CPU 201 acquires a sample (document image or application file in a predetermined format) specified by the user. As described in S333, the acquired sample is used as a reference for layout and design when converting the document image using the generation AI.

[0071] In S405, the CPU 201 inputs a prompt to the document image received in S401 to specify the content to be converted using the generation AI server 103. Specifically, the prompt instructs conversion to a file or the like in a predetermined format specified in the prompt, based on the text of the OCR results of the character strings included in the document image, in accordance with the layout and design of the sample data.

[0072] In S406, the CPU 201 acquires the output from the generation AI server 103 in response to the prompt entered in S402. Specifically, based on the text of the OCR results of the character string contained in the document image, the CPU 201 acquires the result of conversion into a file or the like of a predetermined format specified in the prompt, in accordance with the layout and design of the sample data.

[0073] In S407, CPU 201 determines whether the command (prompt) input in S405 is appropriate. That is, if the command does not produce the result expected by the user (NO in S407), S405 to S406 are repeated while changing the content of the command (prompt) input in S405, and if the result is as expected by the user (YES in S407), the process proceeds to S408.

[0074] In S408, the CPU 201 sets a template for a prompt in accordance with a user instruction. Specifically, as shown in the instruction (preview) 680 for the generation AI in FIG. 6(b), a fixed phrase that can be fixed in the instruction is set as a template. Specifically, for example, as a instruction for realizing the application file conversion 611 in FIG. 6(a), "Please refer to [] in [] and convert [] to the application format of []" is set as a template ([]: parameter).

[0075] In S409, the CPU 201 authenticates the user accessing the generation AI server 103 from the scanner 102, as described in S316 and S317. Here, by using common user information for the user authentication in S403 and the user authentication in S409, the services provided by the generation AI server 103 can be used in common from the computer 101 and the scanner 102. Specifically, for example, the instruction text and document image sent from the scanner 102 can be input to the generation AI server 103, and the output result can be used from the computer 101.

[0076] In S410, the CPU 201 sets one or more controllable parameters 661 to 664 by changing some keywords in the prompt in accordance with a user instruction. Specifically, for example, a sample 661, a reference portion of the sample 662, a manuscript 663, and an application 664 are set as parameters to be used in the prompt for realizing the "application file conversion" exemplified in S405.

[0077] Here, as default values ​​for the parameters 661 to 664, character strings are set in advance as keywords to be used as part of the instruction sentence (prompt), as exemplified by 671 to 674. Furthermore, as options (ranges) of values ​​selectable for the parameters 661 to 664, character strings are also set in advance as keywords to be used as part of the instruction sentence (prompt), as exemplified by 671 to 674.

[0078] In S411, the CPU 201 creates a one-touch button that corresponds to the template of the instruction set in S408 and S410 and the range of customizable parameters (including default values), and arranges it in a selectable form in the scan extension function menu.

[0079] 6A, the scan extension function of the application file conversion 611 is set to "enabled" in the settings 621, and the information set in steps S408 and S410 is associated with the one-touch button in the advanced settings 622. The one-touch button settings can be reflected in the scanner 102 by accessing the scanner 102 from the computer 101 via the network 104 and entering the information on the remote UI screen.

[0080] Fig. 5 is a flowchart illustrating the flow of converting a document image, as described in Fig. 3(b). The series of processes shown in the flowchart in Fig. 5 will be described assuming that the CPU 231 of the scanner 102 loads program code stored in the ROM 232 or storage 238 into the RAM 234 and executes it.

[0081] In S501, the CPU 231 acquires a template of an instruction statement corresponding to the one-touch button set in S411. Specifically, in accordance with a user's instruction input, for example, the CPU 231 acquires a template of the instruction statement 680 to the generation AI in Fig. 6(b) as the instruction statement associated with the one-touch button of the application file conversion 721 in Fig. 7(b).

[0082] In S502, the CPU 231 acquires character strings as parameters to be used in the instruction template set in S410. Specifically, in accordance with the user's instruction input, for example, the CPU 231 acquires default values ​​671 to 674 of parameters to be used in the instruction associated with the one-touch button of the application file conversion 721 in Fig. 7(b).

[0083] 7A using the scanner device 236 of the scanner 102 to obtain document images 702 and 703. The document image 703 is an example of a document image according to the present invention.

[0084] In S504, the CPU 231 executes scan image processing on the document images 702 and 703 acquired in S503. Specifically, for example, by executing OCR on the document images 702 and 703, text included in the document images 702 and 703 is acquired, or handwritten pixel areas included in the document images 702 and 703 are detected.

[0085] The scanned image processing executed in S504 may be configured to selectively execute only the necessary image processing in accordance with the settings acquired in S501 and S502. Specifically, for example, when the setting of the handwritten circled area 966 is acquired as the reference point 912 of the sample shown in Fig. 9(b), the scanned image processing for detecting the handwritten pixel area may be controlled to be executed.

[0086] In S505, the CPU 231 determines whether to use a portion of the scanned document image (image data) or an application file in a predetermined format that has been saved as electronic data, in accordance with the sample designation method setting 661 in FIG. 6(b). The sample designated here is used as a reference for layout and design when converting the document image using the generation AI. The document image and application file are examples of sample files of the present invention.

[0087] In S505, if the designated target of the sample 661 is a scanned document image (YES in S505), the process proceeds to S506, and if the designated target of the sample 661 is a file saved as electronic data (NO in S505), the process proceeds to S507.

[0088] In S506, the CPU 231 selects a document image to be referenced as a sample from among a portion of the scanned document images. A specific example of a method for specifying a portion of the scanned document image will be described later using example screens in FIGS. 8(a) to 8(f).

[0089] In S507, the CPU 231 selects a file (an application file in a predetermined format) to be referenced as a sample from files that have been previously stored as electronic data in the storage 238. A specific example of a method for specifying a stored file will be described later using the example screens in FIGS. 8(a) and 8(h).

[0090] In S508, CPU 231 acquires the template of the instruction statement acquired in S501 and a character string as a parameter to be used in the instruction statement acquired in S502, and generates instruction statement 703 for instructing conversion of the document image acquired in S503. Specifically, instruction statement 761 is generated by reflecting customizable parameters 671 to 674 in the template of instruction statement 680 to generation AI in Fig. 6(b) in the instruction statement associated with the one-touch button of application file conversion 721 in Fig. 7(b).

[0091] In S509, the CPU 231 transmits the document image 703 acquired in S503, the sample 702 acquired in S505, and the instruction statement 704 generated in S506 to the generation AI server 103 via the network 104 and the Internet 105. The output for the document image 703, sample 702, and instruction statement 704 transmitted to the generation AI server 103 can be received by the computer 101.

[0092] Specifically, as shown in Figure 7(c), a set of data input 763 for document image 703, data input 762 for sample 702, and data input 761 for instruction statement 704 is executed as instruction input 760 to the generation AI server 103, and the output result 771 can be received.

[0093] Note that S501 and S502 are examples of an instruction receiving process, S503 is an example of a manuscript image acquisition process, S505 to S507 are examples of a sample acquisition process, S508 is an example of an instruction statement generation process, and S509 is an example of an instruction sending process.

[0094] FIG. 6 is a diagram showing an example of a setting screen for setting the scan extension function.

[0095] 6A is a diagram showing an example of a screen for setting the scan extension function. As shown in FIG. 6A, on the scan extension function setting screen 600, by setting the function to be made available as a one-touch button to "enabled" by operation input 630, the one-touch button of the scan extension function is set to a usable state from the scanner 102. Specifically, for example, as the setting of the application file conversion 611, which is the scan extension function 610, an enable setting 621 for selecting "enabled" or "disabled" and a detailed setting 622 for the enablement are set.

[0096] The scan extension function setting screen 600 in FIG. 6(a) has, for example, a return button 640 for returning without reflecting the changes on the setting screen, and an OK button 641 for reflecting and confirming the changes on the setting screen.

[0097] Fig. 6(b) is a diagram showing an example of a screen for inputting detailed settings of the scan extension function. The setting screen 650 for detailed settings of the scan extension function shown in Fig. 6(b) is displayed by screen transition in response to an operation input 630 for the detailed settings 622 of the application file conversion 611 in Fig. 6(a).

[0098] Specifically, for example, as detailed settings for using the one-touch button of application file conversion 611, some keywords in the text of instruction statement 680 to generation AI can be defined in a changeable form as parameters 661-664. Here, by defining default values ​​and setting default values ​​671-674 corresponding to parameters 661-664, it is possible to set in advance standard settings to be used in response to the one-touch button on scanner 102. Template 680 of instruction statement to generation AI can be modified or changed by command input 631.

[0099] The advanced settings screen 650 for the scan extension function in Figure 6(b) has, for example, a return button 690 for returning without reflecting the changes on the settings screen, and an OK button 691 for reflecting and confirming the changes on the settings screen.

[0100] FIG. 7 shows a data flow for converting a document image and an example of a screen.

[0101] 7(a) is a diagram showing the data flow for converting a document image. As shown in FIG. 7(a), the scanner 102 of the present invention scans a paper document 701 held by the user, and transmits the generated (or specified) sample 702, document image 703, and instruction statement 704 to the generation AI server 103.

[0102] The output results from the generation AI server 103 can be received by the computer 101. Here, as shown in S302, S303, S316, and S317 in Figure 3, since the user using the service provided by the generation AI server 103 has been authenticated from each device, the scanner 102 and the computer 101, the user's data can be referenced regardless of the device.

[0103] Fig. 7(b) is a diagram showing an example of an operation screen 710 of the scanner 102 of Fig. 7(a). As shown in Fig. 7(b), a user instruction input 631 is acquired by selecting a one-touch button for a scan extension function 610 on a touch panel screen having the functions of the input device 239 and display device 240 of the scanner 102. Specifically, for example, as the scan extension function, a one-touch button 721 corresponding to the application file conversion 611 preset in Fig. 6 can be selected.

[0104] A user using the generation AI server 103 from the scanner 102 uses the services provided by the generation AI server 103 as a user who has been authenticated in S302 and S303. Here, the user information 711 using the generation AI server 103 may be managed in association with the user information of the logged-in user of the scanner 102. When the scanner 102 detects that the logout 712 button has been pressed, it terminates reception of the instruction input 731 from the logged-in user.

[0105] Figure 7(c) is a diagram showing an example of a screen when using a service provided by the generation AI server 103 from the display device 207 of the computer 101 in Figure 7(a). As shown in Figure 7(c), on the screen 750 of the service provided by the generation AI server 103, an output result 770 is obtained as an answer from the generation AI in response to a user instruction input 760.

[0106] Here, the sample 702 in Fig. 7(a) is automatically input as an attached image 762 in response to a user's instruction input. Also, the document image 703 in Fig. 7(a) is automatically input as an attached image 763 in response to a user's instruction input. Also, the instruction statement 704 in Fig. 7(a) is automatically input as an instruction statement 761 in response to a user's instruction input.

[0107] Note that a user using the generation AI server 103 from the computer 101 uses the services provided by the generation AI server 103 as a user who has been authenticated in S316 and S317. Here, the user information 751 using the generation AI server 103 may be managed in association with the user information of the logged-in user of the computer 101. When the computer 101 detects that the logout 752 button has been pressed, it terminates the reception of instruction input 760 from the logged-in user of the user information 751.

[0108] 7(b) shows an example of an operation screen using a one-touch button for the scan extension function, but a different screen example may be used as long as it is configured to accept operation input according to the gist of the present invention. Specifically, for example, an operation screen that extends the SEND function for sending a conventional scanned image by email may be configured to accept operation input from the user by sending the email to the user-authenticated generation AI server 103.

[0109] <Variation 1> FIG. 8 is a diagram illustrating a case where a scan flow is added that allows detailed specification of the conversion target and sample after selecting the one-touch button 721 corresponding to the application file conversion 611 described in FIG. 7(b).

[0110] Fig. 8(a) is an example of a screen 800 that is displayed after selecting one-touch button 721 corresponding to application file conversion 611 in Fig. 7(b). In Fig. 8(a), it is possible to select "Scan conversion target and sample" 801 and "Scan conversion target, specify sample file" 802. When "Scan conversion target and sample" 801 is selected, it becomes possible to specify a portion of the document image obtained by scanning an original with the scanner 102 as the sample and a portion as the conversion target.

[0111] Fig. 8(b) is an example of a screen 810 that realizes a scan flow for simultaneously scanning a sample and a document to be used as a conversion target when "Scan conversion target and sample" 801 in Fig. 8(a) is selected. For example, by specifying in advance that the sample document is set as the first page and the conversion target documents are set as the second and subsequent pages, it is possible to obtain document images 702 and 703 corresponding to the sample document and the conversion target document, respectively, with a single instruction to start scanning 811.

[0112] 8(c) and (d) are examples of screens 820 and 821 when scanning the sample and the conversion target separately in two steps, instead of the process equivalent to the single scan in FIG. 8(b). As shown in FIG. 8(c), by setting only the sample document and following the instruction input of Start Scan 821, a document image 702 corresponding to the document used as the sample can be obtained. Similarly, as shown in FIG. 8(d), by setting only the conversion target document and following the instruction input of Start Scan 831, a document image 703 corresponding to the document used as the conversion target can be obtained.

[0113] Figures 8(e) to 8(g) are examples of screens 840, 850, and 860 that realize the process equivalent to one scan in Figure 8(b) using different specification methods. First, as shown in Figure 8(e), a sample document and a document to be converted are set together, and document images corresponding to the sample or document to be used as the conversion target are acquired together in accordance with the instruction input for start scanning 841.

[0114] Next, as shown in Fig. 8(f), from among the acquired document images, a document image 702 corresponding to the sample manuscript is identified in accordance with the input instructions from each of pages 851 to 853. Similarly, as shown in Fig. 8(g), from among the acquired document images, a document image 703 corresponding to the manuscript to be converted is identified in accordance with the input instructions from each of pages 861 to 863.

[0115] Fig. 8(h) is a diagram showing an example of a sample file specification screen 860 when the scan workflow "Scan conversion target, specify sample file" 802 in Fig. 8(a) is selected. As shown in Fig. 8(h), a sample file can be specified from among files stored in advance in the storage 238 of the scanner 102.

[0116] Specifically, for example, a file 874 with a file name 873 of "Ref.docx" stored in a folder with a folder name 871 of an in-house standard template 872 can be specified. Similarly, a file 875 of "Ref.xlsx" and a file 876 of "Ref.pptx" can be prepared as standard templates used in-house, and these can be specified as samples. Note that the document image 703 corresponding to the original to be converted can also be obtained in the scan workflow of "Scan conversion target, specify sample file" 802 in the same way as in FIG. 8(d).

[0117] <Variation 2> FIG. 9 is a diagram showing examples of targets and ranges to be specified as sample reference locations in the detailed settings for application file conversion described in FIG. 6(b).

[0118] Fig. 9(a) is a diagram showing an example of a screen for inputting detailed settings of the scan extension function. The setting screen 900 for detailed settings of the scan extension function shown in Fig. 9(a) is displayed by screen transition in response to an operation input 630 for the detailed settings 622 of the application file conversion 611 in Fig. 6(a).

[0119] Specifically, for example, as detailed settings when using the one-touch button of application file conversion 611, some keywords in the text of instruction statement 940 to generation AI can be defined in a changeable form as parameters 911 to 914.

[0120] Here, by defining default values ​​and setting default values ​​921 to 924 corresponding to parameters 911 to 914, it is possible to preset standard settings to be used in response to one-touch buttons on the scanner 102. Here, the template 940 of instructions to the generation AI can be modified or changed by inputting instructions 931.

[0121] The advanced settings screen 900 for the scan extension function in Figure 9(a) has, for example, a return button 950 for returning without reflecting the changes on the settings screen, and an OK button 951 for reflecting and confirming the changes on the settings screen.

[0122] Fig. 9(b) is a diagram showing examples of the target and range to be specified, corresponding to the reference portion 912 of the sample in Fig. 9(a). As shown in Fig. 9(b), it may be possible to select whether to refer to the entire sample 961 as the reference portion of the sample, or to refer to a template 962, a layout 963, a graph 964, or a table 965 as a partial target included in the sample.

[0123] 9(b), it may be possible to select whether to refer to a handwritten area 966, which is a portion of the sample, as the reference point of the sample. Here, the handwritten area 966 can be identified by executing the scanned image processing for detecting handwritten pixels described in S504. In addition to the above example, similar to 962 to 966, it may also be possible to set, in customization 967, the components of the document image included in the sample as text expressed in a natural language that can be recognized by the generation AI.

[0124] As described above, according to the present invention, by inputting a sample image specified by the user into the generation AI in addition to the text based on the user's instructions, it is possible to output the result of converting the document image into data with a layout and design that meets the user's intentions.

[0125] <Embodiment 2> In the first embodiment, a method is described in which a sample, document image, and instruction text are generated based on a user's paper manuscript by inputting instructions from the scanner 102, and then transmitted to the generation AI server 103. In the second embodiment, a method is described in which a document image and instruction text are generated by inputting instructions from the scanner 102, and then a sample file is input during the interaction when accessing the generation AI server 103 from the computer 101.

[0126] FIG. 10 shows a data flow for converting a document image and an example of a screen.

[0127] Figure 10(a) is a diagram showing the data flow for converting a document image. As shown in Figure 10(a), the scanner 102 of embodiment 2 of the present invention scans a paper document 1001 held by the user and transmits the generated document image 1002 and instruction statement 1003 to the generation AI server 103. Thereafter, by inputting a sample 1004 from the computer 101 during an interaction when the computer 101 accesses the generation AI server 103, the output result from the generation AI server 103 can be received. Here, as shown in S302, S303, S316, and S317 of Figure 3, the user using the service provided by the generation AI server 103 has been authenticated from each device, the scanner 102 and the computer 101, so the user's data can be referenced regardless of the device.

[0128] Figure 10(b) is a diagram showing an example of a screen when using a service provided by the generation AI server 103 from the display device 207 of the computer 101 in Figure 10(a). As shown in Figure 10(b), on the screen 1050 of the service provided by the generation AI server 103, output results 1070, 1090 are obtained as answers from the generation AI in response to user instruction inputs 1060, 1080.

[0129] Here, the document image 1002 in Fig. 10(a) is automatically input as an attached image 1062 in response to a user's instruction input. Also, the instruction statement 1003 in Fig. 10(a) is automatically input as an instruction statement 1061 in response to a user's instruction input. Here, the instruction statement 1061 in the second embodiment of the present invention is automatically input as an instruction statement that pre-includes an instruction statement to the effect that the user is "requested to input a sample to be referenced."

[0130] Accordingly, the generation AI requests input of a reference sample in response 1070 to the user's instruction input 1060. Therefore, in a subsequent interactive interaction with the generation AI, the user can be prompted to manually input a sample 1082 as an attachment from the computer 101 as a user's instruction input 1080.

[0131] Here, the manual input by the user may include an instruction statement 1081 as shown in Figure 10(b), but regardless of the presence or absence of the instruction statement 1081, the generation AI server 103 can interpret the attached file as a sample 1082. In a subsequent interaction, the generation AI server 103 refers to the sample 1082 and outputs the result 1091 of converting the document image 1062 in accordance with the instruction statement 1061 as an answer 1090 from the generation AI.

[0132] Note that a user using the generation AI server 103 from the computer 101 uses the services provided by the generation AI server 103 as a user who has been authenticated in S316 and S317. Here, the user information 1051 using the generation AI server 103 may be managed in association with the user information of the logged-in user of the computer 101. When the computer 101 detects that the logout 1052 button has been pressed, it terminates reception of instruction inputs 1060 and 1080 from the logged-in user of the user information 1051.

[0133] As described above, according to the second embodiment of the present invention, the sample input during the interaction between the computer 101 and the generation AI server 103 can be referenced, and the results converted into data having a layout and design that conforms to the user's intentions can be output. [Explanation of symbols]

[0134] 100 Information Processing Systems 101 Computer 102 Scanner 103 Generative AI Server

Claims

1. an instruction receiving step of receiving an instruction to cause the generation AI to convert the original image according to the sample file; an original image acquisition step of acquiring the original image; a sample acquisition step of acquiring the sample file; an instruction sentence generation step of generating an instruction sentence corresponding to the instruction; an instruction sending step of sending the original image, the sample file, and the instruction sentence to a server of the generation AI; An information processing method comprising:

2. 2. The information processing method according to claim 1, wherein the document image acquisition step acquires the document image by scanning the document.

3. 2. The information processing method according to claim 1, wherein the instruction includes an instruction to acquire a specified part of the sample file as the sample.

4. The part can be designated in the instruction receiving step, 4. The information processing method according to claim 3, wherein the partial options include any one of a template, a layout, a graph, a table, and a region enclosed by handwriting.

5. 2. The information processing method according to claim 1, wherein the sample file is image data obtained by scanning.

6. 2. The information processing method according to claim 1, wherein the sample file is electronic data designated by a user.

7. A program for causing a computer to execute the information processing method according to any one of claims 1 to 6.

8. An instruction receiving means for receiving an instruction to cause the generating AI to convert the original image according to the sample file; an original image acquisition means for acquiring the original image; a sample acquisition means for acquiring the sample file; an instruction statement generating means for generating an instruction statement corresponding to the instruction; an instruction sending means for sending the original image, the sample file, and the instruction sentence to the server of the generating AI; An information processing device comprising:

9. 9. The information processing apparatus according to claim 8, wherein the document image acquisition means acquires the document image by scanning the document.

10. 9. The information processing apparatus according to claim 8, wherein the instruction includes an instruction to acquire a specified part of the sample file as the sample.

11. The part can be designated via the instruction receiving means, 11. The information processing apparatus according to claim 10, wherein the partial options include any one of a template, a layout, a graph, a table, and a region enclosed by handwriting.

12. 9. The information processing apparatus according to claim 8, wherein the sample file is image data obtained by scanning.

13. 9. The information processing apparatus according to claim 8, wherein the sample file is electronic data designated by a user.

14. An instruction receiving means for receiving an instruction to cause the generating AI to convert the original image according to the sample file; an original image acquisition means for acquiring the original image; a sample acquisition means for acquiring the sample file; an instruction statement generating means for generating an instruction statement corresponding to the instruction; an instruction sending means for sending the original image, the sample file, and the instruction sentence to the server of the generating AI; A multifunction peripheral comprising:

Images