Program, information processing device, and information processing method

Abstract

The goal is to enable referencing the original image from an image generated by a generative AI. [Solution] The program of the present invention causes an information processing device to perform the following steps: a reception step of receiving prompt input from a user, and an output step of outputting image data of an image in which code information for accessing the input image data is added to an image generated by a generating AI using the input prompt and input image data.

Description

Technical Field

[0001] The present invention relates to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] Patent Document 1 discloses a scanning system in which a QR code (registered trademark) is added to a monochrome image generated by monochrome scanning a color image, and the user can refer to the color image by reading the QR code.

[0003] In recent years, services using generative AI have been rapidly developed, and a mechanism has been established that allows users to access various services by natural language input.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When an image is edited using generative AI or when another image is generated based on a certain image using generative AI, it may be desired to check the image before editing or the image used as a basis.

[0006] In Patent Document 1, it is conceivable to add a QR code to a monochrome image when it is converted from a color image to a monochrome image, but it is not conceivable to add code information for checking the image before editing when an image is edited using generative AI or the like.

[0007] This invention has been made in view of the above problems, and aims to enable referencing of the original image of an image generated by a generation AI from an image generated by a generation AI. [Means for solving the problem]

[0008] The program of the present invention causes an information processing device to perform a reception step of receiving prompt input from a user, and an output step of outputting image data of an image in which code information for accessing the input image data is added to an image generated by a generation AI using the input prompt and input image data. [Effects of the Invention]

[0009] This invention makes it possible to reference the original image of an image generated by a generation AI from an image generated by a generation AI. [Brief explanation of the drawing]

[0010] [Figure 1] Figure showing an example of the system of the present invention. [Figure 2] This diagram shows an example of the hardware configuration of the computer, smartphone, generation AI server, and ID management server of the present invention. [Figure 3] This figure shows an example of the hardware configuration of the MFP of the present invention. [Figure 4] This diagram shows an example of the software configuration of the computer, smartphone, generation AI server, and ID management server of the present invention. [Figure 5] This diagram shows examples of screens displayed by a document processing application and screens displayed on a smartphone. [Figure 6] This diagram shows an example of an instruction message that a computer sends to an AI server. [Figure 7] A diagram showing examples of content before and after editing. [Figure 8] This diagram shows an example of the information managed by the ID management server. [Figure 9]Sequence diagram showing an example of the processing of the system of the present invention [Figure 10] Flowchart showing an example of the processing of the generation AI server [Figure 11] Diagram showing an example of the software configuration of the MFP [Figure 12] Diagram showing an example of the LUI displayed on the display unit of the MFP [Figure 13] Sequence diagram showing an example of the processing of the system of the present invention

Mode for Carrying Out the Invention

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined.

[0012] <Embodiment 1> <System Configuration> Hereinafter, Embodiment 1 of the present invention will be described. First, referring to FIG. 1, the network configuration of the printing system according to the present embodiment will be described.

[0013] As shown in FIG. 1, the printing system 10 includes a computer 100, a smartphone 200, an MFP (Multi Function Peripheral) 300, a generation AI server 400, and an ID management server 500, which are terminal devices and information processing devices. For example, the computer 100, the smartphone 200, and the MFP 300 are arranged in the office and are communicably connected to each other via the in-house network 600. The in-house network 600 is connected to the external Internet 700 via a router (not shown). Here, the computer 100 is an example of a terminal device and an information processing device, and the generation AI server 400 and the ID management server 500 are examples of information processing devices.

[0014] The generation AI server 400 is communicably connected to the computer 100, the smartphone 200, the MFP 300, and the ID management server 500 via the Internet 700 and the in-company network 600. The generation AI server 400 is a server managed by an operator that provides generation AI services. The ID management server 500 is communicably connected to the computer 100 and the generation AI server 400 via the Internet 700 and the in-company network 600. The ID management server 500 is a server managed by an operator that provides printing services. Note that the computer 100, the smartphone 200, the generation AI server 400, and the ID management server 500 may be separate entities or may exist on the same hardware as a single unit. Also, the in-company network 600 may be a wired connection or a wireless connection.

[0015] <Hardware Configuration> Referring to FIGS. 2 and 3, the hardware configuration of each device of the printing system 10 will be described. FIG. 2 shows the hardware configuration of the computer 100, the smartphone 200, the generation AI server 400, and the ID management server 500. FIG. 3 shows the hardware configuration of the MFP 300.

[0016] As shown in FIG. 2, the computer 100 includes a display unit 101, an operation unit 102, a storage unit 103, a control unit 104, and a network communication unit 105, which are communicably connected to each other. The computer 100 is not particularly limited, and for example, a desktop or laptop personal computer, a tablet terminal, a smartphone, etc. can be applied. The control unit 104 includes a CPU (Central Processing Unit) 141 and a memory 142, and controls the entire computer 100.

[0017] The display unit 101 is composed of a display such as an LCD panel and can display images, etc. The operation unit 102 is composed of a mouse, keyboard, etc. and can accept input operations from the user. The storage unit 103 is composed of a storage medium such as a hard disk or SSD and stores various programs (software) necessary for the operation of the computer 100. The programs are loaded into memory 142 as needed and executed by the CPU 141. The programs include programs that cause the CPU 141 to execute each part and function of the printing system 10. Note that these programs are not limited to those stored in the computer 100. The programs may be stored in the generation AI server 400 or the ID management server 500, for example, or they may be distributed and stored across the computer 100, the generation AI server 400 and the ID management server 500. The network communication unit 105 is a network interface and transmits and receives data to and from external devices via an external network.

[0018] Furthermore, since the hardware configuration of the smartphone 200 is almost the same as that of the computer 100, the hardware configuration of the smartphone 200 will not be described in this embodiment.

[0019] The generation AI server 400 has a display unit 401, an operation unit 402, a storage unit 403, a control unit 404, and a network communication unit 405, which are connected to each other so as to be able to communicate with one another. The control unit 404 is composed of a CPU 441, a memory 442, and a GPU (Graphics Processing Unit) 443, and controls the entire generation AI server 400. As the hardware configuration of the generation AI server 400 is almost the same as that of the computer 100, a detailed explanation will be omitted.

[0020] Furthermore, the ID management server 500 includes a display unit 501, an operation unit 502, a storage unit 503, a control unit 504, and a network communication unit 505, which are connected to each other so as to be able to communicate with one another. The control unit 504 includes a CPU 541, a memory 542, and a GPU 543. The hardware configuration of the ID management server 500 is almost the same as that of the generation AI server 400, so a description is omitted.

[0021] As shown in Figure 3, the MFP300, which is a printing, scanning, image forming, and image processing device, has a display unit 301, an operation unit 302, a storage unit 303, a control unit 304, a network communication unit 305, a scanning unit 306, and a printing unit 307. These are connected to each other so as to be able to communicate. The control unit 304 is configured to include a CPU 341 and a memory 342 and controls the entire MFP300. The scanning unit 306 reads an image of a document and generates image data. The generated image data may be transmitted externally via the network communication unit 305. Alternatively, the printing unit 307 may print the image on paper based on the generated image data. The network communication unit 305 is a network interface and transmits and receives information such as image data via wired or wireless communication. The printing unit 307 can print an image based on the image data received from an external device by the network communication unit 305.

[0022] <Software Configuration> Referring to Figure 4, the software configuration of each device included in the printing system 10 will be explained.

[0023] Computer 100 includes an application 1100, a generation AI client 1200, and an OS (Operating System) 1400. The application 1100 includes any software, such as security software, spreadsheet software, and browser software; in the example shown in Figure 4, it includes a document processing application 1110. The document processing application 1110 is an application that edits and displays content such as text, photographs, and diagrams. The document processing application 1110 is not particularly limited and could include, for example, spreadsheet software, browser software, or presentation software.

[0024] The generation AI client 1200 refers to client software running on computer 100. The generation AI client 1200 receives instruction text (prompt) input from the user, generates a transmission instruction text as shown in Figure 5, and sends it to the generation AI server 400. Here, the transmission instruction text is the instruction text input by the user with additional instructions added. The generation AI server 400 and the generation AI client 1200 together may be referred to as the generation AI system. In this specification, the content generation process performed by the generation AI system may be referred to as the conversion process.

[0025] OS1400 is software that performs the basic operations of computer 110. OS1400 includes GDI1410, printer driver 1420, and spooler 1430. Printer driver 1420 is provided by the vendor that provides OS1400 or MFP300. Applications 1100 and printer driver 1420, etc., are all managed by OS1400 and become usable after being installed on OS1400. Alternatively, printer driver 1420 may be pre-installed in OS1400.

[0026] GDI1410 is an internal OS1400 component that provides an external interface for drawing, such as printing. The printer driver 1420 displays the print settings for printing with the MFP300 as a UI (User Interface). The printer driver 1420 also generates drawing data (PDL: Page Description Language) that the MFP300 can interpret during printing. The spooler 1430 processes the sending of the PDL generated by the printer driver 1420 to the MFP300. Although this description uses Microsoft's Windows® as the OS, other operating systems may be used as OS1400. Furthermore, the print data that the printer driver 1420 receives from the printing application may be in XML Paper Specification format (XPS format), not just GDI format.

[0027] Alternatively, the generation AI client 1200, printer driver 1420, GDI 1410, and spooler 1430 may be installed on OS 1400 as a single program.

[0028] For example, one program may receive prompt input from the user, send the input prompts and image data to the AI ​​server 400, and display the screen 6000 in Figure 5. Alternatively, all the processing of the computer 100 in the sequence in Figure 9 may be performed by a single application.

[0029] Smartphone 200 has applications 2100 and OS 2400. Since OS 2400 has almost the same software configuration as OS 1400, a detailed explanation of OS 2400 is omitted here. Applications 2100 include arbitrary software; in the example in Figure 4, it includes a document processing application 2110 and a QR code reading application 2120. The document processing application 2110, like the document processing application 1110, is an application for editing and displaying content such as text (documents), photographs, and chart data. The QR code reading application 2120 has the function of reading code information such as QR codes placed by the user through the camera built into the smartphone 200. The QR code reading application 2120 can read not only 2D codes but also 1D codes such as barcodes.

[0030] The generation AI server 400 includes an application 4100, an OS 4120, and an LLM (Large Language Model) 4130. Note that the OS 4120 has almost the same software configuration as the OS 1400, so a description of the OS 4120 is omitted here. The application 4100 contains arbitrary software; in the example in Figure 4, it includes the generation AI application 4110. The generation AI application 4110 is an application that has the interpretation processing function for the generation AI. The generation AI application 4110 sends user instruction sentences sent from the generation AI client 1200 to the LLM 4130, and sends the response sentences received from the LLM 4130 back to the generation AI client 1200. The LLM 4130 is a language model (learning model) trained on a vast amount of text data. The LLM 4130 interprets the instruction sentences sent from the generation AI application 4110 and responds with the interpretation result to the generation AI application 4110. Known LLM4130 models include GPT-4, GPT-4o, Bert, and Transformer models.

[0031] The ID management server 500 also includes an application 5100, an OS 5120, and an ID management database 5130. Since the OS 5120 has almost the same software configuration as OS 1400, a detailed explanation of OS 5120 is omitted here. The application 5100 contains arbitrary software; in the example in Figure 4, it includes the ID management application 5110. The ID management application 5110 generates IDs when it receives an ID generation request from any application, including the generation AI application 4110. Here, an ID refers to a unique, non-repeating ID composed of one or more data values. The IDs generated by the ID management application 5110 are either stored in the ID management database 5130 or returned to the application. The ID management database 5130 stores ID-related data as shown in Figure 8.

[0032] Figure 5 shows an example of a screen 6000 displayed on the display unit 101 by a document processing application 1110 on a computer 100 and a screen 6100 displayed on a smartphone 200. The document processing application 1110 displays screen 6000, which includes the generation AI client UI 1250 of the generation AI client 1200 that performs UI display. Furthermore, the document processing application 1110 displays screen 6000, which includes a preview display unit 6005 (image data display unit) that previews the image to be printed and a print button 6006. The generation AI client UI 1250 includes a chat unit 6002 that displays messages sent and received by the user between the generation AI client 1200 and the generation AI server 400. It also includes a prompt input unit 6003 (area display unit) which is an area for entering prompts, and an execution button 6004 for sending the entered prompt or an edited prompt to the generation AI server 400. The chat unit 6002 displays the prompt (string) entered by the user. Furthermore, the LLM4130 of the generation AI server 400 interprets the prompt, and the message output as a response is also displayed. When the user enters a prompt in the prompt input unit 6003 and clicks the execute button 6004, the document processing application 1110 sends an instruction to the generation AI server 400. The document processing application 1110 receives the conversion command output from the generation AI server 400 as a response to the prompt, converts the document based on the conversion command, and displays the edited document in the preview display unit 6005. As an example of the conversion process, if the instruction "Summarize the content to one page or less" is sent, the document processing application 1110 converts content 5001 in Figure 7 to content 5101 and displays it. Then, when the user determines that the edited document is appropriate and presses the print button 6006, a print instruction including the image data generated by the conversion and settings for printing that image data is sent to the MFP300. Then, an image based on the image data generated by the editing is printed.Thus, the prompt input section 6003, which is an area for the user to enter prompts, and the preview display section 6005, which displays the image with the QR code attached, are displayed on the same screen.

[0033] The example described shows the document processing application 1110 converting a document (input image data) based on a conversion command received from the generation AI server 400, but this is not the only example. For example, the document processing application 1110 may send a prompt and the unedited image data (input image data) to the generation AI server 400, and the generation AI server 400 may convert the image data based on the received prompt and image data. The converted image data may then be sent to the document processing application 1110. In that case, the document processing application 1110 displays the edited image data received from the generation AI server 400 on the preview display unit 6005.

[0034] As mentioned above, a single application installed on the computer 100 may display the generation AI client UI 1250, the preview display unit 6005 which shows a preview of the image to be printed, and the print button 6006. In other words, that single application may execute the processing of the document processing application 1110 and the processing of the generation AI client UI 1250 as described above. For example, that single application may accept prompt input from the user, selection of the execute button 6004, and selection of the print button 6006. Furthermore, it may also execute processes such as sending prompts and input image data to the generation AI server 400, receiving image data generated by the generation AI server 400 through conversion, and displaying it on the preview display unit 6005.

[0035] An example of the UI displayed by the document processing application 2110 on the smartphone 200 is shown. The document processing application 2110 includes the image reading unit 6102 of the QR code reading application 2120, the pre-summarization content display unit 6103, the difference display unit 6104, and the prompt display unit 6105.

[0036] The image reading unit 6102 displays the video captured by the camera built into the smartphone 200, and the user places the QR code within the video for scanning. Here, the QR code may contain some or all of the data stored in the storage unit 503 of the ID management server 500, such as ID 901 and URL 902, as described later in Figure 8, and may also contain other data. The smartphone 200 then transmits the ID contained in the QR code to the ID management server 500, and displays the content received from the ID management server 500 on the pre-summarized content display unit 6103. If the QR code indicates a URL (information indicating the storage location of the image data), the smartphone 200 accesses the server indicated by that URL. The information received from that server is displayed on the pre-summarized content display unit 6103. The pre-summarized content display unit 6103 displays the image data with the QR code read by the camera attached, before it is edited or summarized by the generating AI, and the image data received from the ID management server 500 or the server indicated by the URL. In other words, the ID management server 500 and the server indicated by the URL store the image data before it is edited or summarized by the generating AI. That is, the code information such as the QR code attached to the image data contains information indicating the storage location of the image data that was input into the computer 100 and before it was edited by the generating AI.

[0037] The difference display unit 6104 displays the differences between the pre-summarization content and the post-summarization content as differences. In other words, the differences between the image data before and after editing by the generating AI are displayed as text. This allows the user to easily check what information has been omitted, added, or changed as a result of editing the image data by the generating AI.

[0038] Furthermore, the prompt display unit 6105 displays the prompts entered by the user when the document processing application 1110 performs the process of generating the summarized content from the pre-summarized content. In other words, the prompts used by the generation AI to generate the edited image data are displayed as text. This means that the image data entered into the computer 100 and the prompts entered by the user are displayed on the same screen on the display unit of the terminal device that reads the code information such as the QR code. This makes it easy for the user to see how the generation AI generated the image data (original document).

[0039] This prompt may be stored in any of the following storage devices: computer 100, smartphone 200, printer 300, generation AI server 400, and ID management server 500. For example, the prompt may be stored in the ID management server 500 in association with the ID indicated by the QR code, and the smartphone 200 may receive it from the ID management server 500. Alternatively, it may be stored along with the unedited image data (image data input to computer 100) on the server indicated by the URL indicated by the QR code, and the smartphone 200 may receive it from the storage server.

[0040] Furthermore, the screen 6100 in Figure 5, displayed by the document processing application 2110, may also display version information of the generating AI that produced the edited image data when the QR code is read. This version information may be stored in the ID management server 500, similar to the prompt, or it may be stored in the server indicated by the URL shown in the QR code.

[0041] Furthermore, all or part of the information displayed on screen 6100 in Figure 5 may be contained within a code such as a QR code. In other words, the information indicated by the QR code may directly include not only URLs and IDs, but also version information and prompts of the generated AI.

[0042] Figure 6 shows an example of a prompt that computer 100 sends to the generation AI server 400. The prompt consists of, for example, prompt 801 and attached content 802. Prompt 801 contains the prompt entered by the user into the prompt input unit 6003 of application 1110. Also, if the user wants to use the contents stored in the storage unit 103 of computer 100 as input to the LLM of the generation AI server 400, the instruction statement includes the file name and path of the attached content 802.

[0043] Figure 7 shows an example of content before and after editing. Here, we will explain the content before and after editing performed by the generation AI in response to user prompt input. Assume that the user inputs a prompt such as "Summarize the content in one page or less" into the prompt input unit 6003. Content 5001 is the content about the curry recipe before editing, and is image data including text 5002 and illustration 5003. Content 5101 is the content generated by summarizing content 5001 by the generation AI, and includes text 5102 which is a summary (omitted) of the text 5002 contained in the image, and illustration 5003 has been deleted. As a result, the curry recipe is expressed in one page. In other words, the image generated by the generation AI is an image that includes text generated by summarizing the text in the image (image before editing) of the image data input to the computer 100. Alternatively, the image generated by the generation AI may be an image generated by deleting the photograph (illustration) from the text and photograph (illustration) in the image (image before editing) of the image data input to the computer 100.

[0044] Content 5101 includes a QR code 5103 in addition to text 5102. As mentioned above, the QR code 5103 is information for accessing the image data (image data input to computer 100) before it is edited by the generating AI. In other words, image data is generated based on the prompts entered by the user and the image data input to computer 100. Then, image data of the image, with code information for accessing the image data input to computer 100 added to that image data, is output (displayed) on the display unit 101 of computer 100.

[0045] Furthermore, while the image with the added code information is displayed, the computer 100 may accept a prompt input from the user to the prompt input unit 6003 to specify the insertion position of the QR code. In this case, the computer 100 sends the input prompt to the generation AI server 400 to generate image data with the QR code added to the position specified by the user. When the computer 100 adds the QR code, while the image with the added code information is displayed, the computer 100 accepts the user's specification of the QR code position by selecting a button (not shown). The computer 100 then inserts the QR code at the position corresponding to the selected button (bottom right, top right, bottom left, top left, etc.).

[0046] The generation AI server 400 may also be responsible for generating the image data to which code information such as a QR code has been added. When the generation AI server 400 generates the image data, it stores the image data input to the computer 100 in a predetermined storage location and generates a QR code as information indicating that storage location. The generation AI server 400 then adds the generated QR code to the image data edited from the image data input to the computer 100. The generation AI server 400 then transmits the image data with the added QR code to the computer 100. As a result, the image data (content 5101) with the added QR code is displayed on the preview display unit 6005 of the display unit 101 of the computer 100.

[0047] The computer 100 may generate the image data to which code information such as a QR code has been added. In this case, the computer 100 stores the image data input to the computer 100 in a predetermined storage location and generates a QR code as information indicating that storage location. The generation AI server 400 then transmits the image data input to the computer 100. The computer 100 then receives the image data that has been edited and generated by the generation AI server 400. The computer 100 generates image data by adding the generated QR code to the image of the received image data. As a result, the image data (content 5101) with the QR code added is displayed on the preview display unit 6005 of the display unit 101 of the computer 100. The computer 100 then sends print data containing the image data of the image with the code information added to the MFP 300. The print data transmitted here includes the print settings received by the computer 100. The MFP 300 prints the image with the code information added to the paper based on the print settings included in the received print data. This process corresponds to the processes S1724 to S1726 described later. Through these processes, it is possible to output image data of an image generated by a generation AI using the input prompt and input image data, with code information for accessing the input image data added to the image data.

[0048] The MFP300 may also be responsible for generating the image data to which code information such as a QR code has been added. In this case, the computer 100 stores the image data input to the computer 100 in a predetermined storage location and transmits information indicating that storage location to the MFP300. Furthermore, the generation AI server 400 sends print data to the MFP300 that includes the image data edited from the image data input to the computer 100 and sent back to the computer 100. As a result, the MFP300 receives code information indicating the storage locations of the edited image data and the image data before editing. The MFP300 then uses this information to generate image data (content 5101) to which a QR code has been added and prints the generated image data.

[0049] Figure 8 illustrates the ID list stored in the ID management database 5130 of the ID management server 500. The ID list includes content-related information such as ID 901, URL 902, content name 903, and page number 904. This information is used as a search key when searching for unedited content on the ID management server 500 from a QR code read by the smartphone 200, and as related information provided to the user.

[0050] The ID901 column contains the identifiers "NXG9-4PQ7" and "N76-Z1K8". The URL902 column contains the URLs "https: / / www.example.com / AAA.docx" and "https: / / www.example.com / BBB.pdf", which are links to access the respective data. The Content Name903 column contains the document file names "AAA.docx" and "BBB.pdf". The Page Count904 column contains the page counts of the content, "5 pages" and "2 pages". When the ID Management Server 500 receives a QR code generation instruction from the Generation AI Server 400, it adds one row to the ID list and stores a series of information about the content 5001 before editing. Specifically, it stores the IDs included in the generated QR code in the ID901 column, and the URL, content name, and page count of the content 5001 before editing in URL902, content name903, and page count904, respectively.

[0051] Furthermore, when a user scans a QR code with the smartphone 200 and the ID management server 500 receives a command from the smartphone 200 to send the unedited content, the server searches for the unedited content based on the ID corresponding to the QR code scanned by the smartphone 200. If the matching content is found, it is returned to the smartphone 200. Details will be explained later in the description of Figure 9.

[0052] In this explanation, the ID management server 500 stores an example where it stores values ​​such as ID 901, URL 902, content name 903, and page number 904 in its ID list. However, the values ​​stored in the ID list are not limited to these. For example, the ID list may also store values ​​such as content size, content creator name, and organization information.

[0053] In this explanation, we have described an example of searching for relevant content from an ID list using ID 901, but the method of searching for content is not limited to this. For example, if the ID management server 500 stores data on content that has been searched in the past (referred to as past search data) in the storage unit 403, and the ID management server 500 has learned patterns from the past search data, the following may be used. For example, the ID management server 500 may search for content using such a recommendation function.

[0054] Furthermore, although this explanation describes an example of searching for relevant content by matching only one ID, it is also possible to use non-unique IDs or multiple IDs to search for relevant content. Alternatively, the URL of the server where the unedited content is stored can be embedded in the QR code, and the ID management server 500 can search for the unedited content based on the URL of the QR code it reads.

[0055] Referring to Figure 9, the processing procedure of this embodiment in the printing system 10 according to this embodiment will be described. The processing described below corresponds to a series of processes starting from [S700] in the illustrated sequence diagram.

[0056] In S701, the user enters a command (prompt) into the generating AI client 1200, stating, "Please convert the content to one page or less."

[0057] In S702, the CPU 141 of computer 100 executes the generation AI client 1200, receives prompts from the user, and creates instructions to send to the generation AI server 400.

[0058] In S703, the CPU 141 of computer 100 executes the generation AI client 1200 and sends the created instruction to the generation AI application 4110 of the generation AI server 400.

[0059] In S704, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and requests the LLM 4130 to interpret the instruction statement.

[0060] In S705, the control unit 404 of the generation AI server 400 controls the LLM 4130 to interpret the instruction text and generate the interpretation result. This interpretation result includes a request for content conversion.

[0061] In S706, the control unit 404 of the generation AI server 400 controls the LLM 4130 and responds with the interpretation result to the generation AI application 4110.

[0062] In S707, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and responds with the interpretation result to the generation AI client 1200.

[0063] In S708, the CPU 141 of computer 100 runs the generation AI client 1200 and responds with the interpretation result to the document processing application 1110.

[0064] In step S709, the CPU 141 of computer 100 executes the document processing application 1110 and edits the content from the original content 5001 to the edited content 5101 based on the interpretation results. Alternatively, this editing process may be performed by the generation AI application 4110 of the generation AI server 400. In that case, computer 100 receives and displays the edited content 5101 generated by the generation AI application 4110.

[0065] In S710, the CPU 141 of computer 100 runs the document processing application 1110 and sends the unedited content 5001 to the generated AI client 1200.

[0066] In S711, the CPU 141 of computer 100 runs the generation AI client 1200 and sends the unedited content 5001 to the generation AI application 4110.

[0067] In S712, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and sends the unedited content 5001 to the ID management application 5110.

[0068] In step S713, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and sends a request to generate a QR code 5103 to the ID management application 5110.

[0069] In S714, the CPU 541 of the ID management server 500 executes the ID management application 5110 and generates a QR code 5103 with data including the ID embedded in it.

[0070] This QR code may be generated by computer 100, generation AI server 400, or MFP 300, as described above. Furthermore, the information indicated by the QR code may not be an ID, but rather a URL indicating the storage location of the image data. In these cases, the flow described below will differ in part, and the information exchange between the devices described above will occur.

[0071] In step S715, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and saves the unedited content 5001 and ID list to the ID management database 5130.

[0072] In S716, the CPU 541 of the ID management server 500 executes the ID management application 5110 and sends the generated QR code 5103 to the generation AI application 4110.

[0073] In S717, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and adds the QR code 5103 to the edited content 5101.

[0074] In S718, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and returns the edited content 5101 to the generation AI client 1200.

[0075] In S719, the CPU 141 of computer 100 executes the generation AI client 1200 and returns the edited content 5101 to the document processing application 1110.

[0076] In S720, the CPU 141 of computer 100 executes the document processing application 1110 and displays the edited content 5101 on the preview display unit 6005.

[0077] On S721, the user presses the print button 6006 of the document processing application 1110.

[0078] In S722, the CPU 141 of computer 100 executes the document processing application 1110 and sends print data to the spooler 1430. The print data sent at this stage is as described above.

[0079] In S723, the CPU 141 of computer 100 controls the spooler 1430 and sends print data to the printer driver 1420.

[0080] In S724, the CPU 141 of computer 100 controls the printer driver 1420 to generate print jobs.

[0081] In S725, the CPU 141 of computer 100 controls the printer driver 1420 to send the print job to MFP300.

[0082] In S726, the MFP300 prints an image onto paper based on the received print job.

[0083] In S727, the user scans the content's QR code 5103 using the QR code reading application 2120 on their smartphone 200.

[0084] In S728, the CPU 241 of the smartphone 200 executes the document processing application 2110 and sends a request to the ID management application 5110 to retrieve the unedited content 5001. The retrieval request includes the ID 901 of the scanned QR code.

[0085] In S729, the CPU 241 of the smartphone 200 runs the document processing application 2110. Also, the CPU 541 of the ID management server 500 runs the ID management application 5110 and sends a request to the ID management database 5130 to retrieve the unedited content 5001.

[0086] In S730, the CPU 541 of the ID management server 500 controls the ID management database 5130 and determines the unedited content 5001 to be sent based on the ID 901 obtained from the document processing application 2110.

[0087] In S731, the CPU 541 of the ID management server 500 controls the ID management database 5130 and sends the unedited content 5001 to the ID management application 5110.

[0088] In S732, the CPU 541 of the ID management server 500 controls the ID management application 5110 and sends the unedited content 5001 to the document processing application 2110.

[0089] In S733, the CPU 241 of the smartphone 200 executes the document processing application 2110 and displays the unedited content 5001 acquired by the display unit 6103.

[0090] By performing the above processing steps, users can easily refer to the original content from the edited content.

[0091] Figure 10 is a flowchart showing an example of processing performed by the generation AI server 400. The processing described below is achieved, for example, by the CPU 441 of the generation AI server 400 reading a program stored in the storage unit 403 into memory 442 and executing it.

[0092] In S101, when the generation AI server 400 receives an instruction (prompt 801) from the generation AI client 1200, this process begins. This instruction might be something like, "Please convert the content to one page or less."

[0093] In S102, the generation AI application 4110 of the generation AI server 400 adds appropriate information to the instruction text. For example, information such as content information, model information, and organizational information is added through mechanisms called RAG (Retrieval-Augmented Generation) or System Prompt. This is expected to improve the interpretation accuracy of the LLM 4130.

[0094] In S103, the generation AI application 4110 sends an instruction to the LLM 4130.

[0095] In S104, the LLM4130 performs the interpretation of the instruction. The interpretation process performed by the LLM4130 here includes one or more processes such as text preprocessing, text tokenization and vectorization, context understanding, text generation and prediction, and response generation.

[0096] In S105, LLM4130 sends the response statement generated in S104 to the generation AI application 4110.

[0097] In S106, the generation AI application 4110 determines whether the received interpretation result relates to content editing. If the generation AI application 4110 determines that it relates to content conversion, it generates a content conversion instruction in S107 and sends the response statement and content editing instruction to the generation AI client 1200 in S108. Then, it terminates the process in S110. If the generation AI application 4110 determines in S106 that the response statement does not relate to content conversion, it does not generate a content conversion instruction, sends only the response statement to the generation AI client 1200 in S109, and terminates the process in S110.

[0098] By performing the above processing, the generation AI server 400 can change the response message it sends to the generation AI client 1200 depending on whether the prompt entered by the user includes instructions for editing content or not. As a result, the generation AI client 1200 can provide the document processing application 1110 with a means to determine whether or not the content should be summarized.

[0099] As described above, the printing system according to this embodiment includes a computer 100, a smartphone 200, a generation AI server 400 that provides generation AI services, an ID management server 500 that provides ID management services, and an MFP 300.

[0100] In one example, when the generation AI server 400 performs content editing based on user prompt input, it requests the ID management server 500 to generate and save a QR code. The generated QR code is then attached to the content and returned to the document processing application 1110, where the summarized content is printed based on the user's print command. Subsequently, the user scans the QR code of the summarized content with their smartphone 200, which retrieves the pre-edited content from the ID management server 500, allowing the user to easily access the pre-edited content.

[0101] <Second Embodiment> In the second embodiment, the MFP300 has a generation AI client 1200, and the embodiment describes a case in which the user converts an image scanned by the MFP300 using the generation AI system.

[0102] As shown in Figure 11, the MFP300 includes an application 3100, a generation AI client 3200, a controller 3300, and an OS 3400. The application 3100 contains arbitrary software, and in the example in Figure 11, it includes a scan application 3110 that controls the scanning process and a preview display application 3120 that controls the preview display.

[0103] The generation AI client 3200 is an application that controls the operation of the scanning unit 306. The generation AI client 3200 represents client software that runs on the MFP 300. The generation AI client 3200 receives an instruction (prompt) from the user, generates an instruction to send to the generation AI server 400 as shown in Figure 5, and sends it to the generation AI server 400.

[0104] The controller 3300 refers to a controller that performs functions such as sending and receiving data transmitted from the scan unit 306 and any computer, and network management.

[0105] OS3400 is the operating system that performs task management, memory management, file management, peripheral device management, etc., for the MFP300.

[0106] Figure 12 shows an example of the LUI (Local User Interface) 1401 displayed on the display unit 301 of the MFP300.

[0107] LUI1401 includes a scan button 1402, a preview display unit 1403, a generation AI client UI 1404, and a print button 1406.

[0108] The scan button 1402, when selected, causes the scanning unit 306 to perform a scan of the document. The user places the paper to be scanned on the MFP300's document glass and presses the scan button 1402 to initiate the scanning process in the scanning unit 306. The scanning unit 306 can also read documents placed on the glass surface, and can scan documents being transported by the ADF (Auto Document Feeder) placed on the document glass.

[0109] The print button 1406 is a button that, when selected by the user, prints an image onto paper. The image printed here is image data that has been input to the MFP300 in various ways. For example, it may be image data included in a print job received from the computer 100, or image data obtained from a USB memory. It may also be image data generated by scanning a document with the scan unit 306, and stored in the storage unit 303. The user sets the paper they want to use for printing into the MFP300 and presses the print button 1406, which executes the printing process in the print unit 307.

[0110] The preview display unit 1403 can display any content, such as image data generated by scanning a document with the scanning unit 306, or content after processing that image data with the generation AI. In the example in Figure 12, the image of the image data after processing with the generation AI and a QR code 1405 indicating information corresponding to the image data before processing with the generation AI are displayed.

[0111] The generated AI client UI1404 has the same configuration as the generated AI client UI1250, so its explanation will be omitted.

[0112] The system is configured to allow the user to generate new data based on scanned image data by pressing the scan button 1402 and entering prompts in the generation AI client UI 1404.

[0113] Referring to Figure 13, the processing procedure in this embodiment of the printing system 10 will be described.

[0114] In S1701, the user inputs an instruction (prompt) to the MFP300's generation AI client 3200, such as "Please scan the content and extract only the text." At this time, the user inputs the prompt with the document placed on the document glass of the scanning unit 306.

[0115] In S1702, the CPU 341 of the MFP300 executes the generation AI client 3200, receives a prompt from the user, and creates an instruction message to send to the generation AI server 400.

[0116] In S1703, the CPU 341 of the MFP300 executes the generation AI client 3200 and sends the created instruction to the generation AI application 4110 of the generation AI server 400.

[0117] In S1704, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and requests the LLM 4130 to interpret the instruction statement.

[0118] In S1705, the control unit 404 of the generation AI server 400 controls the LLM 4130 to interpret the instruction and generate the interpretation result. This interpretation result includes a request for content conversion.

[0119] In S1706, the control unit 404 of the generation AI server 400 controls the LLM 4130 and responds with the interpretation result to the generation AI application 4110.

[0120] In S1707, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and responds with the interpretation result to the generation AI client 3200.

[0121] In S1708, the CPU 341 of the MFP300 executes the generation AI client 3200 and, based on the interpretation results, sends a scan command to the controller 3300.

[0122] In S1709, the CPU 341 of the MFP300 controls the controller 3300 and sends a scan command to the scan unit 306.

[0123] In S1710, the CPU 341 of the MFP300 controls the scanning unit 306 and performs the scanning process of the document placed on the MFP300's document glass.

[0124] In S1711, the CPU 341 of the MFP300 controls the scanning unit 306 and sends scan data (image data generated by scanning) to the controller 3300. This scan data may be the image file of the scanned image data itself. Alternatively, OCR (Optical Character Recognition) processing may be performed on the image data to obtain a string, and the obtained string (text data) may be sent.

[0125] In S1712, the CPU 341 of the MFP300 controls the controller 3300 and sends the unedited scan data to the generation AI application 4110.

[0126] In S1713, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and performs editing processing on the unedited scanned data based on the prompt. For example, in response to the prompt "Scan the content and extract only the text," the generation AI application 4110 extracts only the text from the scanned data and generates image data.

[0127] In S1714, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and sends the pre-edit scan data to the ID management application 5110.

[0128] In S1715, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and sends a QR code generation request to the ID management application 5110.

[0129] In S1716, the CPU 541 of the ID management server 500 executes the ID management application 5110 and generates a QR code 5103 embedded with data including the ID. The QR code generated here only needs to indicate information for accessing the unedited scanned data, similar to Embodiment 1. For example, it may be a URL indicating the storage location where the unedited scanned data is stored.

[0130] In S1717, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and saves the unedited scan data and ID list to the ID management database 5130.

[0131] In S1718, the CPU 541 of the ID management server 500 executes the ID management application 5110 and sends the generated QR code 5103 to the generation AI application 4110.

[0132] In S1719, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and adds the QR code 5103 to the edited scanned data.

[0133] In S1720, the control unit 404 of the generation AI server 400 executes the generation AI application 4110 and returns the edited scan data to the generation AI client 3200.

[0134] In S1721, the CPU 341 of the MFP300 executes the generation AI client 3200 and returns the edited scan data to the controller 3300.

[0135] In S1722, the CPU 341 of the MFP300 controls the controller 3300 and sends the edited scan data and the instruction to display the edited scan data to the display unit 301.

[0136] In step S1723, the CPU 341 of the MFP300 controls the display unit 301 and displays the edited scan data with the generated QR code attached to the preview display unit 1403.

[0137] In S1724, the user presses the print button 1406, and the preview display unit 1403 The print command for the edited scanned data displayed is sent to the controller 3300.

[0138] In step S1725, the CPU 341 of the MFP300 controls the controller 3300 and sends a print command for the edited scanned data to the print unit 307.

[0139] In S1726, the MFP300's CPU341 controls the printing unit307 and executes the printing process to print the edited scanned image data onto paper. The user can input print settings at any time via the LUI1401 (settings reception unit).

[0140] In S1727, the user reads the QR code 5103 printed on the paper in S1726 using the QR code reading application 2120 on the smartphone 200.

[0141] In S1728, the CPU 241 of the smartphone 200 executes the document processing application 2110 and sends a request to the ID management application 5110 to retrieve the unedited scanned data.

[0142] In S1729, the CPU 541 of the ID management server 500 executes the ID management application 5110 and sends a request to the ID management database 5130 to retrieve the unedited scan data.

[0143] In S1730, the CPU 541 of the ID management server 500 controls the ID management database 5130 and determines the unedited scan data to send based on ID 901.

[0144] In S1731, the CPU 541 of the ID management server 500 controls the ID management database 5130 and sends the unedited scan data to the ID management application 5110.

[0145] In S1732, the CPU 541 of the ID management server 500 executes the ID management application 5110 and sends the unedited scanned data to the document processing application 2110.

[0146] In step S1733, the CPU 241 of the smartphone 200 executes the document processing application 1110 and displays the unedited scanned data acquired by the display unit 6103.

[0147] The processing procedure of this embodiment in the printing system 10 according to this embodiment has been described above. By performing the above processing procedure, the user can seamlessly perform scanning and data conversion processing for the document set in the MFP300.

[0148] The printing system according to this embodiment has been described above in the first and second embodiments. However, the present invention is not limited to the embodiments described above, and various modifications and changes are possible within the scope of its gist. The present invention can also be realized by supplying a program that implements one or more of the functions of the above embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0149] Furthermore, in the above embodiment, the computer 100 or the MFP 300 has an operation unit for inputting output conditions for the MFP 300. However, the present invention is not limited to this, and it is sufficient if at least one of the following devices is present: the computer 100, the smartphone 200, the MFP 300, the generation AI server 400, or the ID management server 500. The operation unit may be displayed on a browser UI such as Microsoft® Edge or Google® Chrome. Alternatively, the operation unit may be displayed on a generation application UI such as ChatGPT, a printing application UI such as Microsoft® Office, or a chat application UI such as Microsoft Teams.

[0150] Furthermore, in the above embodiment, the print setting data stored in the memory unit may actually differ in format depending on the OS 1400 and the printer driver 1420, such as PrintTicket or devmode. The present invention is applicable to the transmission and reception of such diverse data formats. Moreover, in the above embodiment, S637 assumes that the printer driver 1420 sends a print job to the MFP 300 connected to the same network 600 as the computer 100. However, the present invention is not limited to this. For example, in S725, the printer driver 1420 may upload the print job to a print service on the cloud and be configured to print to a printer connected to a different network from the computer 100 via the cloud.

[0151] Furthermore, in the printing system 10, for example, the generation AI server 400 (hereinafter referred to as "server") may be located outside of Japan, while the terminal device, computer 100 (hereinafter referred to as "terminal device"), may be located within Japan. Even in this case, files and data can be transmitted from the server to the terminal device, and the terminal device can receive these files and data. Thus, even when the server is located outside of Japan, the transmission and reception (sending and receiving) of files and data in this system are performed as a single unit. And, since the system functions because the terminal device in Japan receives these files and data, such transmission and reception can be considered as having taken place domestically. In this system, for example, even if the server is located outside of Japan and the terminal device is located within Japan, the terminal device can perform the main functions of this system, and the effects of those functions can be realized within Japan. For example, even if the server is located outside of Japan, if the terminal device constituting this system is located within Japan, it is possible to use the system domestically using that terminal device. And the use of this system may affect the economic interests of, for example, the patent holder.

[0152] Furthermore, in the above embodiment, the method of creating content using the document processing application 1110 was described. However, the present invention is not limited to this, and content may not be created using the document processing application 1110, but rather content stored in the storage unit 103 of the computer 100 may be imported, or content may be imported from an external storage device such as a USB or SD card.

[0153] Furthermore, although the document processing application 1110 and the generation AI client 3200 were described as separate configurations in this embodiment, the processing of the generation AI client 1200 may be included in the document processing application 1110.

[0154] Furthermore, in the first embodiment described above, the printer driver 1420 displays the print preview image. However, another program having a print preview image display function may also display the print preview image. For example, the document processing application 1110 may display the print preview image. The present invention is not limited to the above embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention.

[0155] <Other Embodiments> The present invention provides a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium. It can also be implemented by a process in which one or more processors in the computer of the system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0156] 10 Printing Systems 100 Computers 200 Smartphones 300 MFP 400 AI Generator Servers 500 ID Management Server

Claims

1. A reception process that accepts prompt input from the user, A program that causes an information processing device to execute an output step of outputting image data of an image generated by a generating AI using the input prompt and input image data, to which code information for accessing the input image data has been added.

2. The program according to claim 1, which causes the information processing device to further execute a transmission step of sending print data, including image data of an image to which the code information has been added, to a printing device.

3. The information processing device is further instructed to execute a setting acceptance process for accepting print settings. The program according to claim 2, characterized in that the print data includes image data of an image to which the code information has been added and the print settings.

4. The reception process includes a region display process that displays a region for receiving input from the prompt, The information processing device is further instructed to perform an image data display step, which displays an image to which the aforementioned code information has been added. The program according to claim 1, characterized in that the region and the image to which the code information is added are displayed on the same screen.

5. The program according to claim 4, characterized in that, while an image to which the code information is attached is displayed, the reception step accepts input from the user as a prompt to specify the position to which the code information is attached.

6. The program according to claim 1, characterized in that the code information includes information indicating the storage location of the input image data.

7. The program according to claim 6, characterized in that the information indicating the storage location is a URL.

8. The aforementioned storage location also stores the input prompt, The program according to claim 6, characterized in that the input image data and the input prompt are displayed on the display unit of the terminal device that reads the code information.

9. The aforementioned storage location also includes version information of the generated AI. The program according to claim 6, characterized in that the input image data and the version information are displayed on the display unit of the terminal device that reads the code information.

10. The program according to claim 1, characterized in that the code information is a QR code.

11. A receiving step of receiving image data of an image generated by the generating AI by transmitting the input prompt and the input image data to the generating AI server that provides the generating AI, The program according to claim 1, further characterized in that the information processing device is made to perform an addition step of adding the code information to the image of the received image data.

12. The program according to claim 1, characterized in that the image generated by the generation AI is an image that includes text generated by summarizing the text within the input image data.

13. The program according to claim 1, characterized in that the image generated by the generation AI is an image generated by deleting the photograph from the text and photograph within the input image data.

14. A means of receiving prompt input from the user, An information processing apparatus having output means for outputting image data of an image generated by a generation AI using the input prompt and input image data, to which code information for accessing the input image data has been added.

15. The information processing apparatus according to claim 14, further comprising a transmission means for transmitting print data including image data of an image to which the code information has been added to a printing device.

16. It further has a setting acceptance mechanism for accepting print settings, The information processing apparatus according to claim 15, characterized in that the print data includes image data of an image to which the code information has been added and the print settings.

17. The receiving means includes a region display means that displays a region for receiving input from the prompt, The system further includes an image data display means for displaying an image to which the aforementioned code information has been added, The information processing apparatus according to claim 14, characterized in that the aforementioned region and the image to which the code information is added are displayed on the same screen.

18. The information processing apparatus according to claim 14, characterized in that, while an image to which the code information is attached is displayed, the receiving means accepts input from the user as a prompt to specify the location to which the code information is attached.

19. The information processing apparatus according to claim 14, characterized in that the code information includes information indicating the storage location of the input image data.

20. The information processing apparatus according to claim 19, characterized in that the information indicating the storage location is a URL.

21. The aforementioned storage location also stores the input prompt, The information processing apparatus according to claim 19, characterized in that the input image data and the input prompt are displayed on the display unit of the terminal device that reads the code information.

22. The aforementioned storage location also includes version information of the generated AI. The information processing apparatus according to claim 19, characterized in that the input image data and the version information are displayed on the display unit of the terminal device that reads the code information.

23. The information processing apparatus according to claim 14, characterized in that the code information is a QR code.

24. A receiving means that receives image data of an image generated by the generating AI by transmitting the input prompt and the input image data to the generating AI server that provides the generating AI, The information processing apparatus according to claim 14, further comprising an addition means for adding the code information to the image of the received image data.

25. The information processing apparatus according to claim 14, characterized in that the image generated by the generation AI is an image that includes text generated by summarizing the text in the input image data.

26. The information processing apparatus according to claim 14, characterized in that the image generated by the generation AI is an image generated by deleting the photograph from the text and photograph within the input image data.

27. A reception process that accepts prompt input from the user, An information processing method comprising an output step of outputting image data of an image generated by a generating AI using the input prompt and input image data, to which code information for accessing the input image data has been added.

Images