Program, information processing device, and control method for the information processing device.

Abstract

The aim is to reduce the user's effort when generating images using previously entered prompts. [Solution] The present invention is characterized by causing an information processing device to execute a history display step that displays a history including an image generated based on a prompt entered by a user and an image of a first image data entered, and an image display step that displays an image generated based on a prompt included in the history selected from the displayed history and an image of a second image data entered.

Description

Technical Field

[0001] The present invention relates to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] In recent years, conversational AIs such as chatbots and generative AIs have been rapidly developed. Along with this development, various services using conversational AIs are being provided. Patent Document 1 shows a system that displays an image and edits the image based on a prompt input by a user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Even when the user wants to perform editing (image generation) in the same way as the previously edited image, but with an image different from the previously edited image, it would be time-consuming to input the same prompt again.

[0005] [[ID=​​​​​The present invention is characterized by causing an information processing device to execute a history display step that displays a history including an image generated based on a prompt entered by a user and an image of a first image data entered; and an image display step that displays an image generated based on a prompt included in the history selected from the displayed history and an image of a second image data entered. [Effects of the Invention]

[0007] According to the present invention, when generating an image using previously entered prompts, the effort required of the user can be reduced. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of the overall configuration of this system. [Figure 2] This diagram shows an example of the computer hardware configuration of this system. [Figure 3] This diagram shows an example of the hardware configuration of the AI ​​generation server for this system. [Figure 4] This diagram shows an example of the printer hardware configuration in this system. [Figure 5] This diagram shows an example of the software configuration of this system. [Figure 6] This diagram shows an example of the print application screen in this system. [Figure 7] This diagram shows an example of the print application history display screen in this system. [Figure 8] This diagram shows an example of the structure of historical data in this system. [Figure 9] A diagram showing an example of the settings screen for a printing application. [Figure 10] Sequence diagram showing an example of image conversion and printing via chat. [Figure 11] Sequence diagram showing an example of displaying historical data and applying historical data. [Figure 12] A flowchart illustrating an example of a printing application's processing. [Figure 13] A flowchart illustrating an example of the processing performed by the AI ​​generation server. [Figure 14] A flowchart illustrating an example of a printing application's processing. [Figure 15] A flowchart illustrating an example of a printing application's processing. [Modes for carrying out the invention]

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] <System Configuration> Below, we will first describe the network configuration of the printing system according to this embodiment with reference to Figure 1.

[0011] As shown in Figure 1, this printing system includes a computer 1000, which is a terminal device or information processing device, a printer 2000, which is an example of a printing device, and a generation AI server 30000. For example, the computer 1000 and the printer 2000 are located in an office and are connected to each other so that they can communicate with each other via the internal network 4000. The internal network 4000 is connected to the external internet 5000 via a router (not shown). Here, the computer 1000 is an example of a user terminal or information processing device, and the generation AI server 30000 is an example of a server device. Here, generation AI is an AI that generates content such as input image data, text, and audio based on instructions indicated by input prompts. For example, if a user inputs an image data and a prompt to change the layout of that image data, the generation AI generates image data based on the input image data and the prompts input by the user.

[0012] The generation AI server 30000 is communicably connected to the computer 1000 and the printer 2000 via the Internet 5000 and the in-company network 4000.

[0013] <Hardware Configuration> Referring to FIGS. 2, 3, and 4, the hardware configuration of each device of the printing system according to the present embodiment will be described. FIG. 2 shows an example of the hardware configuration of the computer 1000. FIG. 3 shows an example of the hardware configuration of the generation AI server 30000. FIG. 4 shows an example of the hardware configuration of the printer 2000.

[0014] As shown in FIG. 2, the computer 1000 includes a CPU 111, a ROM 112, a RAM 113, an HDD 114, and a network I / F 115. The computer 1000 also includes a display unit 116 and an operation unit 117. The CPU 111 reads out the control program stored in the ROM 112 and executes various processes to control the overall operation. The RAM 113 is used as a temporary storage area such as the main memory and work area of the CPU 111. The HDD 114 is a large-capacity storage unit that stores image data and various programs. The network I / F 115 is an interface for connecting to the Internet. It receives processing requests from other devices and services via the network I / F 115 and transmits and receives various information.

[0015] The display unit 116 is a display device such as a monitor, and may be integrated with the computer 1000 or connected to the computer 1000 as an external device. The screen information output by the CPU 111 is displayed on the display unit 116.

[0016] The operation unit 117 is an input device such as a keyboard or a mouse, and may be integrated with the computer 1000 or connected to the computer 1000 as an external device. The CPU 111 receives various operations of the user via the operation unit 117.

[0017] The CPU 311, ROM 312, RAM 313, HDD 314, and network I / F 315 of the generation AI server 30000 in Figure 3 are the same as those of the CPU 111, ROM 112, RAM 113, HDD 114, and network I / F 115 of the computer 1000. Therefore, in this embodiment, the hardware configuration of the generation AI server 30000 will not be described. It is preferable that the generation AI server 30000 has a GPU 316.

[0018] The printer 2000 in Figure 4 includes a control unit 210, an operation unit 220, a printer 221, a scanner 222, and an authentication device 223. The control unit 210 controls the overall operation of the printer 2000. The CPU 211 reads the control program stored in the ROM 212 and executes and controls various functions of the printer 2000, such as scanning, printing, and communication. The RAM 213 is used as the main memory and temporary storage area for the CPU 211, such as the work area. In this embodiment, one CPU 211 uses one memory (RAM 213 or HDD 214) to execute each process shown in the flowchart described later, but this is not limited to this. For example, multiple CPUs or multiple RAMs or HDDs may work together to execute each process. The HDD 214 is a large-capacity storage unit that stores image data and various programs. The operation unit I / F 215 is an interface that connects the operation unit 220 and the control unit 210. The control unit 220 is equipped with a touch panel and keyboard, and accepts user operation / input / instructions. The printer I / F 216 is an interface that connects the printer 221 and the control unit 210. Image data for printing is transferred from the control unit 210 to the printer 221 via the printer I / F 216 and printed on a recording medium such as paper. The scanner I / F 217 is an interface that connects the scanner 222 and the control unit 210. The scanner 222 reads a document placed on a document glass (not shown) or an ADF (Auto Document Feeder) to generate image data and inputs it to the control unit 210 via the scanner I / F 217. The printer 2000 can print (copy) the image data generated by the scanner 222 from the printer 221, as well as send it as a file or via email. The authentication device I / F 218 is an interface that connects the authentication device 223 and the control unit 210. Authentication devices 223 include, for example, a card reader that reads IC cards or a fingerprint authentication device that reads fingerprints, and are used when a user authenticates to use the printer. Network I / F 219 is an interface that connects the control unit 210 (printer 2000) to the LAN.The printer 2000 uses the network interface 219 to send image data and information to and receive various types of information from various services on the internet.

[0019] <Software Configuration> Referring to Figure 5, the software configuration of each device included in the printing system according to this embodiment will be described.

[0020] As shown in Figure 5, the computer 1000 has the print application 1100 installed. The print application 1100 is an application that instructs the printer 2000 to print, and it displays a preview of the image to be printed, accepts print settings, and sends the print job to the printer 2000. In other words, it can display images and accept settings. The print application 1100 may be a standalone application or an application integrated with the printer driver built into the OS.

[0021] The print application 1100 includes a request control unit 1101, a chat control unit 1102, a data management unit 1103, and a print job control unit 1104. The request control unit 1101 is in standby mode to receive requests from the computer 1000, printer 2000, and generation AI server 30000, and in response to the requests, it causes each part of the print application 1100 to execute processing. The chat control unit 1102 sends prompts (instructions) and print data entered by the user to the generation AI server 30000. The chat control unit 1102 also displays the data received from the generation AI server 30000. The data management unit 1103 manages the data used by the print application 1100. The data managed includes, for example, application settings used by the print application 1100, print data used by the print application 1100, and information entered into the chat control unit 1102. The print job control unit 1104 converts the print data generated by the print application 1100 and the generation AI server 30000 into a format that can be printed by the printer 2000, and sends it to the printer 2000 as a print job. Alternatively, the print data may be in a format that can be directly printed by the printer 2000 without conversion by the print job control unit 1104.

[0022] The printer 2000 has a request control unit 2101, a print control unit 2102, and a chat control unit 2103. The request control unit 2101 is in standby mode to receive requests from the computer 1000, the printer 2000, and the generation AI server 30000. The print control unit 2102 executes print jobs received from the computer 1000. Received print jobs may be executed immediately, or they may be held until a print command is input from the user via the operation unit 220 before printing. The chat control unit 2103 sends prompts and print data entered by the user to the generation AI server 30000. The chat control unit 2103 also displays data received from the generation AI server 30000.

[0023] The generation AI server 30000 has a generation AI service 3100. The generation AI service 3100 has a request control unit 3101, a chat control unit 3102, a data management unit 3103, and a layout conversion unit 3104. The request control unit 3101, which has the function of interpreting generation AI and the function of changing the layout of print data, is in standby mode to receive requests from the computer 1000, printer 2000, and generation AI server 30000. Then, it causes each part of the generation AI service 3100 to execute processing in response to the request. The chat control unit 3102 has the function of interpreting generation AI. The chat control unit 3102 interprets the received prompt written in natural language and print data, and determines the response and layout conversion processing for the print data. The layout conversion unit 3104 receives and executes the layout conversion processing interpreted by the chat control unit 3102. In this embodiment, the chat control unit 3102 and the layout conversion unit 3104 are described as separate processing units, but they may be processed together as a single conversion processing unit. The data management unit 3103 stores and manages received prompts, received print data, and converted print data as history, linked to information that identifies the user who issued the command. An example of the history data stored here will be described later in Figure 8.

[0024] <Example of a print application screen> An example of the print application screen 100 is described with reference to Figure 6. The print application screen 100 has a preview area 101, a chat area 110, and other objects. The print application screen 100 is also displayed on the display unit 116 of the computer 1000.

[0025] The preview area 101 is an area that displays an image of the image data that the printing application 1100 sends to the printer 2000 as a preview image. This allows the user to check the printed material before it is output by the printer 2000. The chat area 110 accepts and displays prompts from the user using natural language. In addition, the preview area 101 and the chat area 110 are updated each time a response is received from the generation AI server 30000.

[0026] In this embodiment, first, a print preview of image data entered by the user or data such as a website specified by the user is displayed in the preview area 101. In the chat area 110, prompts for editing the print data in natural language from the user are received. The preview area 101 updates its display with the received print data each time it receives print data generated by the AI ​​server 3000 based on the entered prompts and the original image data.

[0027] The preview area 101 includes the preview page count 102, page turning buttons 103, the print header area 104, the toggle button 105, and the header / footer setting button 106. The preview page count 102 includes the total number of pages when the currently displayed print data (image data) is printed, and the page number of the page currently displayed in the preview area 101. The page turning buttons 103 return to the previous page or advance to the next page to change the preview displayed in the preview area. The print header area 104 displays a preview of the header to be printed. This header may be included in the print data, or it may simply be a header displayed as a preview. In other words, the information included in the print header area 104 may or may not be printed. The header may include the print date and time, the name of the file to be printed, and the URL from which the image data to be printed was obtained. Also, if the print data is generated by the generation AI service 3100 and that print data is displayed in the preview area 101, it includes information indicating that it was generated by the generation AI. For example, text such as "This was edited and printed using AI processing" or "This is an image generated by generation AI" may be displayed and printed in the print header area 104. Information indicating that it was generated by generation AI may be displayed and printed not in the print header area 104, but in the print footer area or other margins, or included within the image data itself. When the preview conversion toggle button 105 is pressed, it switches the display of the preview image in the preview area between the current preview image and the unedited preview image received when the print application 1100 was launched. At the same time, it also switches the display of the preview page number 102. At this time, it is indicated on the print application screen whether it is the first preview or the latest preview so that the user can see. Alternatively, pressing the preview conversion toggle button 105 may switch between a screen that displays only the latest preview and a screen that displays the unedited preview and the latest preview side by side.

[0028] The chat area 110 includes a chat input area 121. The chat input area 121 receives instructions from the user to convert print data using natural language input. The print application 1100 sends the input print data conversion instructions along with the print data to the generation AI service 3100. The print application 1100 displays the converted string received from the generation AI service 3100 as a chat response string in the chat area 110. In Figure 6, the instruction "Please delete the image in the sidebar" is entered in Japanese into the chat area 110, and the converted result indicating that the image has been successfully deleted is displayed as a response.

[0029] The print application screen 100 includes, as other objects, a print execution button 120, a history display button 130, and a settings screen display button 131. The print execution button 120 is a button that starts the process of sending a print job to be printed by the printer 2000 in the print application 1100. When the print execution button 120 is selected, a print job containing the image data of the image displayed in the preview area 101 at the time of selection is sent to the printer 2000. The history display button 130 is a button for displaying a UI that shows the conversion history of past print data. The screen for the conversion history of past print data is shown in Figure 7. The settings screen display button 131 is a button that, when pressed, displays the settings screen of the print application 1100 (not shown). This settings screen includes, for example, a printer selection object for selecting the printer to print to, a copy object for specifying the number of copies, and a color setting object for specifying the print colors. The settings screen also includes settings related to the conversion and generation of print data by the generation AI server 30000. For example, this includes enabling / disabling the history management of conversion (generation) results, enabling / disabling the process of embedding the prompts used for conversion into print data, and enabling / disabling the process of the print application 1100 automatically performing the conversion process using the embedded prompts.

[0030] <Example of print application history screen> The print application 1100 displays past history data in order to select previously entered prompts and apply them to the current print job. Referring to Figure 7, the print application history screen 200 will be described as an example of the print application 1100 according to this embodiment displaying past history on the computer 1000 or printer 2000.

[0031] Objects similar to those on the print application screen 100 will not be explained. When the print application 1100 detects that the history display button 130 has been pressed, it retrieves history data from the generation AI service 3100 and displays the history area 201. In Figure 7, the history area 201 is shown to the right of the chat, but it may be in a different location or displayed as a separate window.

[0032] The history area 201 includes a history selection area 202 and a history prompt apply button 206. Since there is a history selection area 202 for each history data, the history area 201 contains multiple history selection areas 202. The history selection area 202 includes a history prompt 203, a history preview 204, and a history selection box 205. The history selection area 202 may also include the conversion date and time of the history data, and the name of the print target such as the file name or URL. The history prompt 203 is the prompt used to convert the print data (a prompt previously entered by the user in the chat input area 121). The history preview 204 displays the preview before and after conversion by the history prompt 203 side by side. The history preview 204 may also include the change in the number of pages before and after conversion. Furthermore, the pages displayed in the history preview 204 may be set to show the pages with the largest difference in changes as a result of conversion by the history prompt. By displaying the history prompts 203 and history previews 204 together in the history selection box, users can review previously used prompts and the image generation results from those prompts, making it easier to select the desired prompt. The history selection box 205 is a selection box for selecting the history prompts to apply to the current print data. Figure 7 shows the state where two history prompts, "Delete image" and "Enlarge text for easier reading," are selected. The history prompt application button 206, when pressed, sends the currently selected history prompts and the current print data to the generation AI service 3100. The generation AI service 3100 applies the prompts to the print data sequentially based on the selection of the history prompt application button 206. The print application 1100 receives the conversion results with all conversions applied and reflects the conversion results in the preview area 101 and the chat area 110.

[0033] In other words, when the history selection box 205 is selected and the history prompt application button 206 is selected, the print application 1100 sends the prompts included in the selected history to the generation AI server 30000. Furthermore, it also sends the image data of the image currently displayed in the preview area 101 to the generation AI server 3000. As a result, the generation AI server 30000 generates (converts) image data based on the received prompts and image data. When the print application 1100 receives the image data generated by the generation AI server 3000, it displays the image of the received image data in the preview area 101. Then, when the user selects the print execution button 120, the image data received from the generation AI server 3000 is sent to the printer 2000. What is sent at this time is a print job that includes the above image data and the print settings set by the user in the print application 1100. The printer 2000 that receives the print job then executes the received print job and prints the image on paper.

[0034] In other words, the printing application 1100 can display a history that includes images generated based on prompts entered by the user and the images of the first image data entered.

[0035] Furthermore, the printing application 1100 can also display an image generated based on a prompt included in the selected history and a second image data entered by the user, which is different from the first image data previously entered.

[0036] Furthermore, the printing application 1100 can send the prompts entered by the user and the first image data entered by the user to the generation AI server 30000. The printing application 1100 can then receive the image data of the image generated based on the transmitted prompts and the first image data. By performing this process, the combination of the transmitted prompts and the image data generated based on those prompts can be stored as history data. The user can view this history data in the history area 201. In other words, the history, including the input prompts and the image data received from the generation AI server 30000, is displayed in the history area.

[0037] Furthermore, the print application 1100 can send the prompt included in the selected history from the displayed history, along with the input second image data, to the generation AI server 30000. The print application 1100 then receives the image data of the image generated based on the selected prompt included in the history and the second image data from the generation AI server 30000. The print application 1100 then displays the received image data in the preview area 101. Also, as shown in the screen of Figure 7, the history list is selected while the second image data is displayed.

[0038] Furthermore, the print application 1100 can accept print settings such as color mode, duplex printing, and paper size. When the print execution button 120 is selected, the print application 1100 sends the print settings specified by the user and the image data of the image displayed in the preview area 101 to the printer 2000.

[0039] Furthermore, after the history is displayed, the printing application 1100 does not accept input from the user for the prompts displayed in the history, but instead sends the prompts included in the selected history and the second image data to the generation AI server 3000. The generation AI server 3000 then generates image data based on the prompts and the second image data. The printing application 1100 receives this image data and displays the image data in the preview area 101.

[0040] In addition, the preview area 101 displays an image generated based on the prompts included in the selected history and the image of the second input image data, along with information indicating that the image was generated by the generation AI.

[0041] The operation of the above-mentioned printing application 1100 is achieved by the CPU 111 of the computer 1000 executing the printing application 1100.

[0042] <Historical Data Structure> Referring to Figure 8, the structure of the history data stored in the data management unit 3103 of the generation AI service 3100 is shown. The table of the history data DB shown in Figure 11 includes prompt 301, pre-conversion data 302, post-conversion data 303, job identifier 304, and date and time 305 as items of history data. Each row in the history data DB 300 represents one piece of history data. Prompt 301 is the prompt received by the generation AI service 3100 as a conversion request. Pre-conversion data 302 is the print data before conversion is performed by the prompt. Post-conversion data 303 is the print data after conversion is performed by the prompt. The pre-conversion and post-conversion data may not be directly stored as print data within the history data DB 300, but rather as files in external storage, and the history data DB 300 may store information to identify the files in the external storage. The job identifier 304 stores, for example, the file name or URL of the file to be printed. The date and time 305 is the date and time when conversion was performed by the prompt. The history data DB300 is stored on the HDD314 of computer 1000.

[0043] <Example of a screen showing the use of file metadata in a printing application> The print application 1100 utilizes file metadata to eliminate the need to repeatedly enter the same information when printing a file. Referring to Figure 9, the print application metadata setting screen 300-A and the print application metadata application screen 300-B, which applies file metadata as a prompt, will be described. Objects similar to those in the print application screen 100 will not be described.

[0044] First, the print application metadata settings screen 300-A will be explained. After the print application 1100 starts printing when the print execution button 120 is pressed, it displays a metadata embedding selection area 901 in the chat area 110 to ask whether to embed prompts for the instructions used for this conversion into the file to be printed. The metadata embedding selection area 901 may include objects that the user can choose whether or not to embed, in order to make it easier for the user to respond. Alternatively, the user may respond by natural language input in the chat input area 121. The user may also input and respond in a way that embeds some of the prompts that have been entered, or new prompts that have not been entered. When the print application 1100 detects a metadata embedding instruction, it embeds the conversion prompts into the metadata of the print file. After that, it displays a metadata embedding result area 902 in the chat area 110.

[0045] An example of reading and using metadata embedded in the print application metadata settings screen 300-A is explained using the print application metadata application screen 300-B, which applies file metadata as a prompt. When the print application 1100 reads print data, it checks whether the metadata contains a prompt, and if it does, it displays the metadata application display area 903. The metadata application display area 903 displays the content of the embedded prompt and also indicates that the conversion based on that prompt is in progress. Once the conversion is complete, the print application 1100 reflects the conversion results in the preview area 101 and the chat area 110.

[0046] <Processing flow> Referring to Figure 10, the sequence of layout conversion processing in the generation AI service 3100 of this embodiment will be described. The sequence of processing that takes place between the computer 1000, the printer 2000, and each element constituting the generation AI service 3100 will be described. The numbers following "S" below indicate the step numbers in the sequence.

[0047] The request control unit 1101 of the print application 1100 detects a print request from the computer 1000 (S101). The print request includes the print data to be printed.

[0048] The request control unit 1101 displays the received print data as a preview using the preview control unit 1105 (S102).

[0049] The chat control unit 1102 of the print application 1100 detects that a conversion request has been entered by the user (S103) and sends the conversion request to the request control unit 1101 (S104). The request control unit 1101 sends the conversion request to the request control unit 3101 of the generation AI service 3100 (S105). The conversion request includes the prompt entered by the user in the chat input area 121 and the print data. The request control unit 3101 sends the conversion request to the chat control unit 3102 (S106). The chat control unit 3102 interprets the received prompt written in natural language and the print data and determines the layout conversion process for the print data (S107). The chat control unit 3102 sends a request for layout conversion processing to the layout conversion unit 3104 (S108). The layout conversion unit 3104 executes the layout conversion process included in the layout conversion request on the print data included in the layout conversion request (S109). The layout conversion unit 3104 returns the conversion result to the chat control unit 3102 (S110). The chat control unit 3102 saves the conversion result as history data to the data management unit 3103. An example of the saved history data is shown in Figure 8. The chat control unit 3102 generates conversion result text indicating the success or failure of the conversion from the conversion result received in S107 (S111). The chat control unit 3102 returns the conversion request result to the request control unit 3101 (S113). The request control unit 3101 returns the conversion request result to the request control unit 1101 of the print application 1100. The conversion request result includes the converted print data returned from the layout conversion unit 3104 in S110 and the conversion result text generated in S112.

[0050] The request control unit 1101 previews the print data included in the received conversion request result using the preview control unit 1105 (S115). The request control unit 1101 displays the conversion result text included in the received conversion request result in the chat area using the chat control unit 1102 (S116). The request control unit 1101 saves the received conversion result to the data management unit 1103 of the print application 1100 (S117). Subsequently, by repeating steps S103 to S117 any number of times, print data with the user's desired layout can be generated. In this embodiment, print data is described as being sent as a conversion request each time, but to reduce the amount of communication, the print data sent in S105 may be saved to the data management unit 3103 of the generation AI service 3100. Furthermore, for the second and subsequent conversion requests, the print data may not be included, and the latest converted print data saved in the data management unit 3103 may be targeted for the conversion process in S109. When the print data expected by the user is ready, the request control unit 1101 detects the print command and sends it to the request control unit 2101 of the printer 2000 (S118). The request control unit 2101 then executes the print at the print control unit 2102 (S119). The request control unit 1101 of the print application 1100 discards the conversion results that are no longer needed from the data management unit 1103 (S120).

[0051] Referring to Figure 11, the sequence for displaying the history and applying the history prompt to the current print data in the print application 1100 of this embodiment will be described. The sequence of processing performed between the print application 1100 and the generation AI service 3100 will be described. The numbers following "S" below indicate the step numbers in the sequence.

[0052] The print application 1100 detects the press of the history display button (S201). The print application 1100 sends a history acquisition request to the generation AI service 3100 (S202). The generation AI service 3100 acquires history data from the data management unit 3103 (S203). The history data acquired here is the history data of the user who sent the acquisition request. The generation AI service 3100 returns the history data acquisition request result to the print application 1100 (S204). The history data acquisition request result includes multiple history prompts and their associated pre-conversion and post-conversion data, date and time information, and a job identifier. The print application 1100 displays the history prompts included in the history data received in S204 and images of the print data before and after the change as a set (S205). If there are multiple histories, each is set up and multiple histories are displayed. The display may also include a display of the change in the number of pages before and after the change. In addition, the images of the print data before and after the change that are displayed may be set to show the pages with the largest difference in change as a result of conversion by the history prompt. This allows users to view historical data, prompts, and images before and after conversion, and select the prompt they want to apply to the current print data.

[0053] The print application 1100 detects the selection of history data (S206). This selection may be made for multiple history data. If multiple selections are made, the selection order is stored. The UI of the print application 1100 may display selection numbers to indicate the selection order. The print application 1100 detects the press of the Apply History Data button (S207). The print application 1100 sends a request to apply the selected history data to the generation AI service 3100 (S208). The request to apply history data includes the prompts for the selected history data, their selection order, and the image data currently displayed in the preview area 101. The generation AI service 3100 interprets the received history data prompts in order according to the selection order and performs layout conversion processing (image generation) on the print data (S209). The generation AI service 3100 generates text indicating the result of applying the history prompts (S210). The generation AI service 3100 sends the result of the history data application request to the print application 1100 (S211). The results of this application include the converted print data generated in S209 and S210, and text indicating the conversion results.

[0054] Referring to Figure 12, the process by which the print application 1100 automatically applies a prompt if a prompt is embedded as metadata in the file to be printed, and the process of suggesting a prompt suitable for the print data to the user using history data are shown.

[0055] The print application 1100 receives a print request from the computer 1000 (401). The print application 1100 checks whether the print data included in the received print request is a file and whether a prompt is included as metadata for that file (402). To determine if the metadata indicates a prompt, it checks whether the metadata name or the metadata itself contains an identification string defined by the print application 1100. If the metadata contains a prompt, it sends the current print data and the prompt to the generation AI service 3100 to execute the prompt immediately (418). If it does not contain a prompt, it sends only the print data to the generation AI service 3100 (403). The print application 1100 checks whether it receives a candidate prompt from the generation AI service 3100 (404). If it does not receive a candidate, it assumes there is no candidate and checks whether to wait for and detect prompt input from the user (417). If it receives a candidate, it assumes there is a candidate and displays the candidate prompt received in 404 as an object that the user can select in the chat area (405). The print application 1100 checks if it has received conversion data from the candidate prompt (406). If it has not received it, it proceeds to flow 409 without doing anything. If it has received it, it detects whether the object displayed in 405 is selected (407). For example, it checks if the object with the candidate prompt displayed has detected an action such as mouseover or long press. If detected, it temporarily displays the conversion data received in 406 in the preview area 101 while detection is occurring (408). When detection stops, for example, when the mouse moves away from the object with the candidate prompt displayed, the data displayed in the preview area 101 is returned to the original print data. This allows the user to easily check the conversion result before selecting a candidate prompt and confirming the conversion. If detection has not occurred in 407, the print application 1100 proceeds to flow 409 without doing anything even if the mouse is hovered over. The print application 1100 detects that the object with the candidate prompt has been selected and an execution command has been issued (409).If no candidate is detected, the print application 1100 assumes that no candidate was selected and waits for prompt input from the user to check for detection (417). If a candidate is detected, the print application 1100 sends the selected candidate prompt to the generation AI service 3100 (410). The print application 1100 checks if it has received the conversion result of the candidate prompt sent in 410 (410). If it has not yet received it, the print application 1100 waits until it receives the conversion result (412). If it has received it, the print application 1100 updates the print data using the received conversion result (413). The print application 1100 updates the preview area 101 with the print data updated in 413 (414). The print application 1100 displays the conversion result text received in 412 in the chat area 110 (415). The print application 1100 stores the transmitted prompt, the original print data, and the current print data (after conversion) in the data management unit 1103 (416). This enables the print application 1100 to automatically apply prompts if they are embedded as metadata in the file to be printed, and to suggest prompts suitable for the print data to the user using historical data. The print application 1100 can continue accepting prompt input from the user and continue the conversion process. The print application 1100 detects the instruction to send a prompt (417), sends the current print data and prompt to the generation AI service 3100 (418), and terminates the processing in this flowchart.

[0056] Referring to Figure 13, the process by which the generation AI service 3100 proposes a prompt suitable for the received print data to the user using history data is shown. The generation AI service 3100 receives print data from the print application 1100 (501). The generation AI service 3100 checks whether there is any history data that matches the job identifier contained in the print data received in 501, or similar history data that matches the job identifier 304 of the history data stored in the data management unit 3103 (502). If none exists, the generation AI service 3100 does not generate a candidate prompt and proceeds to the prompt reception process from the print application 1100 (509). If it does exist, the generation AI service 3100 sends the prompt 301 of the matching or similar history data as a candidate prompt to the print application 1100 (503). At this time, if there are multiple candidate prompts, multiple prompts may be sent. The generation AI service 3100 starts the layout conversion using the candidate prompt in advance, before receiving an execution instruction for the candidate prompt from the print application 1100 (504). If there are multiple candidate prompts, the generation AI service 3100 starts the layout conversion using all of the candidate prompts. Once the conversion is complete, the generation AI service 3100 sends the converted print data to the print application 1100 (505). By starting the conversion process for candidate prompts in advance in this way, the waiting time for users of the print application 1100 who apply a candidate prompt can be reduced. In 503, the generation AI service 3100 checks whether it has received the candidate prompts that have been sent (506). If it has received them, the generation AI service 3100 saves the history data to the data management unit 3103 (507) and deletes the print data that has been converted by the candidate prompts as it is no longer needed (508). If it has not received them, the generation AI service 3100 similarly deletes the converted print data as if it had not been selected (508). The generation AI service 3100 receives the print data and prompts from the print application 1100 (509). In 509, the generation AI service 3100 interprets the received prompts and performs the conversion process on the print data (510).The generation AI service 3100 sends the conversion result to the printing application (511), and the processing of this flowchart ends.

[0057] Referring to Figure 14, the process by which the print application 1100 adds a print footer and prints, and the process of embedding a prompt in the file metadata are shown. The print application 1100 detects a print command by pressing the print button (601). The print application 1100 checks whether the AI-enabled printing setting is enabled in the settings of the print application 1100 (602). If it is enabled, it adds a string to the print job header indicating that it has been edited using AI. The string may be, for example, "This has been edited and printed using AI processing." This allows the user viewing the printout to recognize that the file or website has not been printed as is. The string indicating that the printout has been edited using AI may be previewed and printed to be included in the print footer, other margins of the printout, or within the print body, rather than in the print header area. The print application 1100 creates a print job from the current print data and sends the print job to the printer 2000 (604). The print application 1100 checks if the print target is a "file" (605). If it is not a file, it terminates this flow. If it is a file, the print application 1100 checks if the setting to embed prompts in files is enabled in the print application 1100's settings (606). If it is not enabled, it terminates this flow. If it is enabled, the print application 1100 displays in the chat area whether to embed the prompt used for the conversion as file metadata (609). For example, it displays the string "Do you want to embed "Delete image" in "news.docs" so that it will be automatically applied next time?" along with a button control that the user can press to indicate "Yes" or "No". The print application 1100 checks if it detects an instruction to embed metadata (610). If it does not detect an instruction, it terminates this flow. If it detects an instruction, the print application 1100 embeds the prompt in the file metadata (611). The embedded metadata includes the prompt itself, along with an identifier to indicate that the metadata is a prompt handled by the print application 1100.For example, an identifier may be used as the key name of the metadata, or the identifier may be included in the metadata itself. The print application 1100 displays the embedded result in the chat area (612) and terminates this flow. As a result, the prompt will be automatically applied the next time the file is printed with the print application 1100.

[0058] Referring to Figure 15, the process by which the print application 1100 switches between pre-conversion and post-conversion previews is shown. The print application 1100 detects when the pre-conversion preview toggle button is pressed (701). The print application 1100 checks whether the currently displayed preview is the post-conversion print data or the initial pre-conversion print data (702). If it is the post-conversion data, the print application 1100 changes the display to preview the initial pre-conversion print data (703) and terminates this flow. At the same time, the display of the preview page count also switches back to the number of pages before the change. If it is the pre-conversion data, the print application 1100 changes the display to preview the latest post-conversion print data (704) and terminates this flow. At the same time, the display of the preview page count also switches back to the number of pages before the change. Alternatively, pressing the pre-conversion preview toggle button 105 may switch between a screen that displays only the latest preview and a screen that displays the unedited preview and the latest preview side by side.

[0059] As described above, in this embodiment, the printing system uses a generation AI service 3100 to perform layout conversion based on natural language prompts entered by the user into the printing application 1100, and updates the print preview. The generation AI service 3100 saves the layout conversion history along with the prompts and the print data before and after the conversion, and displays it as history in the printing application 1100.

[0060] The image data (print data) conversion process and layout conversion process described above may also refer to the process of generating new image data based on the original image data and prompts.

[0061] In the above embodiment, the printing application 1100 is described as running on the computer 1000, but it may also be run on the printer 2000, and the conversion process via chat may be performed on the printer 2000. In this case, the process of opening the initial printing application involves selecting a print job from the print job list of held print jobs or the print job history screen of the printer 2000, and starting the printing application with that print job as the print data.

[0062] <Other Embodiments> The present invention provides a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium. It can also be implemented by a process in which one or more processors in the computer of the system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0063] 1000 computers 2000 Printer 3000 Generating AI Servers 1100 Printing Applications

Claims

1. A history display step that displays a history including an image generated based on a prompt entered by the user and an image of a first image data entered, A program characterized by causing an information processing device to execute an image display step, which involves displaying an image generated based on a prompt included in the history selected from the displayed history and an image of a second image data input.

2. A reception process for receiving prompt input from the user, The information processing device is further made to perform a transmission step of sending the input prompt and the input first image data to the generation AI server, and a reception step of receiving the image data of the image generated based on the input prompt and the input first image data transmitted in the transmission step. The program according to claim 1, characterized in that the history display step displays a history including the input prompt and the image data received in the reception step.

3. In the transmission step, the prompt included in the history selected from the displayed history and the input second image data are transmitted to the generation AI server, and in the reception step, an image generated based on the prompt included in the selected history and the input second image data transmitted in the transmission step is received. The program according to claim 2, characterized in that an image generated based on the prompt included in the selected history and the input second image data received in the receiving step is displayed in the image display step.

4. The program according to claim 1, characterized in that, while the second image data is displayed, the history displayed in the history display step is selected.

5. The process for accepting print settings, The program according to claim 1, which causes the information processing device to further execute a transmission step of sending to a printing device the image data of the image to be displayed in the image display step, which is an image generated based on the prompt included in the selected history and the image of the second input image data, and the print settings.

6. The program according to claim 1, characterized in that, in the image display step, after the history is displayed in the history display step, an image generated based on the prompt included in the selected history and the input second image data is displayed without accepting input from the user for the prompt.

7. The program according to claim 1, characterized in that the history includes the image generated based on the input prompt and the input first image data image, and the input first image data image.

8. The program according to claim 1, characterized in that in the history display step, an image generated based on the prompts included in the history, the prompts entered by the user, and the image of the first image data entered is displayed.

9. The program according to claim 1, characterized in that the image display step displays the image generated based on the prompt included in the selected history and the image of the second input image data, and information indicating that the image was generated by the generation AI.

10. A history display means that displays a history including an image generated based on a prompt entered by the user and an image of a first image data entered, An information processing apparatus having an image display means for displaying an image generated based on a prompt included in the history selected from the displayed history and an image of a second image data input.

11. A reception means for receiving prompt input from the user, The system further comprises a transmission means for transmitting the input prompt and the input first image data to a generating AI server, and a receiving means for receiving image data of an image generated based on the input prompt and the input first image data transmitted by the transmission means. The information processing apparatus according to claim 10, characterized in that the history display means displays a history including the input prompt and an image of the image data received by the receiving means.

12. The transmitting means sends the prompt included in the history selected from the displayed history and the input second image data to the generating AI server, and the receiving means receives the image generated based on the prompt included in the selected history and the input second image data transmitted by the transmitting means. The information processing apparatus according to claim 11, characterized in that the image display means displays an image generated based on the prompt included in the selected history and the input second image data image received by the receiving means.

13. The information processing apparatus according to claim 10, characterized in that, while the second image data is displayed, the history displayed by the history display means is selected.

14. A setting acceptance method for accepting print settings, The information processing apparatus according to claim 10, further comprising a transmission step of transmitting to a printing device the image data of an image to be displayed by the image display means, which is generated based on the prompt included in the selected history and the image of the second input image data.

15. The information processing apparatus according to claim 10, characterized in that the image display means displays an image generated based on the prompt included in the selected history and the image of the second image data that was input, without accepting prompt input from the user after the history display means has displayed the history.

16. The information processing apparatus according to claim 10, characterized in that the history includes the image generated based on the input prompt and the input first image data image, and the input first image data image.

17. The information processing apparatus according to claim 10, characterized in that the history display means displays an image generated based on the prompts included in the history, the prompts entered by the user, and the image of the first image data that was entered.

18. The information processing apparatus according to claim 10, characterized in that the image display means displays the image generated based on the prompt included in the selected history and the image of the second input image data, and information indicating that the image was generated by the generation AI.

19. A history display step that displays a history including an image generated based on a prompt entered by the user and an image of a first image data entered, An information processing method comprising an image display step of displaying an image generated based on a prompt included in the history selected from the displayed history and an image of a second image data that has been input.

Images