Program, information processing device, information processing method

Abstract

When repeatedly editing an input image based on user prompts, the aim is to reduce the effort required to revert to an edited image at a certain point. [Solution] The program of the present invention displays a first image generated based on a first prompt input by the user and an image of the input image data in a first display area; while a second image generated based on a second prompt input by the user and the generated first image is displayed in the first display area, a history corresponding to the first image is displayed in a second display area; when a print instruction is received, print data including the image data of the image displayed in the first display area is sent to a printing device; and while the second image is displayed in the first display area, a history corresponding to the first image is selected from the displayed history, thereby displaying the first image in the first display area.

Description

Technical Field

[0001] The present invention relates to a program, an information processing apparatus, and an information processing method.

Background Art

[0002] In recent years, conversational AIs such as chatbots and generative AIs have been rapidly developed. Along with this development, various services using conversational AIs are being provided. Patent Document 1 shows a system that displays an image and edits the image based on a prompt input by a user.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When editing or generating an image to be printed and repeating the editing based on a prompt input by the user for the input image, there may be a time when one wants to return to the image edited at a certain point. For this purpose, it is troublesome to input the same prompt again.

[0005] The present invention has been made in view of the above problems, and an object thereof is to reduce the trouble of returning to the image edited at a certain point when repeating the editing based on a prompt input by the user for the input image.

Means for Solving the Problems

[0006] The present invention provides a program that causes an information processing device to perform the following steps: an image display step of displaying a first image generated based on a first prompt input by a user and an image of the input image data in a first display area; a history display step of displaying a history corresponding to the first image in a second display area while a second image generated based on a second prompt input by the user and the generated first image is displayed in the first display area in the image display step; and a print data transmission step of receiving a print instruction and transmitting print data including image data of the image displayed in the first display area to a printing device, wherein in the image display step, while the second image is displayed in the first display area, a history corresponding to the first image is selected from the displayed history, and the first image is displayed in the first display area. [Effects of the Invention]

[0007] When repeatedly editing an input image based on user prompts, it reduces the effort required to revert to a previous edited image at a certain point. [Brief explanation of the drawing]

[0008] [Figure 1] This diagram shows an example of the overall configuration of this system. [Figure 2] This diagram shows an example of the computer hardware configuration of this system. [Figure 3] This diagram shows an example of the hardware configuration of the AI ​​generation server for this system. [Figure 4] This diagram shows an example of the printer hardware configuration in this system. [Figure 5] This diagram shows an example of the software configuration of this system. [Figure 6] This diagram shows an example of the print application screen in this system. [Figure 7] This diagram shows an example of the print application history display screen in this system. [Figure 8]Figure showing an example of the structure of history data in this system [Figure 9] Figure showing an example of the settings screen of the printing application in this system [Figure 10] Sequence diagram showing an example of the process of image conversion and printing by chat [Figure 11] Sequence diagram showing an example of the display of history data and the application process of history data [Figure 12] Flowchart showing an example of the process of the printing application [Figure 13] Flowchart showing an example of the process of the generation AI server [Figure 14] Flowchart showing an example of the process of the printing application [Figure 15] Flowchart showing an example of the process of the printing application [Figure 16] Figure showing an example of the timeline display screen of the printing application [Figure 17] Figure showing an example of the structure of timeline data [Figure 18] Flowchart showing an example of the timeline display process of the printing application and the process of returning print data at a certain point

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and duplicate explanations are omitted.

[0010] <System Configuration> Hereinafter, first, referring to FIG. 1, the network configuration of the printing system according to this embodiment will be described.

[0011] As shown in FIG. 1, this printing system includes a computer 1000 which is a terminal device or an information processing device, a printer 2000 which is an example of a printing device, and a generation AI server 3000. For example, the computer 1000 and the printer 2000 are arranged in an office and are communicably connected to each other via an in-company network 4000. The in-company network 4000 is connected to an external Internet 5000 via a router (not shown). Here, the computer 1000 is an example of a user terminal or an information processing device, and the generation AI server 3000 is an example of a server device.

[0012] The generation AI server 3000 is communicably connected to the computer 1000 and the printer 2000 via the Internet 5000 and the in-company network 4000. Here, the generation AI is an AI that can newly generate contents such as text, images, videos, and voices, and can generate contents based on prompts and data input by a user. Also, the generation AI repeats learning using deep learning and outputs an answer based on the results of that learning.

[0013] <Hardware Configuration> Referring to FIGS. 2, 3, and 4, the hardware configurations of the devices of the printing system according to this embodiment will be described. FIG. 2 shows an example of the hardware configuration of the computer 1000. FIG. 3 shows an example of the hardware configuration of the generation AI server 3000. FIG. 4 shows an example of the hardware configuration of the printer 2000.

[0014] As shown in Figure 2, the computer 1000 includes a CPU 111, ROM 112, RAM 113, HDD 114, and network I / F 115. The computer 1000 also includes a display unit 116 and an operation unit 117. The CPU 111 controls the overall operation by reading control programs stored in the ROM 112 and executing various processes. The RAM 113 is used as the CPU 111's main memory and temporary storage area such as the work area. The HDD 114 is a large-capacity storage unit that stores image data and various programs. The network I / F 115 is an interface for connecting to the internet. It receives processing requests from other devices and services via the network I / F 115 and sends and receives various information.

[0015] The display unit 116 is a display device such as a monitor, and may be integrated with the computer 1000 or connected to the computer 1000 as an external device. The screen information output by the CPU 111 is displayed on the display unit 116.

[0016] The control unit 117 is an input device such as a keyboard or mouse, and may be integrated with the computer 1000 or connected to the computer 1000 as an external device. The CPU 111 receives various user operations via the control unit 117.

[0017] The CPU 311, ROM 312, RAM 313, HDD 314, and network I / F 315 of the generation AI server 3000 in Figure 3 are the same as those of the CPU 111, ROM 112, RAM 113, HDD 114, and network I / F 115 of the computer 1000. Therefore, in this embodiment, the hardware configuration of the generation AI server 3000 will not be described. It is preferable that the generation AI server 3000 has a GPU 316.

[0018] The printer 2000 in Figure 4 includes a control unit 210, an operation unit 220, a printer 221, a scanner 222, and an authentication device 223. The control unit 210 controls the overall operation of the printer 2000. The CPU 211 reads the control program stored in the ROM 212 and executes and controls various functions of the printer 2000, such as scanning, printing, and communication. The RAM 213 is used as the main memory and temporary storage area for the CPU 211, such as the work area. In this embodiment, one CPU 211 uses one memory (RAM 213 or HDD 214) to execute each process shown in the flowchart described later, but this is not limited to this. For example, multiple CPUs or multiple RAMs or HDDs may work together to execute each process. The HDD 214 is a large-capacity storage unit that stores image data and various programs. The operation unit I / F 215 is an interface that connects the operation unit 220 and the control unit 210. The control unit 220 is equipped with a touch panel and keyboard, and accepts user operation / input / instructions. The printer I / F 216 is an interface that connects the printer 221 and the control unit 210. Image data for printing is transferred from the control unit 210 to the printer 221 via the printer I / F 216 and printed on a recording medium such as paper. The scanner I / F 217 is an interface that connects the scanner 222 and the control unit 210. The scanner 222 reads a document placed on a document glass (not shown) or an ADF (Auto Document Feeder) to generate image data and inputs it to the control unit 210 via the scanner I / F 217. The printer 2000 can print (copy) the image data generated by the scanner 222 from the printer 221, as well as send it as a file or via email. The authentication device I / F 218 is an interface that connects the authentication device 223 and the control unit 210. Authentication devices 223 include, for example, a card reader that reads IC cards or a fingerprint authentication device that reads fingerprints, and are used when a user authenticates to use the printer. Network I / F 219 is an interface that connects the control unit 210 (printer 2000) to the LAN.The printer 2000 uses the network interface 219 to send image data and information to and receive various types of information from various services on the internet.

[0019] <Software Configuration> Referring to Figure 5, the software configuration of each device included in the printing system according to this embodiment will be described. As shown in Figure 5, the computer 1000 has the printing application 1100 installed. The printing application 1100 is an application for instructing the printer 2000 to print, and it is an application that displays a preview of the image to be printed, accepts print settings, and sends a print job to the printer 2000. In other words, it can display images and accept settings. The printing application 1100 may be an independent application or an application integrated with the printer driver built into the OS.

[0020] The print application 1100 includes a request control unit 1101, a chat control unit 1102, a data management unit 1103, and a print job control unit 1104. The request control unit 1101 is in standby mode to receive requests from the computer 1000, printer 2000, and generation AI server 3000, and in response to the requests, it causes each part of the print application 1100 to execute processing. The chat control unit 1102 sends prompts (instructions) and print data entered by the user to the generation AI server 3000. The chat control unit 1102 also displays the data received from the generation AI server 3000. The data management unit 1103 manages the data used by the print application 1100. The managed data, for example, includes prompts entered into the chat control unit 1102, print data before conversion sent to the generation AI service 3100, and converted print data received from the generation AI service 3100, which are stored and managed as timeline data. An example of the timeline data stored here will be described later in Figure 17. Furthermore, the application settings used by the print application 1100 are also saved. The print job control unit 1104 converts the print data generated by the print application 1100 and the generation AI server 3000 into a format that can be printed by the printer 2000 and sends it to the printer 2000 as a print job. Note that the print data may also be in a format that can be printed directly by the printer 2000 without conversion by the print job control unit 1104.

[0021] The printer 2000 has a request control unit 2101, a print control unit 2102, and a chat control unit 2103. The request control unit 2101 is in standby mode to receive requests from the computer 1000, the printer 2000, and the generation AI server 3000. The print control unit 2102 executes print jobs received from the computer 1000. Received print jobs may be executed immediately, or they may be held until a print command is input from the user via the operation unit 220 before printing. The chat control unit 2103 sends prompts and print data entered by the user to the generation AI server 3000. The chat control unit 2103 also displays data received from the generation AI server 3000.

[0022] The generation AI server 3000 has a generation AI service 3100. The generation AI service 3100 is a server that has a generation AI interpretation processing function and a print data layout modification function, and has a request control unit 3101, a chat control unit 3102, a data management unit 3103, and a layout conversion unit 3104. The request control unit 3101 is in standby mode to receive requests from the computer 1000, printer 2000, and generation AI server 3000, and in response to the request, it causes each part of the generation AI service 3100 to execute processing. The chat control unit 3102 has a generation AI interpretation processing function. The chat control unit 3102 interprets the received prompts written in natural language and print data, and determines the response and layout conversion processing for the print data. The layout conversion unit 3104 receives and executes the layout conversion processing interpreted by the chat control unit 3102. In this embodiment, the chat control unit 3102 and the layout conversion unit 3104 are described as separate processing units, but they may be processed together as a single conversion processing unit. The data management unit 3103 stores and manages received prompts, received print data, and converted print data as history, linked to information that identifies the user who issued the command. An example of the history data stored here will be described later in Figure 8.

[0023] <Example of a print application screen> An example of the print application screen 100 is described with reference to Figure 6. The print application screen 100 has a preview area 101, a chat area 110, and other objects. The print application screen 100 is also displayed on the display unit 116 of the computer 1000.

[0024] The preview area 101 is an area that displays an image of the image data that the printing application 1100 sends to the printer 2000 as a preview image. This allows the user to check the printed material before it is output by the printer 2000. The chat area 110 accepts and displays prompts from the user using natural language. In addition, the preview area 101 and the chat area 110 update their displays each time they receive a response from the generation AI server 3000.

[0025] In this embodiment, first, a print preview of image data entered by the user or image data entered from a website specified by the user is displayed in the preview area 101. In the chat area 110, prompts for editing the print data in natural language from the user are received. The preview area 101 updates its display with the received print data each time it receives print data generated by the generation AI server 3000 based on the entered prompts and the original image data.

[0026] The preview area 101 includes the preview page count 102, page turning buttons 103, the print header area 104, the toggle button 105, and the header / footer setting button 106. The preview page count 102 includes the total number of pages when the currently displayed print data (image data) is printed, and the page number of the page currently displayed in the preview area 101. The page turning buttons 103 return to the previous page or advance to the next page to change the preview displayed in the preview area. The print header area 104 displays a preview of the header to be printed. This header may be included in the print data, or it may simply be a header displayed as a preview. In other words, the information included in the print header area 104 may or may not be printed. The header may include the print date and time, the name of the file to be printed, and the URL from which the image data to be printed was obtained. Also, if the print data is generated by the generation AI service 3100 and that print data is displayed in the preview area 101, it includes information indicating that it was generated by the generation AI. For example, text such as "This was edited and printed using AI processing" or "This is an image generated by generation AI" may be displayed and printed in the print header area 104. Information indicating that it was generated by generation AI may be displayed and printed not in the print header area 104, but in the print footer area or other margins, or included within the image data itself. When the preview conversion toggle button 105 is pressed, it switches the display of the preview image in the preview area between the current preview image and the unedited preview image received when the print application 1100 was launched. At the same time, it also switches the display of the preview page number 102. At this time, it is indicated on the print application screen whether it is the first preview or the latest preview so that the user can see. Alternatively, pressing the preview conversion toggle button 105 may switch between a screen that displays only the latest preview and a screen that displays the unedited preview and the latest preview side by side.

[0027] The chat area 110 includes a chat input area 121. The chat input area 121 receives instructions from the user to convert print data using natural language input. The print application 1100 sends the input print data conversion instructions along with the print data to the generation AI service 3100. The print application 1100 displays the converted string received from the generation AI service 3100 as a chat response string in the chat area 110. In Figure 6, the instruction "Please delete the image in the sidebar" is entered in Japanese into the chat area 110, and the converted result indicating that the image has been successfully deleted is displayed as a response.

[0028] The print application screen 100 includes, as other objects, a print execution button 120, a history display button 130, a settings screen display button 131, and a timeline display button 132. The print execution button 120 is a button that starts the process of sending a print job to be printed by the printer 2000 in the print application 1100. When the print execution button 120 is selected, a print job containing the image data of the image displayed in the preview area 101 at the time of selection is sent to the printer 2000. The history display button 130 is a button for displaying a UI that shows the conversion history of past print data. The screen for the conversion history of past print data is shown in Figure 7. The settings screen display button 131 is a button that, when pressed, displays the settings screen of the print application 1100 (not shown). This settings screen includes, for example, a printer selection object for selecting the printer to print to, a copy object for specifying the number of copies, and a color setting object for specifying the print colors. The settings screen also includes settings related to the conversion and generation of print data by the generation AI server 3000. For example, this includes enabling / disabling the history management of conversion (generation) results, enabling / disabling the process of embedding the prompts used for conversion into print data, and enabling / disabling the process of the print application 1100 automatically performing the conversion process using the embedded prompts.

[0029] The timeline display button 132 is a button that, when pressed, displays the conversion history of the current print data in a timeline format on the UI. The timeline screen is shown in Figure 16.

[0030] <Example of print application history screen> The print application 1100 displays past history data in order to select previously entered prompts and apply them to the current print job. Referring to Figure 7, the print application history screen 200 will be described as an example of the print application 1100 according to this embodiment displaying past history on the computer 1000 or printer 2000.

[0031] Objects similar to those on the print application screen 100 will not be explained. When the print application 1100 detects that the history display button 130 has been pressed, it retrieves history data from the generation AI service 3100 and displays the history area 201. In Figure 7, the history area 201 is shown to the right of the chat, but it may be in a different location or displayed as a separate window.

[0032] The history area 201 includes a history selection area 202 and a history prompt apply button 206. Since there is a history selection area 202 for each history data, the history area 201 contains multiple history selection areas 202. The history selection area 202 includes a history prompt 203, a history preview 204, and a history selection box 205. The history selection area 202 may also include the conversion date and time of the history data, and the name of the print target such as the file name or URL. The history prompt 203 is the prompt used to convert the print data (a prompt previously entered by the user in the chat input area 121). The history preview 204 displays the preview before and after conversion by the history prompt 203 side by side. The history preview 204 may also include the change in the number of pages before and after conversion. Furthermore, the pages displayed in the history preview 204 may be set to show the pages with the largest difference in changes as a result of conversion by the history prompt. By displaying the history prompts 203 and history previews 204 together in the history selection box, users can review previously used prompts and the image generation results from those prompts, making it easier to select the desired prompt. The history selection box 205 is a selection box for selecting the history prompts to apply to the current print data. Figure 7 shows the state where two history prompts, "Delete image" and "Enlarge text for easier reading," are selected. The history prompt application button 206, when pressed, sends the currently selected history prompts and the current print data to the generation AI service 3100. The generation AI service 3100 applies the prompts to the print data sequentially based on the selection of the history prompt application button 206. The print application 1100 receives the conversion results with all conversions applied and reflects the conversion results in the preview area 101 and the chat area 110.

[0033] In other words, when the history prompt application button 206 is selected, the print application 1100 sends the prompts included in the selected history and the image data of the image displayed in the preview area 101 to the generation AI server 300. As a result, the generation AI server 3000 generates (converts) image data based on the received prompts and image data. When the print application 1100 receives the image data generated by the generation AI server 300, it displays the image of the received image data in the preview area 101. Then, when the user selects the print execution button 120, the image data received from the generation AI server 300 is sent to the printer 2000. What is sent at this time is a print job that includes the above image data and the print settings set by the user in the print application 1100. The printer 2000 that receives the print job then executes the received print job and prints the image on paper.

[0034] In other words, the printing application 1100 can display a history that includes images generated based on prompts entered by the user and the images of the first image data entered.

[0035] Furthermore, the printing application 1100 can also display an image generated based on a prompt included in the history selected from the displayed history and a second image of data that is different from the first image data previously entered.

[0036] Furthermore, the printing application 1100 can send the prompts entered by the user and the first image data entered by the user to the generation AI server 3000. The printing application 1100 can then receive the image data of the image generated based on the transmitted prompts and the first image data. By performing this process, the combination of the transmitted prompts and the image data generated based on those prompts can be stored as history data. The user can view this history data in the history area 201. In other words, the history, including the input prompts and the image data received from the generation AI server 3000, is displayed in the history area.

[0037] Furthermore, the print application 1100 can send the prompt included in the selected history from the displayed history, along with the input second image data, to the generation AI server 3000. The print application 1100 then receives the image data of the image generated based on the selected prompt included in the history and the second image data from the generation AI server 3000. The print application 1100 then displays the received image data in the preview area 101. Also, as shown in the screen of Figure 7, the history list is selected while the second image data is displayed.

[0038] Furthermore, the print application 1100 can accept print settings such as color mode, duplex printing, and paper size. When the print execution button 120 is selected, the print application 1100 sends the print settings specified by the user and the image data of the image displayed in the preview area 101 to the printer 2000.

[0039] Furthermore, after the history is displayed, the printing application 1100 receives image data from the AI ​​server 3000 without accepting any input from the user regarding the prompts displayed in the history. This image data is the image data of an image generated based on the prompts included in the selected history and the image of the second image data. The image of this image data is then displayed in the preview area 101.

[0040] In addition, the preview area 101 displays an image generated based on the prompts included in the selected history and the image of the second input image data, along with information indicating that the image was generated by the generation AI.

[0041] The operation of the printing application 1100 described above is achieved when the CPU 111 of the computer 1000 executes the printing application 1100.

[0042] <Historical Data Structure> Referring to Figure 8, the structure of the history data stored in the data management unit 3103 of the generation AI service 3100 is shown. The table of the history data DB shown in Figure 11 includes prompt 301, pre-conversion data 302, post-conversion data 303, job identifier 304, and date and time 305 as items of history data. Each row in the history data DB 300 represents one piece of history data. Prompt 301 is the prompt received by the generation AI service 3100 as a conversion request. Pre-conversion data 302 is the print data before conversion is performed by the prompt. Post-conversion data 303 is the print data after conversion is performed by the prompt. The pre-conversion and post-conversion data may not be directly stored as print data within the history data DB 300, but rather as files in external storage, and the history data DB 300 may store information to identify the files in the external storage. The job identifier 304 stores, for example, the file name or URL of the file to be printed. The date and time 305 is the date and time when conversion was performed by the prompt. The history data DB300 is stored on the HDD314 of computer 1000.

[0043] <Example of a print application timeline screen> The printing application 1100 displays a timeline of conversion results to allow the user to select a previously entered prompt for the current print job and return to that point in time.

[0044] Let's explain with a concrete example. For instance, a user inputs some image data into the print application 1100 and gives an execution command by entering a first prompt such as "Delete image". As a result, the print application 1100 sends the first prompt and the input image data to the generation AI service 3100. The print application 1100 then receives image data of an image generated from the generation AI service 3100 based on the first prompt and the transmitted image data. This is image data of an image generated by the generation AI service 3100 based on the first prompt and the transmitted image data.

[0045] The printing application 1100 then displays the image data (first image) received from the generation AI service 3100 in the preview area 101.

[0046] Next, the user enters an additional prompt (second prompt) to further edit the image data (first image) received from the generation AI service 3100. For example, the user enters a prompt such as "Make the text larger for easier reading" and gives an execution command.

[0047] As a result, the printing application 1100 sends the image data of the first image and the second prompt to the generation AI service 3100. The generation AI service 3100 similarly generates an image (the second image) based on the first image and the second prompt, and sends the image data of the generated image to the printing application 1100.

[0048] The printing application 1100 displays the second image in the preview area 101 based on the image data of the second image received from the generation AI service 3100.

[0049] Then, the user checks the second image in the preview area 101 and realizes that the first image was more appropriate, so the process of switching the image to be printed back to the first image is performed. The present invention can reduce the effort required for this process.

[0050] When the user selects the timeline display button 132, the timeline area 133 is displayed on the print application screen 100.

[0051] The timeline area 133 displays one or more histories in chronological order. Each history, like history 134, displays a timeline preview 136 including the image before and after conversion, and the prompt used for the conversion process (timeline prompt 135). A timeline apply button 137 is also displayed, and when this button is selected, the image corresponding to the selected timeline item (history) is displayed in the preview area 101. In other words, the image displayed in the preview area 101 can be reverted to a previously converted or generated image. Each history in the timeline area 133 also includes the date and time when the history was created, and this date and time are also displayed.

[0052] Furthermore, each history entry is created when the user enters a prompt, issues an instruction to execute image generation, and receives image data of the generated image from the generation AI service 3100. The created history entries are stored on the HDD 114 of the computer 1000.

[0053] The print application screen 500 in Figure 16 is the screen after the first image is displayed in the preview area 101, following the instructions to generate the first image and the second image as described above, and then the timeline apply button 137 is selected, causing the first image to be displayed again.

[0054] As described above, the printing application of this embodiment displays a first image in a first display area, which is generated based on a first prompt entered by the user and the image data entered. Furthermore, while a second image, generated based on a second prompt entered by the user and the first image, is displayed in the first display area, the history corresponding to the first image is displayed in the second display area. Then, upon receiving a print command, the application sends print data, including the image data of the image displayed in the first display area, to the printing device. Furthermore, while the second image is displayed in the first display area, the history corresponding to the first image is selected from the displayed history, and the first image is displayed in the first display area. The first prompt is a prompt entered before the second prompt.

[0055] When the print execution button 120 is selected by the user, the image displayed in the preview area 101 (first display area) is the image that will be sent as print data. In other words, when the print execution button 120 is selected while the first image is displayed in the preview area 101, the print application 1100 has a print data transmission function that sends print data, including the image data of the first image and print settings, to the printer 2000. The print settings sent here are those received by the print application 1100 on a print settings screen (not shown). The image included in the history 134 of Figure 16 is the image entered by the user and the first image generated based on that image and the first prompt entered by the user (the left side of the arrow is the entered image, and the right side is the generated first image). The image included in the history 138 of Figure 16 is the second image generated based on the first image and the second prompt entered by the user (the left side of the arrow is the first image, and the right side is the generated second image). As can be seen from the date and time, history 138 is created and displayed after history 134.

[0056] Furthermore, the history displayed in the timeline area 133 may be reset when a print command is issued by the print application 1100. In other words, until a print command is issued, the user can immediately revert to the original image for any images derived from a single image.

[0057] In Figure 16, the timeline area 133 is shown to the right of the chat, but it may be in a different location or displayed as a separate window. The timeline preview 136 may also include a display of the change in the number of pages before and after conversion. Furthermore, the pages displayed in the timeline preview 136 may be set to display the pages with the largest change differences as a result of the conversion by the history prompt. When the timeline apply button 137 is pressed, it returns the print data to the state at the time of selection and updates the print data displayed in the preview area 101 with the print data at the time of selection. At this time, the result of returning to the chat area 110 may be displayed as a string.

[0058] Figure 16 shows the state after two prompts, "Delete image" and "Enlarge text for easier reading," have been entered and the converted state is displayed. When the timeline apply button 137 for "Delete image" is pressed, the current print data is changed to the converted data of the timeline item including the pressed timeline apply button, and the print data displayed in the preview area 101 is also updated accordingly. Note that although an example of reverting to the converted print data is shown here, it is also possible to revert to the print data before conversion.

[0059] <Timeline Data Structure> Refer to Figure 17 to see the structure of the timeline data stored in the data management unit 1103 of the print application 1100. The table of the timeline data DB 400 shown in Figure 17 includes prompt 401, pre-conversion data 402, post-conversion data 403, and date and time 405 as timeline data items. Each row in the timeline data DB 400 represents one timeline data.

[0060] Prompt 401 is a prompt sent by the print application 1100 to the generation AI service as a conversion request. Pre-conversion data 402 is the print data before conversion is performed by the prompt. Post-conversion data 303 is the print data received by the print application 1100 as the result of conversion performed by the generation AI service in response to the prompt. The pre-conversion and post-conversion data may be stored as files in external storage rather than directly in the timeline data DB 400, and the timeline data DB may store this information to identify the files in the external storage. Date and time 405 is the date and time when the conversion was performed by the prompt.

[0061] <Example of a screen showing the use of file metadata in a printing application> The print application 1100 utilizes file metadata to eliminate the need to repeatedly enter the same information when printing a file. Referring to Figure 9, the print application metadata setting screen 300-A and the print application metadata application screen 300-B, which applies file metadata as a prompt, will be described. Objects similar to those in the print application screen 100 will not be described.

[0062] First, let's explain the print application metadata settings screen 300-A. After the print execution button 120 is pressed, the print application 1100 starts printing and displays a metadata embedding selection area 901 in the chat area 110 to ask whether to embed prompts for the instructions used in this conversion into the file to be printed. The metadata embedding selection area 901 may include objects that the user can choose whether or not to embed, to make it easier for the user to respond. Alternatively, the user may respond by natural language input in the chat area 110. The user may also input and respond in a way that embeds some of the prompts that have been entered, or new prompts that have not been entered. When the print application 1100 detects a metadata embedding instruction, it embeds the conversion prompts into the metadata of the print file. After that, it displays a metadata embedding result area 902 in the chat area 110.

[0063] An example of reading and using metadata embedded in the print application metadata settings screen 300-A is explained using the print application metadata application screen 300-B, which applies file metadata as a prompt. When the print application 1100 reads print data, it checks whether the metadata contains a prompt, and if it does, it displays the metadata application display area 903. The metadata application display area 903 displays the content of the embedded prompt and also indicates that the conversion based on that prompt is in progress. Once the conversion is complete, the print application 1100 reflects the conversion results in the preview area 101 and the chat area 110.

[0064] <Processing flow> Referring to Figure 10, the sequence of layout conversion processing in the generation AI service 3100 of this embodiment will be described. The sequence of processing that takes place between the computer 1000, the printer 2000, and each element constituting the generation AI service 3100 will be described. The numbers following "S" below indicate the step numbers in the sequence.

[0065] The request control unit 1101 of the print application 1100 detects a print request from the computer 1000 (S101). The print request includes the print data to be printed.

[0066] The request control unit 1101 displays the received print data as a preview using the preview control unit 1105 (S102).

[0067] The chat control unit 1102 of the print application 1100 detects that a conversion request has been entered by the user (S103) and sends the conversion request to the request control unit 1101 (S104). The request control unit 1101 sends the conversion request to the request control unit 3101 of the generation AI service 3100 (S105). The conversion request includes the prompt entered by the user in the chat input area 121 and the print data. The request control unit 3101 sends the conversion request to the chat control unit 3102 (S106). The chat control unit 3102 interprets the received prompt written in natural language and the print data and determines the layout conversion process for the print data (S107). The chat control unit 3102 sends a request for layout conversion processing to the layout conversion unit 3104 (S108). The layout conversion unit 3104 executes the layout conversion process included in the layout conversion request on the print data included in the layout conversion request (S109). The layout conversion unit 3104 returns the conversion result to the chat control unit 3102 (S110). The chat control unit 3102 saves the conversion result as history data to the data management unit 3103. An example of the saved history data is shown in Figure 8. The chat control unit 3102 generates conversion result text indicating the success or failure of the conversion from the conversion result received in S107 (S111). The chat control unit 3102 returns the conversion request result to the request control unit 3101 (S113). The request control unit 3101 returns the conversion request result to the request control unit 1101 of the print application 1100. The conversion request result includes the converted print data returned from the layout conversion unit 3104 in S110 and the conversion result text generated in S112.

[0068] The request control unit 1101 previews the print data included in the received conversion request result using the preview control unit 1105 (S115). The request control unit 1101 displays the conversion result text included in the received conversion request result in the chat area using the chat control unit 1102 (S116). The request control unit 1101 saves the received conversion result, the date and time the conversion was performed, the prompt used for the conversion, and the print data before conversion as timeline data in the data management unit 1103 of the print application 1100 (S117). Subsequently, by repeating steps S103 to S117 any number of times, print data with the layout desired by the user can be generated. In this embodiment, print data is described as being sent as a conversion request each time, but in order to reduce the amount of communication, the print data sent in S105 may be saved in the data management unit 3103 of the generation AI service 3100. Furthermore, for the second and subsequent conversion requests, the print data may not be included, and the latest converted print data saved in the data management unit 3103 may be targeted for the conversion process in S109. When the print data expected by the user is ready, the request control unit 1101 detects the print command and sends it to the request control unit 2101 of the printer 2000 (S118). The request control unit 2101 then executes the print at the print control unit 2102 (S119). The request control unit 1101 of the print application 1100 discards the timeline data that is no longer needed from the data management unit 1103 (S120).

[0069] Referring to Figure 11, the sequence for displaying the history and applying the history prompt to the current print data in the print application 1100 of this embodiment will be described. The sequence of processing performed between the print application 1100 and the generation AI service 3100 will be described. The numbers following "S" below indicate the step numbers in the sequence.

[0070] The print application 1100 detects the press of the history display button (S201). The print application 1100 sends a history acquisition request to the generation AI service 3100 (S202). The generation AI service 3100 acquires history data from the data management unit 3103 (S203). The history data acquired here is the history data of the user who sent the acquisition request. The generation AI service 3100 returns the history data acquisition request result to the print application 1100 (S204). The history data acquisition request result includes multiple history prompts and their associated pre-conversion and post-conversion data, date and time information, and a job identifier. The print application 1100 displays the history prompts included in the history data received in S204 and images of the print data before and after the change as a set (S205). If there are multiple histories, each is set up and multiple histories are displayed. The display may also include a display of the change in the number of pages before and after the change. In addition, the images of the print data before and after the change that are displayed may be set to show the pages with the largest difference in change as a result of conversion by the history prompt. This allows users to view historical data, prompts, and images before and after conversion, and select the prompt they want to apply to the current print data.

[0071] The print application 1100 detects the selection of history data (S206). This selection may be made for multiple history data. If multiple selections are made, the selection order is stored. The UI of the print application 1100 may display selection numbers to indicate the selection order. The print application 1100 detects the press of the Apply History Data button (S207). The print application 1100 sends a request to apply the selected history data to the generation AI service 3100 (S208). The request to apply history data includes the prompts for the selected history data, their selection order, and the image data currently displayed in the preview area 101. The generation AI service 3100 interprets the received history data prompts in order according to the selection order and performs layout conversion processing (image generation) on the print data (S209). The generation AI service 3100 generates text indicating the result of applying the history prompts (S210). The generation AI service 3100 sends the result of the history data application request to the print application 1100 (S211). The results of this application include the converted print data generated in S209 and S210, and text indicating the conversion results.

[0072] Referring to Figure 18, the process of displaying timeline data and returning print data to a selected point in time in the print application 1100 of this embodiment is shown. The flow in Figure 18 is explained assuming that the conversion process by the generation AI service 3100 shown in S103 to S117 of Figure 10 has been executed multiple times. In other words, the timeline data DB 400 has multiple sets of timeline data (history data) stored in it.

[0073] The print application 1100 confirms that the timeline display button 132 has been pressed and that a timeline display instruction has been detected (801). If a timeline display instruction is detected, the print application 1100 retrieves the timeline data stored in the timeline data DB 400 of the data management unit 1103 of the print application 1100 (802). The print application 1100 displays the prompt, date and time, print data before conversion, and print data after conversion included in the timeline data as history 134 for each of the retrieved timeline data (803). At this time, the display order of the history 134 is sorted by the date and time included in the displayed timeline data and displayed in chronological order. The print application 1100 checks whether the press of the timeline apply button included in one of the timeline items in the displayed timeline area has been detected (804). When the press of the Timeline Apply button is detected, the print application 1100 updates the preview area 101 with the converted print data included in the timeline item containing the pressed Timeline Apply button (805). At this time, the result of returning to the chat area 110 may also be displayed as a string. Subsequently, the print application 1100 updates the timeline data DB 400 (806). Specifically, the time when the Timeline Apply button was pressed is saved as the date and time, the print data before pressing is saved as the pre-conversion data, the print data after pressing is saved as the post-conversion data, and a string indicating that it has returned from the timeline is saved as the prompt, as timeline data. Through these processes, the print application 1100 can easily return to the print data at any point in time converted by the prompt.

[0074] Subsequently, the print application 1100 detects the print command (807) and sends a print job to the printer 2000 (808). The print data sent at this point includes image data of the image displayed in the preview area 101 when the print execution button 120 is selected. Once printing is performed, the print application 1100 deletes all timeline data stored in the timeline data DB 400 (809).

[0075] Referring to Figure 12, the process by which the print application 1100 automatically applies a prompt if a prompt is embedded as metadata in the file to be printed, and the process of suggesting a prompt suitable for the print data to the user using history data are shown.

[0076] The print application 1100 receives a print request from the computer 1000 (401). The print application 1100 checks whether the print data included in the received print request is a file and whether a prompt is included as metadata for that file (402). To determine if the metadata indicates a prompt, it checks whether the metadata name or the metadata itself contains an identification string defined by the print application 1100. If the metadata contains a prompt, it sends the current print data and the prompt to the generation AI service 3100 to execute the prompt immediately (418). If it does not contain a prompt, it sends only the print data to the generation AI service 3100 (403). The print application 1100 checks whether it receives a candidate prompt from the generation AI service 3100 (404). If it does not receive a candidate, it assumes there is no candidate and checks whether to wait for and detect prompt input from the user (417). If it receives a candidate, it assumes there is a candidate and displays the candidate prompt received in 404 as an object that the user can select in the chat area (405). The print application 1100 checks if it has received conversion data from the candidate prompt (406). If it has not received it, it proceeds to flow 409 without doing anything. If it has received it, it detects whether the object displayed in 405 is selected (407). For example, it checks if the object with the candidate prompt displayed has detected an action such as mouseover or long press. If detected, it temporarily displays the conversion data received in 406 in the preview area 101 while detection is occurring (408). When detection stops, for example, when the mouse moves away from the object with the candidate prompt displayed, the data displayed in the preview area 101 is returned to the original print data. This allows the user to easily check the conversion result before selecting a candidate prompt and confirming the conversion. If detection has not occurred in 407, the print application 1100 proceeds to flow 409 without doing anything even if the mouse is hovered over. The print application 1100 detects that the object with the candidate prompt has been selected and an execution command has been issued (409).If no candidate is detected, the print application 1100 assumes that no candidate was selected and waits for prompt input from the user to check for detection (417). If a candidate is detected, the print application 1100 sends the selected candidate prompt to the generation AI service 3100 (410). The print application 1100 checks if it has received the conversion result of the candidate prompt sent in 410 (410). If it has not yet received it, the print application 1100 waits until it receives the conversion result (412). If it has received it, the print application 1100 updates the print data using the received conversion result (413). The print application 1100 updates the preview area 101 with the print data updated in 413 (414). The print application 1100 displays the conversion result text received in 412 in the chat area 110 (415). The print application 1100 stores the transmitted prompt, the original print data, and the current print data (after conversion) in the data management unit 1103 (416). This process enables the print application 1100 to automatically apply a prompt if one is embedded as metadata in the file to be printed, and to suggest a prompt suitable for the print data to the user using the history data. The print application 1100 can continue the conversion process by accepting prompt input from the user. The print application 1100 detects a prompt transmission instruction (417), sends the current print data and prompt to the generation AI service 3100 (418), and terminates the process in this flowchart.

[0077] Referring to Figure 13, the process by which the generation AI service 3100 proposes a prompt suitable for the received print data to the user using history data is shown. The generation AI service 3100 receives print data from the print application 1100 (501). The generation AI service 3100 checks whether there is any history data that matches the job identifier contained in the print data received in 501, or similar history data that matches the job identifier 304 of the history data stored in the data management unit 3103 (502). If none exists, the generation AI service 3100 does not generate a candidate prompt and proceeds to the prompt reception process from the print application 1100 (509). If it does exist, the generation AI service 3100 sends the prompt 301 of the matching or similar history data as a candidate prompt to the print application 1100 (503). At this time, if there are multiple candidate prompts, multiple prompts may be sent. The generation AI service 3100 starts the layout conversion using the candidate prompt in advance, before receiving an execution instruction for the candidate prompt from the print application 1100 (504). If there are multiple candidate prompts, the generation AI service 3100 starts the layout conversion using all of the candidate prompts. Once the conversion is complete, the generation AI service 3100 sends the converted print data to the print application 1100 (505). By starting the conversion process for candidate prompts in advance in this way, the waiting time for users of the print application 1100 who apply a candidate prompt can be reduced. In 503, the generation AI service 3100 checks whether it has received the candidate prompts that have been sent (506). If it has received them, the generation AI service 3100 saves the history data to the data management unit 3103 (507) and deletes the print data that has been converted by the candidate prompts as it is no longer needed (508). If it has not received them, the generation AI service 3100 similarly deletes the converted print data as if it had not been selected (508). The generation AI service 3100 receives the print data and prompts from the print application 1100 (509). In 509, the generation AI service 3100 interprets the received prompts and performs the conversion process on the print data (510).The generation AI service 3100 sends the conversion result to the printing application (511), and the processing of this flowchart ends.

[0078] Referring to Figure 14, the process by which the print application 1100 adds a print footer and prints, and the process of embedding a prompt in the file metadata are shown. The print application 1100 detects a print command by pressing the print button (601). The print application 1100 checks whether the AI-enabled printing setting is enabled in the settings of the print application 1100 (602). If it is enabled, it adds a string to the print job header indicating that it has been edited using AI. The string may be, for example, "This has been edited and printed using AI processing." This allows the user viewing the printout to recognize that the file or website has not been printed as is. The string indicating that the printout has been edited using AI may be previewed and printed to be included in the print footer, other margins of the printout, or within the print body, rather than in the print header area. The print application 1100 creates a print job from the current print data and sends the print job to the printer 2000 (604). The print application 1100 checks if the print target is a "file" (605). If it is not a file, it terminates this flow. If it is a file, the print application 1100 checks if the setting to embed prompts in files is enabled in the print application 1100's settings (606). If it is not enabled, it terminates this flow. If it is enabled, the print application 1100 displays in the chat area whether to embed the prompt used for the conversion as file metadata (609). For example, it displays the string "Do you want to embed "Delete image" in "news.docs" so that it will be automatically applied next time?" along with a button control that the user can press to indicate "Yes" or "No". The print application 1100 checks if it detects an instruction to embed metadata (610). If it does not detect an instruction, it terminates this flow. If it detects an instruction, the print application 1100 embeds the prompt in the file metadata (611). The embedded metadata includes the prompt itself, along with an identifier to indicate that the metadata is a prompt handled by the print application 1100.For example, an identifier may be used as the key name of the metadata, or the identifier may be included in the metadata itself. The print application 1100 displays the embedded result in the chat area (612) and terminates this flow. As a result, the prompt will be automatically applied the next time the file is printed with the print application 1100.

[0079] Referring to Figure 15, the process by which the print application 1100 switches between pre-conversion and post-conversion previews is shown. The print application 1100 detects when the pre-conversion preview toggle button is pressed (701). The print application 1100 checks whether the currently displayed preview is the post-conversion print data or the initial pre-conversion print data (702). If it is the post-conversion data, the print application 1100 changes the display to preview the initial pre-conversion print data (703) and terminates this flow. At the same time, the display of the preview page count also switches back to the number of pages before the change. If it is the pre-conversion data, the print application 1100 changes the display to preview the latest post-conversion print data (704) and terminates this flow. At the same time, the display of the preview page count also switches back to the number of pages before the change. Alternatively, pressing the pre-conversion preview toggle button 105 may switch between a screen that displays only the latest preview and a screen that displays the unedited preview and the latest preview side by side.

[0080] As described above, in this embodiment, the printing system uses a generation AI service 3100 to perform layout conversion based on natural language prompts entered by the user into the printing application 1100, and updates the print preview. The generation AI service 3100 saves the layout conversion history along with the prompts and the print data before and after the conversion, and displays it as history in the printing application 1100.

[0081] The image data (print data) conversion process and layout conversion process described above may also refer to the process of generating new image data based on the original image data and prompts.

[0082] By performing the process described above, when repeatedly editing an input image based on user prompts, the effort required to revert to the edited image at a certain point can be reduced.

[0083] In the above embodiment, the printing application 1100 is described as running on the computer 1000, but it may also be run on the printer 2000, and the conversion process via chat may be performed on the printer 2000. In this case, the process of opening the initial printing application involves selecting a print job from the print job list of held print jobs or the print job history screen of the printer 2000, and starting the printing application with that print job as the print data.

[0084] <Other Embodiments> The present invention provides a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium. It can also be implemented by a process in which one or more processors in the computer of the system or device read and execute the program. Furthermore, it can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of symbols]

[0085] 1000 computers 2000 Printer 3000 Generating AI Servers 1100 Printing Applications

Claims

1. Image display step of displaying a first image generated based on a first prompt input by the user and an image of the input image data in a first display area, In the image display step, while the second image generated based on the second prompt input by the user and the generated first image is displayed in the first display area, a history display step is performed in which the history corresponding to the first image is displayed in the second display area. The information processing device is instructed to perform a print data transmission step, which involves receiving a print instruction and then sending print data, including image data of the image displayed in the first display area, to a printing device. The program is characterized in that, in the image display step, while the second image is displayed in the first display area, a history corresponding to the first image is selected from the displayed history, thereby displaying the first image in the first display area.

2. The information processing device is further instructed to perform a reception process in which it receives prompt input from the user. The program according to claim 1, characterized in that the first prompt is a prompt that received input before the input of the second prompt.

3. The information processing device is further made to perform a transmission step of sending the input first prompt and the input image data to the generation AI server, and a reception step of receiving the image data of the first image generated based on the input prompt and the input image data transmitted in the transmission step. The program according to claim 1, characterized in that, in the image display step, the first image of the received image data is displayed in the first display area.

4. The information processing device is further instructed to perform a setting acceptance process that accepts print settings, The program according to claim 1, wherein the print data transmission step transmits the print data, including the print settings, to the printing device.

5. The program according to claim 1, characterized in that the history includes the input image and the first image.

6. The program according to claim 5, characterized in that in the history display step, the input image, the first image, and the first prompt are displayed in the second display area.

7. The program according to claim 1, characterized in that the first display area is a preview area.

8. The program according to claim 1, characterized in that the history is displayed in chronological order in the second display area.

9. Image display means that displays a first image generated based on a first prompt input by the user and an image of the input image data in a first display area, The image display means includes a history display means that, while the second image generated based on the second prompt input by the user and the generated first image is displayed in the first display area, displays the history corresponding to the first image in the second display area. The system includes a print data transmission means that, upon receiving a print command, transmits print data, including image data of the image displayed in the first display area, to a printing device. The image display means is an information processing device characterized in that, while the second image is displayed in the first display area, a history corresponding to the first image is selected from the displayed history, thereby displaying the first image in the first display area.

10. The system further includes a means for receiving prompt input from the user, The information processing apparatus according to claim 9, characterized in that the first prompt is a prompt that received input before the input of the second prompt.

11. The system further comprises a transmission means for transmitting the input first prompt and the input image data to a generation AI server, and a receiving means for receiving image data of the first image generated based on the input prompt and the input image data transmitted by the transmission means. The information processing apparatus according to claim 9, characterized in that the image display means displays the first image of the received image data in the first display area.

12. It further includes a setting acceptance means for accepting print settings, The information processing apparatus according to claim 9, characterized in that the print data transmission means transmits the print data, including the print settings, to the printing device.

13. The information processing apparatus according to claim 9, characterized in that the history includes the input image and the first image.

14. The information processing apparatus according to claim 13, characterized in that the history display means displays the input image, the first image, and the first prompt in the second display area.

15. The information processing apparatus according to claim 9, characterized in that the first display area is a preview area.

16. The information processing apparatus according to claim 9, characterized in that the history is displayed in chronological order in the second display area.

17. Image display step of displaying a first image generated based on a first prompt input by the user and an image of the input image data in a first display area, In the image display step, while the second image generated based on the second prompt input by the user and the generated first image is displayed in the first display area, a history display step is performed in which the history corresponding to the first image is displayed in the second display area. The system includes a print data transmission step which, upon receiving a print instruction, transmits print data including image data of the image displayed in the first display area to a printing device. The information processing method is characterized in that, in the image display step, while the second image is displayed in the first display area, a history corresponding to the first image is selected from the displayed history, thereby displaying the first image in the first display area.

Images