Information processing device and control method therefor, and program

Abstract

To provide a mechanism capable of easily inserting an AI image generated by a desired keyword into a desired area of an image.SOLUTION: A CPU 170 of a general-purpose computer 102 designates an area into which an AI image is inserted in a document on the basis of a vector 305 when the vector is inputted in a document image being displayed by a user. After that, the CPU 170 displays a window, an AI image generation request including a keyword 306 is transmitted to an image generation server 101 when the keyword 306 of the AI image is inputted to the window by a user, acquires an AI image generated by inputting the keyword 306 to a learned model 164 from the image generation server 101, and inserts the acquired AI image into the designated area in the document.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing apparatus, a control method thereof, and a program, and more particularly to an information processing apparatus, a control method thereof, and a program for inserting an image into a document. [Background technology]

[0002] 2. Description of the Related Art Conventionally, when editing a document using a computer, in order to insert an image into the document, the user must obtain the required image from the Internet or the like and insert it into the document.

[0003] Patent Document 1 discloses a mechanism for synthesizing an image possessed by a user with image material data prepared on the server side and providing the obtained synthesized image to the user.

[0004] In recent years, a service has become known that allows users to enter keywords into document editing software and then have an image generated by an image generation AI based on the keywords provided, making it possible to prepare the image the user needs on the spot. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2008-158602 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, when a user inserts an image obtained from the Internet or an image generated by an image generation AI into a document, the image may not be the right size for the area in the document. In such cases, it becomes necessary to trim or enlarge the image to fit the area, which is time-consuming. In addition, trimming may destroy the original balance of the image.

[0007] In addition, it is generally known that the composition of images generated by image generation AI is greatly affected by their size and aspect ratio, so it is desirable for the image generation AI to generate images of the size actually required.

[0008] In view of the above-mentioned problems, the present invention aims to provide a mechanism that allows an AI image generated using a desired keyword to be easily inserted into a desired area of ​​an image. [Means for solving the problem]

[0009] In order to solve the above problems, the information processing device of the present invention is characterized in that it comprises an area designation means for designating an area from an image for inserting an AI image, a first keyword acquisition means for acquiring a keyword for the AI ​​image, an AI image acquisition means for acquiring an AI image generated by inputting input data including the acquired keyword into a trained model, and an insertion means for inserting the acquired AI image into the designated area of ​​the image. Effect of the Invention

[0010] According to the present invention, an AI image generated by a desired keyword can be easily inserted into a desired area of ​​an image. [Brief description of the drawings]

[0011] [Figure 1A] FIG. 1 is a schematic diagram of a network configuration of an information processing system including a general-purpose computer as an information processing device according to a first embodiment of the present invention. [Figure 1B] FIG. 2 is a block diagram showing a hardware configuration of an image generation server appearing in FIG. [Figure 1C] FIG. 1 is a block diagram showing a hardware configuration of a general-purpose computer. [Diagram 2] 11 is a flowchart of an image insertion process in the first embodiment, which is executed in a general-purpose computer and an image generation server. [Diagram 3]FIG. 11 is a diagram for explaining a specific example of the image insertion process in the first embodiment of the present invention. [Figure 4] FIG. 11 is a diagram for explaining a specific example of a modified example of the image insertion process in the first embodiment of the present invention. [Diagram 5] 13 is a flowchart of a modified example of the image insertion process in the first embodiment, which is executed in a general-purpose computer and an image generation server. [Figure 6] FIG. 11 is a block diagram showing a hardware configuration of an image forming apparatus as an information processing apparatus according to a second embodiment of the present invention. [Figure 7] FIG. 7 is a cross-sectional view of the image forming apparatus in FIG. [Figure 8A] FIG. 4 is a diagram showing a preview display screen displayed on an operation unit of the image forming apparatus. [Figure 8B] FIG. 13 is a diagram showing an image insertion position designation screen displayed on the operation unit. [Figure 8C] FIG. 4 is a diagram showing a keyword input screen displayed on an operation unit. [Figure 8D] FIG. 13 is a diagram showing a preview display screen that displays a preview of a document image after an AI image is inserted into the document. [Figure 9] 13 is a flowchart of an image insertion process executed in the image forming apparatus and the image generation server according to the second embodiment. [Figure 10] FIG. 13 is a diagram for explaining a specific example of the image insertion process in the third embodiment of the present invention. [Figure 11] FIG. 13 is a diagram for explaining a first modified example of the image insertion process in the third embodiment of the present invention. [Figure 12] 13 is a flowchart of a first modified example of an image insertion process executed in a general-purpose computer and an image generation server. [Figure 13] FIG. 13 is a diagram for explaining a second modified example of the image insertion process in the third embodiment of the present invention. [Figure 14] 13 is a flowchart of a second modified example of an image insertion process executed in a general-purpose computer and an image generation server. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the present invention, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present invention.

[0013] <Example 1> Hereinafter, the configuration of an information processing system 1 including a general-purpose computer 102 as an information processing device according to this embodiment will be described with reference to FIGS. 1A to 1C.

[0014] FIG. 1A is a schematic diagram of a network configuration of an information processing system 1. As shown in FIG.

[0015] In the information processing system 1, an image generating server 101, a general-purpose computer 102, and an image forming apparatus 100 are connected to a LAN 103. The LAN 103 may be either a wired network or a wireless network.

[0016] In the present invention, it is sufficient to have either the general-purpose computer 102 or the image forming device 100 as an information processing device, and it is not necessary to have both. In this embodiment, since the general-purpose computer 102 functions as the information processing device according to the present invention, the image forming device 100 does not need to be in the information processing system 1. On the other hand, in a second embodiment described later, since the image forming device 100 functions as the information processing device according to the present invention, the general-purpose computer 102 does not need to be in the information processing system 1. The general-purpose computer 102 or the image forming device 100 communicates with the image generation server 101 via the LAN 103, transmits an instruction for AI image generation, and receives the generated image to acquire an AI image. Note that AI is an abbreviation for Artificial Intelligence. Also, if the general-purpose computer 102 or the image forming device 100 has sufficient computing power for AI image generation, the image generation server 101 may not be included. In this case, the general-purpose computer 102 or the image forming device 100 acquires an AI image by performing AI image generation therein (AI image acquisition means).

[0017] The details of the image forming apparatus 100 will be described with reference to FIG.

[0018] FIG. 1B is a block diagram showing the hardware configuration of the image generation server 101. As shown in FIG.

[0019] The image generation server 101 is equipped with a CPU 160, a RAM 161, a ROM 162, a storage unit 163, a GPU 166, an input device 167, a display 168, and a network I / F 169.

[0020] The CPU 160 controls the operation of the image generation server 101 and operates based on a program stored in the RAM 161. The ROM 162 is a boot ROM, and stores a boot program for an OS (operating system) that operates the image generation server 101. The storage unit 163 is a non-volatile device such as an HDD or SSD, and stores at least one of each of a trained model 164 and an image generation program 165 used in AI image generation as image generation means.

[0021] In this embodiment, the GPU 166 as an image generating means uses an arbitrary trained model as the trained model 164, and uses Stable Diffusion, an existing program, as the image generating program 165. However, the image generating program 165 does not have to be an existing program, and may be any other program as long as it is an image generating program. The data of the trained model 164 and the image generating program 165 are loaded into the RAM 161 and executed by the GPU 166. The AI ​​image generation technology is a known technology, so details will be omitted.

[0022] In the image generation server 101, the GPU 166 inputs parameters included in the AI ​​image generation request as input data to the trained model 164 in response to an instruction from the CPU 160, and performs an AI image generation process that outputs an AI image. The generated AI image is transmitted to the image forming device 100 or the general-purpose computer 102 via the LAN 103, or is stored in the RAM 161 or the storage unit 163. The AI ​​image generation request will be described in detail later.

[0023] An input device 167 is an input device such as a mouse or a keyboard, and a display 168 is a display device that displays a predetermined screen on the image generating server 101. A network I / F 169 is connected to the LAN 103 and controls input and output of various information via the network.

[0024] FIG. 1C is a block diagram showing the hardware configuration of the general-purpose computer 102. As shown in FIG.

[0025] The general-purpose computer 102 is a device including the various units shown in FIG. 1C, such as a PC, a smartphone, or a tablet.

[0026] The general-purpose computer 102 includes a CPU 170 , a RAM 171 , a ROM 172 , a storage unit 173 , a network I / F 174 , an input device 175 , and a display 176 .

[0027] The CPU 170 controls the operation of the general-purpose computer 102, and operates based on a program stored in a RAM 171. The ROM 172 is a boot ROM, and stores a boot program of the OS that operates the general-purpose computer 102.

[0028] The storage unit 173 is a non-volatile device such as an HDD or SSD, and stores system software, programs for controlling the operation of the general-purpose computer 102, etc. The programs stored in the storage unit 173 are loaded into the RAM 171, and the CPU 170 controls the operation of the general-purpose computer 102 based on these programs.

[0029] The input device 175 is an input device such as a mouse or a keyboard, and the display 176 is a display device that displays a predetermined screen on the general-purpose computer 102. If the general-purpose computer 102 is a tablet or a smartphone, the input device 175 and the display 176 may be integrated as a touch panel. The network I / F 174 is connected to the LAN 103 and controls the input and output of various information via the network.

[0030] In the general-purpose computer 102, document data can be edited by loading document editing software stored in the storage unit 173 into the RAM 171, or by executing a cloud-based document editing service or the like that can be executed via the LAN 103 by the CPU 170. Note that the document data may be image data and does not necessarily have to include a document.

[0031] 1C and has sufficient computing power to perform AI image generation, the image generation server 101 is not necessary and the general-purpose computer 102 may perform AI image generation using its internally mounted GPU. In this case, the trained model 164 and the image generation program 165 are stored in the memory unit 173 of the general-purpose computer 102.

[0032] In the following, an example in which a PC is used as the general-purpose computer 102 will be described.

[0033] Here, two examples of a method in which a user inserts a material image, which is an AI image, into a document in this embodiment will be described.

[0034] 3(a) is a diagram showing a document image being edited by the document editing software of the general-purpose computer 102. The document image is displayed on the display 176 by the document editing software, and the screen can be operated by the input device 175. Information such as the title, body, and page numbers has been input to the example document. The first method will be explained using this diagram.

[0035] In the first method, the user specifies the position on the display 176 where he / she wants to insert the AI ​​image into the document using coordinates. In this embodiment, an image generation button (not shown) is provided near the document image of FIG. 3(a) displayed by the document editing software on the display 176, and when this button is selected by the user, the mode switches to one in which the position 300 is specified using coordinates. After switching to this mode, the user can specify the coordinates of the position 300 (FIG. 3(a)) in the document image (area specification UI) displayed on the display 176 by left-clicking the mouse (input device 175: area specification means). Note that UI is an abbreviation for User Interface.

[0036] Note that the method is not limited to the method of this embodiment as long as the user can specify coordinates. For example, when the user selects the coordinates of any position on the image in FIG. 3(a) by left-clicking the mouse (input device 175) and then right-clicks, the document editing software may display a menu including an image generation item on the display 176. In this case, if the user selects the image generation item from the menu, the position selected by the left-click is specified as the coordinates for inserting an AI image into the document. The user may specify the coordinates for inserting an AI image into the document by another method.

[0037] When the coordinates of the position 300 are specified, the window in Fig. 3(b) (AI image information specification UI) is then displayed on the display 176. Fig. 3(b) is an image diagram of the window for specifying information on the AI ​​image to be generated.

[0038] As shown in FIG. 3(b), the user inputs numerical values ​​indicating the size of the AI ​​image to be generated, including width information 301 and height information 302, via a keyboard (input device 175). Furthermore, the user inputs characters for a keyword 303 for the AI ​​image to be generated via a keyboard (input device 175: first keyword acquisition means). When these inputs are completed and the user selects the OK button (FIG. 3(b)), CPU 170 generates an AI image generation request including information on width information 301, height information 302, and keyword 303. Thereafter, CPU 170 (AI image acquisition means) transmits the AI ​​image generation request to image generation server 101 via LAN 103 in order to acquire the AI ​​image from image generation server 101.

[0039] In the image generation server 101, the GPU 166 inputs parameters such as the keyword 303, width information 301, and height information 302 included in the AI ​​image generation request transmitted from the general-purpose computer 102 as input data to the trained model 164. The GPU 166 then executes inference processing using the trained model 164, and generates an AI image based on the keyword 303 and having a width and height specified by the width information 301 and height information 302. When the generation of the AI ​​image is complete, the CPU 160 transmits the AI ​​image to the general-purpose computer 102.

[0040] The AI ​​image generation request includes parameters necessary for generating an AI image, in addition to the above-mentioned width information 301, height information 302, and keyword 303. For example, it includes the number of drawing steps, which indicates how many times the image is updated internally in one image generation, and a scale value, which indicates to what extent the keyword is taken into consideration when generating the image. In addition, it also includes information such as a seed value, which is basically a random value used in the initial state of AI image generation, and the type of sampler used to update the image. These pieces of information may be fixed values ​​or may be values ​​set by the document generation software. However, it is preferable that the seed value be a random value specified for each image generation using a random number.

[0041] The configuration of the image generation request is not limited to that of this embodiment. For example, the image generation request may not include information such as the number of rendering steps described above, and the image generation server 101 may assign fixed or designated values ​​to the information. Even in this case, it is preferable that a random value is designated as the seed value for each image generation.

[0042] In this embodiment, the keyword 303 input by the user is "black camera on desk" as shown in Fig. 3(b), but other keywords may be added to the keyword 303. For example, a keyword such as "Best Quality" for generating a high quality image may be added, or a keyword such as "business" or "work" may be added based on the theme of the image to be generated, for example, if the document being edited is a business document.

[0043] 3(b), in addition to keywords 303 indicating the characteristics of the AI ​​image to be generated, negative keywords specifying objects that the user does not want to appear in the AI-generated image may be input. In that case, an AI image generation request including the negative keywords input by the user is sent to the image generation server 101.

[0044] FIG. 3(c) is a diagram showing a document image in which an AI image generated by the image generation server 101 is inserted into the document. The CPU 170 (insertion means) inserts the preview image of the AI ​​image received from the image generation server 101 as an image 304 in an area starting from position 300 on the document image. The image 304 shown in FIG. 3(c) is merely an example and is not an image generated by the image generation AI, but is actually an image generated by the image generation AI. The document image in FIG. 3(c) is displayed on the screen of the display 176 by the document editing software when the document editing software acquires the AI ​​image generated by the image generation server 101.

[0045] This allows you to insert a material image (AI image) of the desired size at a specified location in a document.

[0046] Although not shown in FIG. 3, a message such as "AI image generating" may be displayed on the display 176 from when the user selects the OK button in the window of FIG. 3(b) until the image of FIG. 3(c) is displayed on the display 176. Also, the user may be able to specify multiple positions at which he or she wants to insert AI images into the image of FIG. 3(a), and may be able to request the generation of an AI image for each position in the window of FIG. 3(b). In this case, a message such as "First AI image generating" may be displayed on the display 176 while the first requested AI image is being generated, and a message such as "Second AI image generating" may be displayed on the display 176 while the second requested AI image is being generated. This allows the user to be notified when it takes time to generate an AI image and it takes time to display the image of FIG. 3(c).

[0047] Furthermore, multiple AI images may be generated at once by the image generation server 101. In this case, the CPU 170 may be configured to present the multiple AI images generated by the image generation server 101 on the display 176 so that the user can select them, and to insert the AI ​​image selected by the user into the document.

[0048] Furthermore, the larger the image size of the image generated by the image generation AI, the more detailed the image tends to be. Therefore, instead of the image size specified by the width information 301 and the height information 302, a value obtained by multiplying the image size by a certain amount may be notified to the image generation server 101 as an AI image generation request. This allows the GPU 166 to generate a more detailed AI image than the image size specified by the width information 301 and the height information 302. In this case, the document editing software can reduce the received AI image to the specified image size and insert it into the specified area of ​​the document, but the time required to generate the AI ​​image increases.

[0049] 3(d) is a diagram showing another document image being edited by the document editing software of the general-purpose computer 102. The second method will be described using this diagram.

[0050] In the second method, the user specifies the area in the document where he / she wants to insert the AI ​​image with vector 305. In this embodiment, an image generation button (not shown) is provided near the image of FIG. 3(d) displayed by the document editing software on display 176, and when the user selects it, the mode switches to an area specification mode. After switching to this mode, the user can specify an area in the document image (area specification UI) displayed on display 176 by dragging vector 305 shown in FIG. 3(d) with the mouse (input device 175: area specification means).

[0051] Note that the method is not limited to the method of this embodiment as long as the user can specify the area. For example, when the user specifies the vector 305 in FIG. 3(d) by dragging it with the mouse (input device 175) and then right-clicks, the document editing software may display a menu including an item for image generation on the display 176. At this time, if the user selects the item for image generation from the menu, the area specified by the vector selected by dragging while left-clicking is set as the area in which the AI ​​image is to be inserted. The vector specifying the area in which the AI ​​image is to be inserted in the document may be specified by the user by another method.

[0052] When the region is specified by the vector 305, the window in Fig. 3(e) is then displayed on the display 176. Fig. 3(e) is a diagram showing a window for specifying information on the AI ​​image to be generated.

[0053] Since the width and height of the AI ​​image to be generated from the vector 305 have been determined, the user inputs only the keyword 306 of the AI ​​image to be generated using a keyboard (input device 175: first keyword acquisition means). When the input is complete and the user selects the OK button (Fig. 3(e)), the CPU 170 transmits an AI image generation request including the width and height of the image determined from the vector 305 and information on the keyword 306 to the image generation server 101 via the LAN 103. Subsequent processing is the same as in the first method, and a preview image of the received AI image data is inserted into the document as image 304 in Fig. 3(c).

[0054] Compared to method 1, method 2 allows the user to more intuitively specify the area into which to insert the image, so it is expected to produce a document that is more precisely as intended.

[0055] 2 is a flowchart of the image insertion process in the embodiment 1, which is executed in the general-purpose computer 102 and the image generation server 101. This process is an image insertion process corresponding to the method 1 and the method 2 described in FIG. 3. The process on the general-purpose computer 102 side (steps S201 to S203, S206) is executed by the CPU 170 expanding the program stored in the ROM 172 into the RAM 171. The process on the image generation server 101 side (steps S204, S205) is executed by the CPU 160 expanding the program stored in the ROM 162 into the RAM 161.

[0056] This process begins when an image generation button (not shown) is selected by the user.

[0057] In step S201, the process waits for the user to operate the input device 175 to specify the position and size (area) of the image to be inserted on the document image. Specifically, in the case of method 1, the process waits for the user to specify the position and size of the image to be inserted by inputting the position 300 (FIG. 3(a)) on the document image and the width and height of the image in the window in FIG. 3(b). In the case of method 2, the process waits for the user to specify the area to insert the AI ​​image by inputting the vector 305 (FIG. 3(d)) on the document image. Although not shown in the flowchart in FIG. 2, if an abnormally small size or area is specified, the process may be interrupted as an error. In addition, if an abnormally large size or area is specified, a warning may be displayed on the display 176 asking whether the image generation is really to be performed, although it will take a long time to generate the image. When the user specifies the position and size (area) of the image to be inserted on the document image, the process proceeds to step S202.

[0058] In step S202, the window of Fig. 3(b) or Fig. 3(d) is displayed on the display 176, and the process waits for the user to input a keyword for the image to be generated by operating the input device 175. When the user inputs a keyword into the window of Fig. 3(b) or Fig. 3(d), the process proceeds to step S203.

[0059] In step S203, an AI image generation request including the size of the image to be generated input in step S201 and the keyword information of the image to be generated input in step S202 is transmitted to the image generation server 101. Then, the process proceeds to step S204.

[0060] After that, the user may be able to continue editing the document on the general-purpose computer 102 until the process proceeds to step S206.

[0061] In step S204, the GPU 166 executes the image generation program 165 in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103 to generate an AI image. The AI ​​image generation request includes parameters required for image generation, such as image size, keywords, and the number of drawing steps. If image generation is already being processed in the image generation server 101 due to another AI image generation request, the requests received earlier are processed in order, and the processing of the relevant request is put on hold until its turn comes. Then, the process proceeds to step S205.

[0062] In step S205, the image data of the AI ​​image generated in step S204 is transmitted to the general-purpose computer 102 via the LAN 103. After that, the process proceeds to step S206.

[0063] In step S206, the AI ​​image acquired from the image generation server 101 is inserted into the document image at the position designated in step S201, and this process ends.

[0064] In the case where multiple images are generated in step S204 in response to a single AI image generation request, the generated images may be presented to the user on display 176 in step S206, and the selected image may be inserted.

[0065] As an example of another configuration, instead of inserting an image in step S206, the generated image may be registered in the material image library of the document editing software. In this case, it is not necessary to specify the position where the image is to be inserted in step S201.

[0066] As yet another example of a configuration, AI image generation may be performed within the general-purpose computer 102, and a series of processes may be completed within the general-purpose computer 102 without using the image generation server 101.

[0067] As yet another example of a configuration, assuming that the user is not satisfied with the generated image, a regeneration button may be provided that allows the user to instruct regeneration of the image without going through the window of Fig. 3(b) or Fig. 3(e). In this case, the content entered in Fig. 3(b) or Fig. 3(e) or an AI image generation request (hereinafter referred to as an initial request) is stored in RAM 171 of general-purpose computer 102. This makes it possible to send an AI image generation request for regeneration to image generation server 101 using the stored content when regeneration is instructed.

[0068] However, it is not preferable to make the keyword information, seed value, and other parameters used as input data to the trained model 164 included in the AI ​​image generation request for regeneration the same as those in the initial request. This is because an AI image regenerated in response to such an AI image generation request for regeneration will essentially be the same as the image generated in response to the initial request. Therefore, in the AI ​​image generation request for regeneration, parameters other than the seed value (keyword information, number of drawing steps, etc.) may be the same as those in the initial request, and the seed value may be set to a value different from that in the initial request.

[0069] When regenerating an image with the above configuration, a relatively different image is likely to be generated each time because the seed value is changed to a value different from the initial request. For this reason, it is not suitable for cases where the user likes the image generated by the initial request to some extent but wants to regenerate an image with only a few changes. Below, two methods for this case are explained. One is a method of regenerating an image based on an AI image already generated by the initial request (hereinafter, generated image), which will be explained in Figures 4 and 5 below, and the other is a method of fixing the seed value.

[0070] The method of fixing the seed value is a method of regenerating an image using an AI image generation request that has the same seed value as the initial request but contains different keyword information. By using the same seed value, it is possible to generate similar images when generating an image and when regenerating an image, and the keywords are changed according to the changes the user wishes to make when regenerating an image. In this configuration, pressing the regenerate button displays the window in Figure 3(b) or Figure 3(e) with the initial request information already entered, and the image is regenerated when the user changes the keywords and selects the OK button.

[0071] A user may be allowed to select which of these regeneration methods is to be used to regenerate the image.

[0072] A method for regenerating an image based on a generated image will be described below.

[0073] In this method, image generation is performed based on a generated image (FIG. 4(a)) based on a specified keyword. Here, the document image 304 of FIG. 3(c) displayed on the display 176 is treated as a generated image 401 (FIG. 4(a)), and an AI image is generated based on this.

[0074] As shown in Fig. 4(a), the document image includes information such as the title, the body, the page number, and image data of the generated image 401. When the user right-clicks the generated image 401 in the document image shown in Fig. 4(a) on the display 176, a window (re-designation UI) shown in Fig. 4(b) to be described later is displayed on the display 176.

[0075] An image generation item may be provided in a menu (not shown) displayed on display 176, and when selected, Fig. 4(b) may be displayed. Alternatively, the document editing software may display a button on display 176 instructing image generation based on a generated image, and when the user selects the button and then selects the generated image to be used as the base, Fig. 4(b) may be displayed.

[0076] FIG. 4(b) is a diagram showing a window for specifying information of an AI image to be generated based on a generated image 401. In this window, the user inputs a keyword 306 (regeneration keyword: in this case, "antique-style camera on a desk") of the AI ​​image to be generated, and then selects the OK button (FIG. 4(b)). In response to this, the CPU 170 transmits an AI image generation request including information such as the width and height information of the selected image, the keyword 306, and the selected image data to the image generation server 101 via the LAN 103. The GPU 166 generates an image based on the AI ​​image generation request transmitted from the general-purpose computer 102. Thereafter, the CPU 160 transmits the image generated by the image generation server 101 to the general-purpose computer 102 via the LAN 103. The CPU 170 replaces the data of the generated image that was the base with the image transmitted from the image generation server 101.

[0077] Furthermore, rather than replacing the generated image with a newly generated image based on it, the newly generated image may be inserted so as to be superimposed on the generated image, or the image may be registered in the source image library of the document editing software.

[0078] The technology for generating AI images based on a specific image is well known, so we will not go into detail here.

[0079] When using a method of regenerating an image based on a previously generated image, the AI ​​image generation request must include the extent to which the previously generated image is to be taken into consideration as a parameter required for AI image generation. This value may be a fixed value, or may be set by the document editing software, and may be a value set by the user in advance, or may be specified by the user on the window in Figure 4(b).

[0080] Fig. 4(c) is a diagram showing a document image when the generated image 401 in the document image of Fig. 4(a) is replaced with a preview image of the AI ​​image received from the image generation server 101. When the document editing software acquires the AI ​​image from the image generation server 101, the document image of Fig. 4(c) is updated and displayed by the document editing software on the screen of the display 176.

[0081] As shown in FIG. 4(c), a preview image 403 of the AI ​​image generated by the image generation server 101 is placed in the area where the generated image 401 of the document image in FIG. 4(a) was originally located.

[0082] If the user is not satisfied with the preview image 403 in the document image in Fig. 4(c), a button may be provided that allows the user to instruct re-generation without going through the window display in Fig. 4(b). The image data included in the AI ​​image generation request generated by CPU 170 in response to this re-generation instruction may be the image data of the generated image 401 or the image data of the preview image 403.

[0083] Also, the AI ​​image on which the preview image 403 is based does not have to be inserted in advance into the document like the generated image 401 in Fig. 4(a), and may be specified by the user from image data that can be referenced by the general-purpose computer 102. In this configuration, it is necessary to display the document image in Fig. 3(a) or the document image in Fig. 3(d) on the display 176 using separate document editing software, and to specify the position and size (or area) for inserting the AI ​​image into the document.

[0084] 5 is a flowchart of a modified example of the image insertion process in the first embodiment, which is executed in the general-purpose computer 102 and the image generation server 101. This process is an image insertion process corresponding to the method of regenerating an image based on the generated image explained in FIG. 4. The process on the general-purpose computer 102 side (steps S501 to S504, S507) is executed by the CPU 170 loading the program stored in the ROM 172 into the RAM 171. The process on the image generation server 101 side (steps S505, S506) is executed by the CPU 160 loading the program stored in the ROM 162 into the RAM 161.

[0085] In step S501, the process waits for a user to operate the input device 175 to specify a generated image (generated image 401 in FIG. 4(a) in this embodiment) that is to be used as the basis for image generation from the document image being displayed on the display 176. When a generated image is specified by the user, the process proceeds to step S502.

[0086] In step S502, information on the position in the document of the generated image 401 specified in step S501, and the width and height of the image is obtained. Then, the process proceeds to step S503.

[0087] In step S503, the window in Fig. 4(b) is displayed on display 176, and awaits user input of a keyword for an image to be generated on the document editing software by operating input device 175. When the user inputs a keyword in the window in Fig. 4(b), the process proceeds to step S504.

[0088] In step S504, an AI image generation request including the image width and height information acquired in step S502, the keyword input in step S503, and the image data of the generated image 401 specified in step S501 is transmitted to the image generation server 101. Then, the process proceeds to step S505.

[0089] After that, the user may be able to continue editing the document on the general-purpose computer 102 until the process proceeds to step S507.

[0090] In step S505, the GPU 166 executes the image generation program 165 in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103 to generate an AI image. If image generation is already being processed in the image generation server 101 due to another AI image generation request, the requests received earlier are processed in order, and the processing of the relevant request is put on hold until its turn comes. Proceed to step S506.

[0091] In step S506, the image data of the image generated in step S505 is transmitted to the general-purpose computer 102 via the LAN 103. After that, the process proceeds to step S507.

[0092] In step S507, the generated image 401 specified in step S501 is replaced with the image of the image data received from the image generation server 101, and the document image after the replacement (FIG. 4(c) is displayed on the display 176, and this process ends.

[0093] By following the above procedure, in this embodiment, the user of the general-purpose computer 102 simply specifies the position and size of the image to be inserted into the document and inputs keywords for the image to be generated, and the image desired by the user is inserted in the area desired by the user at an optimal resolution. This reduces the user's effort in editing documents on the general-purpose computer 102, and allows the use of more appropriate images.

[0094] <Example 2> In the first embodiment, the user specifies the position and size of an image to be inserted into a document and inputs keywords for the image to be generated on the general-purpose computer 102. In contrast, in the second embodiment, the user specifies the position and size of an image to be inserted into a document and inputs keywords for the image to be generated on the image forming apparatus 100. The following describes the second embodiment, focusing on the differences from the first embodiment. Note that the same reference numbers are used for configurations similar to those in the first embodiment, and duplicate descriptions are omitted.

[0095] FIG. 6 is a block diagram showing the hardware configuration of the image forming apparatus 100. As shown in FIG.

[0096] The image forming apparatus 100 includes a control unit 110, a scanner 130, a printer 140, and an operation unit 150.

[0097] The control unit 110 includes a CPU 111 , a RAM 112 , a ROM 113 , a storage unit 114 , a display I / F 116 , an operation unit I / F 117 , an image processing unit 118 , and an image memory 119 .

[0098] The control unit 110 is connected to a scanner 130, which is an image input device, and a printer 140, which is an image output device, and controls the input and output of image information. The control unit 110 is also connected to a LAN 103, and receives print jobs and images generated by the image generation server 101 via the LAN 103.

[0099] The CPU 111 controls the operation of the image forming apparatus 100, and operates based on a program stored in the RAM 112. The ROM 113 is a boot ROM, and stores a boot program of the OS that operates the image forming apparatus 100.

[0100] Storage unit 114 is a non-volatile device such as an HDD or SSD, and stores system software, image data, programs for controlling the operation of image forming apparatus 100, etc. The programs stored in storage unit 114 are loaded into RAM 112, and CPU 111 controls the operation of image forming apparatus 100 based on these programs.

[0101] A network I / F 115 is connected to the LAN 103 and controls input and output of various information via the network. A device I / F 116 connects the control unit 110 to an image input / output device such as a scanner 130 or a printer 140, and performs synchronous / asynchronous conversion of image data.

[0102] The operation unit I / F 117 is an interface that connects the operation unit 150 and the control unit 110, and outputs image data to be displayed on the operation unit 150 to the operation unit 150. The operation unit I / F 117 also transmits information input by the user from the operation unit 150 to the CPU 111. In this embodiment, the operation unit 150 will be described taking a touch panel display as an example, but is not limited to this as long as it has a display unit for receiving output from the control unit 110 and a reception unit for user input. For example, the operation unit 150 may be configured with the display unit and the reception unit separately.

[0103] An image processing unit 118 performs image processing on print data received via the LAN 103, and also performs image processing on image data input / output from the device I / F 116. An image memory 119 is a memory for temporarily expanding image data processed by the image processing unit 118.

[0104] FIG. 7 is a cross-sectional view of an image forming apparatus 100 in this embodiment.

[0105] As shown in FIG. 7, in the image forming apparatus 100, a scanner 130 is disposed above a printer 140. The scanner 130 is a printer that is connected to the printer 140 via a printer port 142. As shown in FIG.

[0106] The printer 140 includes paper feed cassettes 201a to 201c, transport rollers 202a to 202c, a printing unit 203, transport rollers 204, 209, and 211, paper discharge trays 205 and 208, feed rollers 206 and 207, a transport path 210 for double-sided printing, and a stapling device 212.

[0107] Each of the paper feed cassettes 201a to 201c stores paper. Although the image forming apparatus 100 has the three paper feed cassettes 201a to 201c, the number of paper feed cassettes is not limited to three.

[0108] Conveyor rollers 202a-c feed paper stored in corresponding paper feed cassettes 201a-c to printing unit 203. Printing unit 203 prints an image on the fed paper. Printing unit 203 may employ an inkjet method that prints an image by spraying ink onto paper, or an electrophotographic method that prints an image by fixing toner onto paper. Paper printed by printing unit 203 is discharged to paper discharge tray 205 via conveyor rollers 204.

[0109] In the case of double-sided printing, the paper, whose front side has been printed by the printing unit 203, passes through feed rollers 206 and 207 instead of conveyance rollers 204 and is once sent to a paper output tray 208, and is then sent to a double-sided printing conveyance path 210 by reversely rotating feed rollers 207 and conveyance rollers 209. The paper is then conveyed to conveyance rollers 211 and fed again to the printing unit 203. Thereafter, the paper, whose back side has been printed by the printing unit 203, is discharged to a paper output tray 205 via conveyance rollers 204.

[0110] The stapler 212 is capable of stapling the sheets output to the paper output tray 205 .

[0111] Next, a method in which a user inserts a material image, which is an AI image, into a document in this embodiment will be described with reference to FIGS. 8A to 8D.

[0112] FIG. 8A shows a preview display screen displayed on operation unit 150. As shown in FIG.

[0113] 8A, on the preview display screen, a document into which the user wants to insert an AI image is displayed as a preview image 800, and a button 801 for switching to a mode for inserting an image into the document is provided. The document into which the user wants to insert an AI image is acquired by CPU 111 (image acquisition means) in response to a user operation on operation unit 150. Specifically, CPU 111 acquires data on any one of an original image read by scanner 130, an image obtained by RIP processing PDL data received via LAN 103, and a document image stored in advance in storage unit 114.

[0114] Preview image 800 is an image generated based on the data of a document into which a user wants to insert an AI image, which data was acquired by CPU 111. When button 801 is selected by the user, a mode for inserting an image into a document is entered, and the screen displayed on operation unit 150 transitions from the preview display screen of Fig. 8A to an image insertion position designation screen of Fig. 8B.

[0115] FIG. 8B shows an image insertion position designation screen displayed on operation unit 150. As shown in FIG.

[0116] As shown in FIG. 8B, the image insertion position designation screen is a screen for designating an area of ​​a document to be printed into which an AI image generated by the image generation server 101 is to be inserted.

[0117] The user touches two points on preview image 800 via operation unit 150 to specify area 802 into which the user wishes to insert an image. At this time, area 802 is displayed with a dotted line and the two touched points displayed as white circles so that the area specified by the user's operation can be identified.

[0118] Furthermore, as shown in Fig. 8B, the image insertion position designation screen may further include an enlarged display button 806. The enlarged display button 806 is a button whose position within the preview image 800 can be moved by a drag operation by the user, and a part of the preview image 800 near the enlarged display button 806 is enlarged and displayed by a tap operation by the user. This enlarged display allows the user to easily perform the above two touch operations. This enlarged display button 806 may be provided on a screen that displays the document images of Figs. 3(a) and (d) of the first embodiment on the display 176.

[0119] Also, a blank space in preview image 800 may be recognized, and area 802 may be displayed in the blank space so that the user can change the size and position of area 802. This allows the user to easily specify area 802 even if the screen size of operation unit 150 (touch panel display) is small.

[0120] When area 802 is specified by a user operation, the screen displayed on operation unit 150 transitions from the image insertion position specification screen in FIG. 8B to a keyword input screen in FIG. 8C.

[0121] FIG. 8C is a diagram showing a keyword input screen displayed on operation unit 150. As shown in FIG.

[0122] As shown in FIG. 8C, the keyword input screen is a screen for the user to input a keyword for an image that is to be generated as an image to be inserted into a document.

[0123] When the user inputs a keyword using the software keyboard or numeric keypad (not shown) at the bottom of the keyword input screen and selects the OK button (FIG. 8C), CPU 111 (AI image acquisition means) starts acquiring an AI image. Specifically, CPU 111 generates an AI image generation request including information on the width and height of area 802 (FIG. 8B) and the input keyword (FIG. 8C), and transmits it to image generation server 101 via LAN 103. At this time, the data of the transmitted AI image generation request itself, or information included in the request, is stored in RAM 112.

[0124] In the image generation server 101, the GPU 166 generates an AI image based on the AI ​​image generation request transmitted from the image forming apparatus 100. When the generation of the AI ​​image is completed, the CPU 160 transmits the AI ​​image to the image forming apparatus 100.

[0125] As described above, the user may specify the area 802 after changing the display magnification of the preview image 800 using the enlarged display button 806 (FIG. 8B). In this case, however, information on the width and height converted to the size of the area 802 at the original magnification is included in the AI ​​image generation request.

[0126] Fig. 8D is a diagram showing a preview display screen that displays a preview of a document image after an AI image is inserted into a document. An image 803 generated for preview display from the AI ​​image received from the image generation server 101 is displayed in an area 802. In this embodiment, the image 803 may be displayed on the preview display screen of Fig. 8D after being synthesized with the preview image 800, or may be displayed superimposed on the preview image 800. The preview display screen of Fig. 8D is displayed by the CPU 111 on the screen of the operation unit 150 when the AI ​​image is transmitted from the image generation server 101 to the image forming apparatus 100.

[0127] This allows you to insert a material image (AI image) of the desired size at a specified location in a document.

[0128] The preview display screen of FIG. 8D further includes a button 804 for instructing regeneration of an AI image if the user is not satisfied with the image 803.

[0129] When the button 804 is selected by the user, the CPU 111 starts acquiring the regenerated AI image. Specifically, the CPU 111 first calls up information such as the width and height of the area 802 and the keyword entered on the keyword entry screen of FIG. 8C from the RAM 112, and generates an AI image generation request from this information. Note that if data of the AI ​​image generation request, which is the initial request, is stored in the RAM 112, the data may be read out. Next, the CPU 111 transmits the generated (or read) AI image generation request to the image generation server 101, and causes the image generation server 101 to generate an AI image again. Thereafter, the CPU 111 updates the image 803 with an image generated for preview display from the AI ​​image received from the image generation server 101. This procedure may be repeated any number of times.

[0130] On the preview display screen of FIG. 8D, if the user is satisfied with the image 803, the user selects a button 805, and the image data of the original image of the image 803 is synthesized into an area 802 in the image data of the original image of the preview image 800, and the obtained synthesized image is stored in the storage unit 114. In addition, in order to leave the image data of the original image of the preview image 800, the image data of the original image of the image 803 may be synthesized with a copy of this image data to obtain a synthesized image. When the button 805 is selected by the user, the button 801 displayed inverted in FIG. 8D becomes selectable by the user. Therefore, the user may then select the button 801 to further try to insert an AI image into the document image, or may select the back button (FIG. 8D) to end the preview display on the operation unit 150.

[0131] The preview image 800 and the image 803 may be stored in the RAM 112 or in the image memory 119 .

[0132] It should be noted that if the user selects the back button before selecting button 805 and cancels the preview display on operation unit 150, no changes are made to the image data of the document image previewed in FIG. 8A.

[0133] 9 is a flowchart of image insertion processing in the second embodiment, which is executed in the image forming apparatus 100 and the image generating server 101. The processing on the image forming apparatus 100 side (steps S900 to S904, S907 to S909) is executed by the CPU 111 loading a program stored in the ROM 113 into the RAM 112. The processing on the image generating server 101 side (steps S905, S906) is executed by the CPU 160 loading a program stored in the ROM 162 into the RAM 161.

[0134] In step S900, a document (original) into which the user wants to insert an AI image is placed on a platen (not shown) of scanner 130, and the process waits for the user to instruct scanner 130 to scan the original image. When the placement of the document on the platen is detected and the user instructs scanner 130 to scan the original image, the original image is scanned by scanner 130 and the obtained document image is stored in storage unit 114 as a BOX document, and the process proceeds to step S901.

[0135] In step S901, the document image stored in storage unit 114 in step S900 is expanded in RAM 112 or image memory 119, and a preview image 800 is generated from the document image and displayed on operation unit 150. Thereafter, the process waits for a user instruction to generate an image, i.e., for the user to select button 801. Since the generation of preview image 800 is a known technique, details will be omitted. When the user instructs to generate an image, the process proceeds to step S902.

[0136] 8B is displayed on the operation unit 150, and a user specification of the position and size at which the AI ​​image is to be inserted on the preview screen (user specification of area 802 in this embodiment) is awaited. When the user specifies area 802, the process proceeds to step S903.

[0137] 8B is displayed on the operation unit 150, and a user input of a keyword for an image to be generated as an AI image to be inserted is awaited. If a keyword is input by the user, the process proceeds to step S904.

[0138] In step S904, an AI image generation request including the width and height of the area 802 specified in step S902 and the keyword information of the image to be generated input in step S903 is transmitted to the image generation server 101. Then, the process proceeds to step S905.

[0139] After that, the user may be able to continue editing the document on the general-purpose computer 102 until the process proceeds to step S907.

[0140] In step S905, the GPU 166 executes the image generation program 165 in response to the AI ​​image generation request received from the image forming apparatus 100 via the LAN 103 to generate an AI image. If image generation is already being processed in the image generation server 101 due to another AI image generation request, the requests received earlier are processed in order, and the processing of the relevant request is put on hold until its turn comes. Then, the process proceeds to step S906.

[0141] In step S906, the image data generated in step S905 is transmitted to the image forming apparatus 100 via the LAN 103. After that, the process proceeds to step S907.

[0142] In step S907, a preview image 803 is generated from the image data received from the image generation server 101, and the image 803 is synthesized or superimposed on the area 802 specified in step S902 in the preview image displayed in step S901 and displayed. Then, the process proceeds to step S908.

[0143] In step S908, it is determined whether the AI ​​image to be inserted into the document has been finalized. Specifically, if the user selects button 804 on operation unit 150, the process proceeds to step S904 to regenerate the AI ​​image to be inserted (NO in step S908), and the image generation server 101 generates the AI ​​image again. On the other hand, if the user selects button 805 on operation unit 150, the process proceeds to step S909 to finalize the AI ​​image to be inserted (YES in step S908).

[0144] In step S909, the AI ​​image determined in step S908 is synthesized in the area 802 specified in step S902 for the original image of the preview image 800 generated in step S901, and the synthesized image is stored in the storage unit 114. After that, this process ends.

[0145] In this embodiment, when regenerating an AI image, a specific image in a document may be specified and image generation may be performed based on the image data, as in embodiment 1. In this case, image forming apparatus 100 uses image data obtained by cutting out the image of specified area 802.

[0146] The various configurations described in the first embodiment are also applicable to the second embodiment.

[0147] By the above procedure, in this embodiment, the user can insert the image desired by the user at an optimal resolution into the area of ​​the document desired by the user simply by specifying the position and size of the image to be inserted into the document and inputting keywords for the image to be generated in image forming apparatus 100. This reduces the user's effort in editing documents in image forming apparatus 100 and allows the use of more appropriate images.

[0148] <Example 3> In the first and second embodiments, the user specified the position and size of the image to be inserted into the document and input keywords for the image to be generated. In contrast, in the third embodiment, the information processing device determines the position and size of the image to be inserted into the document and the keywords for the image to be generated. In the following, a case will be described in which the information processing device according to the present invention is a general-purpose computer 102, but it may also be an image forming device 100. In this embodiment, the same components as those in the first and second embodiments are assigned the same numbers and duplicated descriptions will be omitted.

[0149] 10 is a diagram for explaining a specific example of the image insertion process in this embodiment. In this embodiment, the CPU 170 analyzes the contents of the document to extract keywords, and includes the keywords in the AI ​​image generation request as keywords for the image to be generated.

[0150] 10A is a diagram showing an image of a document being edited by document editing software of the general-purpose computer 102. Information such as a title, a body text, and a page number is input to the document shown as an example.

[0151] The user specifies the area in which the AI ​​image is to be inserted by the vector 1000 in the document editing software using the second method described in the first embodiment. When the area is specified by the vector 1000, the window in Fig. 10(b) (keyword extraction target specification UI) is displayed on the display 176. Note that the position and size in which the AI ​​image is to be inserted may be specified using the first method described in the first embodiment.

[0152] Fig. 10(b) is a diagram showing a window for specifying the target to be analyzed for keyword extraction. The user selects the target items in the document from which the user wants to extract keywords from among the group of check boxes 1001. In this embodiment, as shown in Fig. 10(b), the title and the body of the text are selected from the group of check boxes 1001 as the target items in the document from which the user wants to extract keywords.

[0153] Thereafter, when the user presses the OK button (FIG. 10(b)), the CPU 170 extracts keywords from character strings included in the selected items (title and body in this embodiment) of the check box group 1001. In this embodiment, keywords that are expected to be extracted from the document in FIG. 10(a) are, for example, "social gathering, summer, seafood dish, sake, Japanese cuisine."

[0154] In addition, the keyword extraction may be performed by transmitting image data of the document to be analyzed and the items selected on the window of FIG. 10(b) to an external device such as the image generation server 101 for analysis.

[0155] The technology for extracting keywords from a document by text analysis is well known, so a detailed description is omitted here. However, when this technology is applied to the image forming apparatus 100, OCR is first performed on the document image (FIG. 10(a)), and keywords are extracted from the obtained character string data.

[0156] In this embodiment, all keywords extracted from a document by text analysis are determined as keywords for the AI ​​image to be generated, but some of the extracted keywords may be used as keywords for the AI ​​image to be generated. For example, multiple keywords extracted from a document may be associated with scores indicating their importance, and the top few keywords with the highest scores may be used as keywords for the AI ​​image to be generated. Alternatively, keywords with a score above a predetermined certain value may be used as keywords for the AI ​​image to be generated.

[0157] Then, CPU 170 transmits an AI image generation request including the extracted AI image keyword to image generating server 101, and causes GPU 166 to generate an AI image. After that, when the AI ​​image is transmitted from image generating server 101, CPU 170 inserts the AI ​​image into the area specified by vector 1000 in the document image of FIG. 10(a).

[0158] The CPU 170 may be configured to display a screen that displays the keywords determined as keywords for the AI ​​image to be generated and allows the user to confirm them before sending an AI image generation request to the image generation server 101. In this configuration, all keywords extracted from the document may be displayed on the screen in a format that allows the user to identify whether they are keywords for the AI ​​image to be generated, and the user may be able to add and delete keywords for the AI ​​image on that screen.

[0159] If no keyword having a score equal to or higher than a predetermined value is obtained, an error screen may be displayed, and a window may be displayed prompting the user to reconsider the analysis range.

[0160] The strength of each keyword of the AI ​​image to be generated, which is included in the AI ​​image generation request sent to the image generation server 101, may be set to change the ease with which the keyword is reflected in the AI ​​image generated by the GPU 166. For example, the CPU 170 may transmit to the image generation server 101 an AI image generation request including, as a keyword for the AI ​​image to be generated, a keyword whose strength is weighted according to the score of the keyword obtained by text analysis.

[0161] Fig. 11 is a diagram for explaining a first modified example of the image insertion process in this embodiment. In the first modified example, in addition to the configuration explained in Fig. 10, the user can specify the range of document analysis for extracting keywords on a window to be described later in Fig. 11(c).

[0162] 11(a) and 11(b) are diagrams showing an image of a document being edited by document editing software executed on the general-purpose computer 102. In this embodiment, a document consisting of two pages will be described.

[0163] The user specifies the area where the AI ​​image is to be inserted by the vector 1100 in the document editing software using the second method described in the first embodiment. When the area is specified by the vector 1100, the window in Fig. 11(c) (keyword extraction range specification UI) is displayed on the display 176. Note that the position and size where the AI ​​image is to be inserted may be specified using the first method described in the first embodiment.

[0164] Fig. 11(c) is a diagram showing a window for specifying the range of document analysis to extract keywords (hereinafter referred to as "document analysis range"). The user selects the document analysis range from a group of check boxes 1101. As shown in Fig. 11(c), this embodiment is configured so that the user can select one of five ranges ("same page", "specified page", "specified section", "entire document", and "free selection") as the document analysis range.

[0165] If "Same Page" is selected, the page on which the AI ​​image is to be inserted is specified as the document analysis range. If "Specified Page" is selected, the page of a separately specified page number (here, the page number entered into page number input field 1101a) is specified as the document analysis range. If "Specified Section" is selected, the chapter or section identified by a separately specified section number (here, the section number entered into section number input field 1101b) is specified as the document analysis range. If "Entire Document" is selected, the entire document is specified as the document analysis range. If "Free Selection" is selected, the user can specify the area of ​​the document analysis range, for example by dragging with the left mouse button to surround the area they want to use as the document analysis range.

[0166] Fig. 11(d) is a diagram showing a window for specifying a target to be analyzed for keyword extraction. The window in Fig. 11(d) (keyword extraction target specification UI) has the same structure as the window in Fig. 10(b), and the user selects target items in a document from which keywords are to be extracted from among checkbox group 1102. Finally, the items specified by checkbox group 1102 (Fig. 11(d)), which are included in the document analysis range specified by checkbox group 1101 (Fig. 11(c)), are analyzed to extract keywords.

[0167] FIG. 12 illustrates a process executed on a general-purpose computer 102 and an image generation server 101. 12 is a flowchart of a first modified example of the image insertion process of the present embodiment 3. The process on the general-purpose computer 102 side (steps S1201 to S1205, S1208) is executed by the CPU 170 expanding the program stored in the ROM 172 into the RAM 171. The process on the image generation server 101 side (steps S1206, S1207) is executed by the CPU 160 expanding the program stored in the ROM 162 into the RAM 161.

[0168] This process begins when an image generation button (not shown) is selected by the user.

[0169] In step S1201, similarly to step S201, the process waits for the user to specify the position and size (area) of the image to be inserted on the document image through the user's operation of the input device 175. Specifically, the process waits for the user to specify the area into which the AI ​​image is to be inserted by inputting a vector 1100 (FIG. 11(b)) onto the document image. When the user specifies the position and size (area) of the image to be inserted on the document image, the process proceeds to step S1202.

[0170] In step S1202, the window in Fig. 11C is displayed on the display 176, and the process waits for the user to specify the document analysis range. Depending on the configuration, this step may be skipped by setting a fixed value. If the user specifies the document analysis range, the process proceeds to step S1203.

[0171] In step S1203, the window in Fig. 11(d) is displayed on the display 176, and awaits user designation for extraction of keywords included in the document analysis range designated in step S1202. Depending on the configuration, this step may be skipped by setting a fixed value. If a user designation for extraction of keywords is received, the process proceeds to step S1204.

[0172] In step S1204, text analysis is performed on the keyword extraction target in step S1203, which is included in the document analysis range specified in step S1202 in the document, to extract keywords for the AI ​​image to be generated. Then, the process proceeds to step S1205.

[0173] In step S1205, an AI image generation request including the size of the image to be generated input in step S1201 and keyword information of the AI ​​image to be generated acquired in step S1204 is sent to the image generation server 101. Then, the process proceeds to step S1206.

[0174] After that, the user may be able to continue editing the document on the general-purpose computer 102 until the process proceeds to step S1208.

[0175] In step S1206, the GPU 166 is caused to generate an AI image in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103. If image generation is already being processed in the image generation server 101 due to another AI image generation request, the requests received earlier are processed in order, and the processing of the relevant request is made to wait until its turn comes. Then, the process proceeds to step S1207.

[0176] In step S1207, the image data of the AI ​​image generated in step S1206 is transmitted to the general-purpose computer 102 via the LAN 103. After that, the process proceeds to step S1208.

[0177] In step S1208, the AI ​​image acquired from the image generation server 101 is inserted into the document image at the position designated in step S1201, and this process ends.

[0178] 13 is a diagram for explaining a second modified example of the image insertion process in this embodiment. In the second modified example, a document image is analyzed to detect free space into which an AI image can be inserted, the size of the detected free space is set as the size of the AI ​​image to be generated, and the generated AI image is inserted into the detected free space.

[0179] 13(a) is a diagram showing an image of a document being edited by the document editing software of the general-purpose computer 102. The document shown as an example has information such as the title, text, and page number entered. There is also a blank area at the bottom of the document.

[0180] In the second modified example, when a user instructs to generate an AI image to be inserted into a document, the CPU 170 detects an area suitable for inserting an AI image generated from the document image (a free area into which an AI image can be inserted) and generates an AI image generation request using the detection result. This eliminates the need for the user to specify the position and size at which to insert the AI ​​image into the document image. In this embodiment, the largest rectangular area 1300 contained in a free area in the document image that is a certain distance or more away from some element such as a page edge, main text, or other image is detected as an area suitable for inserting an image.

[0181] In addition, the configuration may allow the user to input a size at which the detected rectangular area is automatically divided (i.e., the area is divided when the detected rectangular area becomes this size or larger), or the configuration may be such that the area is divided when the detected rectangular area is equal to or larger than a fixed value.

[0182] Fig. 13(b) shows a window for specifying the target to be analyzed for keyword extraction. As in the window in Fig. 10(b), the user selects from the check boxes 1301 the target items in the document from which the user wants to extract keywords.

[0183] Thereafter, when the user presses the OK button (FIG. 13(b)), the CPU 170 performs keyword extraction on character strings included in the selected items of the check box group 1001 (title and body text in this embodiment).

[0184] Then, CPU 170 transmits an AI image generation request including the size of the area suitable for inserting the detected AI image to be generated and the extracted keywords to image generation server 101. As described above, if the area in which the AI ​​image is to be generated is divided into multiple areas, multiple AI image generation requests corresponding to each area may be transmitted.

[0185] Moreover, the keywords may be designated by the user as in the first embodiment, or may be configured to allow the user to specify which range of the document is to be analyzed for keyword extraction as in the third embodiment.

[0186] FIG. 14 illustrates a process executed on the general-purpose computer 102 and the image generation server 101. 14 is a flowchart of the image insertion process in the second modified example described with reference to Fig. 13 in the third embodiment. The process on the general-purpose computer 102 side (steps S1401 to S1406, S1409, S1410) is executed by the CPU 170 expanding the program stored in the ROM 172 into the RAM 171. The process on the image generation server 101 side (steps S1407, S1408) is executed by the CPU 160 expanding the program stored in the ROM 162 into the RAM 161.

[0187] This process begins when an image generation button (not shown) is selected by the user.

[0188] In step S1401, blank areas in the document are analyzed, and then the process proceeds to step S1402.

[0189] In step S1402, it is determined whether the size of the free space detected in step S1401 is equal to or larger than a threshold. If it is equal to or larger than the threshold (YES in step S1402), it is determined that the free space is large enough to insert an AI image, and the process proceeds to step S1403. If only free space less than the threshold is detected (NO in step S1402), the process proceeds to step S1410.

[0190] In step S1403, the position and size of the image to be inserted into the document (rectangular area 1300 in FIG. 13(a)) are determined from the position and size of the free area detected in step S1401, and are stored in RAM 112. The process proceeds to step S1404.

[0191] In step S1404, the process waits for a user specification of which item in the document is to be extracted as a keyword, i.e., a user selection in the check box group 1301 in Fig. 13(b). Depending on the configuration, this step may be skipped by setting a fixed value. After the user selects at least one item in the check box group 1301, when the OK button (Fig. 13(b)) is pressed, the process proceeds to step S1405.

[0192] In step S1405, text analysis is performed on the items in the document designated by the user in step S1404, keywords are extracted, and these are set as keywords for the image to be generated by the image generation server 101. After that, the process proceeds to step S1406.

[0193] In step S1406, an AI image generation request including information on the size of the image to be inserted into the document saved in RAM 112 in step S1403 and the keywords for the image to be generated, which were set in step S1405, is sent to image generation server 101. Then, the process proceeds to step S1407.

[0194] After that, the user may be able to continue editing the document on the general-purpose computer 102 until the process proceeds to step S1409.

[0195] In step S1407, the GPU 166 executes the image generation program 165 in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103 to generate an AI image. If image generation is already being processed in the image generation server 101 due to another AI image generation request, the requests received earlier are processed in order, and the processing of the relevant request is put on hold until its turn comes. Then, the process proceeds to step S1408.

[0196] In step S1408, the image data of the AI ​​image generated in step S1407 is transmitted to the general-purpose computer 102 via the LAN 103. After that, the process proceeds to step S1409.

[0197] In step S1409, the AI ​​image acquired from the image generation server 101 is inserted into the document image at the position stored in the RAM 112 in step S1403, and this process ends.

[0198] In step S1410, an error screen is displayed to inform the user that an area large enough to insert an AI image was not found in the document, and the image generation server 101 does not generate an AI image, and the process ends.

[0199] Although details are omitted, the various configurations described in the first and second embodiments are also applicable to the third embodiment.

[0200] According to the above procedure, in this embodiment, the CPU 170 determines the area where the AI ​​image should be inserted and the keywords to be used for generating it based on the document image, and the AI ​​image is generated by the image generation server 101 based on this information and inserted into the document. As a result, this embodiment can further reduce the user's effort in editing documents compared to the first and second embodiments.

[0201] (Other embodiments) In this embodiment, a program for implementing one or more functions may be provided to a computer of the system or device via a network or storage medium, and the system controller of the system or device may read and execute the program. The system controller may have one or more processors or circuits, and may include multiple separate system controllers or a network of multiple separate processors or circuits to read and execute the executable instructions.

[0202] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0203] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0204] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) An information processing device comprising: an area designation means for designating an area from an image for inserting an AI image; a first keyword acquisition means for acquiring a keyword for the AI ​​image; an AI image acquisition means for acquiring an AI image generated by inputting input data including the acquired keyword into a trained model; and an insertion means for inserting the acquired AI image into the designated area of ​​the image. (Configuration 2) An information processing device as described in configuration 1, characterized in that in response to a user's instruction to generate an AI image to be inserted into the image, the image is displayed in an area designation UI, user designation of the area in the displayed image is accepted, and user input of information about the AI ​​image is accepted in an AI image information designation UI. (Configuration 3) The information processing device according to configuration 2, wherein the area designation means determines the position of the area based on coordinates designated by the user for the image displayed in the area designation UI, and obtains the size of the area input by the user in the AI ​​image information designation UI. (Configuration 4) The information processing device according to configuration 2, wherein the area designation means determines a position and a size of the area based on a vector designated by a user for the image displayed on the area designation UI. (Configuration 5) The information processing device according to any one of configurations 2 to 4, wherein the area designation UI enlarges and displays a portion of the image in response to a user operation. (Configuration 6) The information processing apparatus according to configuration 1, wherein the area designation means designates the area based on an empty area detected by analyzing the image. (Configuration 7) The information processing device according to any one of configurations 2 to 6, wherein the first keyword acquisition means acquires the keyword input by a user in the AI ​​image information designation UI. (Configuration 8) An information processing device according to any one of configurations 2 to 7, characterized in that the AI ​​image is displayed as a generated image in the specified area of ​​an image displayed in the area designation UI. (Configuration 9) The information processing device described in Configuration 8, characterized in that the input data includes a seed value, and when a user selects the generated image displayed in the area designation UI, the AI ​​image acquisition means inputs input data including a seed value different from the seed value, the size of the specified area, and the acquired keyword into the trained model to acquire a regenerated AI image, and the insertion means replaces the generated image with the regenerated AI image. (Configuration 10) The information processing device according to configuration 9, characterized in that a generated image displayed in the area designation UI is updated to the regenerated AI image. (Configuration 11) The information processing device described in Configuration 8, characterized in that the input data includes a seed value, and when a generated image displayed in the area designation UI is selected by a user, a re-designation UI accepts user input of a regeneration keyword, the AI ​​image acquisition means inputs input data including the seed value, the specified size, and the acquired regeneration keyword into the trained model to acquire a regenerated AI image, and the insertion means replaces the generated image with the regenerated AI image. (Configuration 12) The information processing device according to configuration 11, characterized in that a generated image displayed in the area designation UI is updated to the regenerated AI image. (Configuration 13) An information processing device described in any one of configurations 1 to 12, characterized in that the first keyword acquisition means, in response to a user's instruction to generate an AI image to be inserted into the image, accepts user designation of target items from which the keyword is to be extracted in a keyword extraction target designation UI, and performs text analysis on the user-designated target items of the image to extract the keyword. (Configuration 14) The information processing device described in any one of configurations 1 to 12, characterized in that the first keyword acquisition means, in response to a user's instruction to generate an AI image to be inserted into the image, accepts a user designation of one of the same page, a specified page, a specified section, the entire document, or free selection in a keyword extraction range designation UI as the range from which the keyword is to be extracted from the image, and accepts a user designation of a target item from which the keyword is to be extracted in a keyword extraction target designation UI, and extracts the keyword by performing text analysis on the user-designated target item in the user-designated range of the image. (Configuration 15) An information processing device described in any one of configurations 1 to 14, characterized in that the AI ​​image acquisition means sends an AI image generation request including the specified size and the acquired keyword to an image generation server, and acquires the AI ​​image generated in response to the AI ​​image generation request from the image generation server. (Configuration 16) The information processing device according to configuration 15, wherein the information processing device is an image forming device. (Configuration 17) The information processing apparatus according to configuration 16, further comprising image acquisition means for reading an original image and acquiring the image. (Method 1) A control method for an information processing device, comprising: an area designation step for designating an area from an image for inserting an AI image; a first keyword acquisition step for acquiring a keyword for the AI ​​image; an AI image acquisition step for inputting input data including the acquired keyword into a trained model to acquire an AI image generated by inputting the acquired AI image into the designated area of ​​the image; and an insertion step for inserting the acquired AI image into the designated area of ​​the image. (Program 1) A program for causing a computer to function as each of the means of the information processing device described in any one of configurations 1 to 17. [Explanation of symbols]

[0205] 100 Image forming device 101 Image Generation Server 102 General-purpose Computers 103 LAN 170,111 CPU 175 Input Devices 176 Display 130 Scanner 150 Operation section

Claims

1. An area designation means for designating an area from an image into which an AI image is to be inserted; A first keyword acquisition means for acquiring a keyword of the AI ​​image; An AI image acquisition means for acquiring an AI image generated by inputting input data including the acquired keyword into a trained model; an inserting means for inserting the acquired AI image into the designated area of ​​the image; An information processing device comprising:

2. In response to an instruction from a user to generate an AI image to be inserted into the image, displaying the image in a region designation UI and accepting user designation of the region in the displayed image; 2. The information processing apparatus according to claim 1, wherein an AI image information designation UI accepts user input of information on the AI ​​image.

3. The area designation means determining a position of the region based on coordinates designated by a user for the image displayed in the region designation UI; 3. The information processing apparatus according to claim 2, further comprising: acquiring a size of the area input by a user through the AI ​​image information designation UI.

4. The area designation means 3. The information processing apparatus according to claim 2, wherein the position and size of the region are determined based on a vector designated by a user for the image displayed on the region designation UI.

5. 3. The information processing apparatus according to claim 2, wherein the area designation UI enlarges and displays a part of the image in response to a user operation.

6. The area designation means 2. The information processing apparatus according to claim 1, wherein the area is designated based on a free area detected by analyzing the image.

7. The first keyword acquisition means includes: The information processing apparatus according to claim 2, further comprising: acquiring the keyword input by a user through the AI ​​image information designation UI.

8. 3. The information processing apparatus according to claim 2, wherein the AI ​​image is displayed as a generated image in the specified area of ​​the image displayed by the area designation UI.

9. the input data includes a seed value; The AI ​​image acquisition means includes: When the generated image displayed in the area designation UI is selected by a user, a seed value different from the seed value, the size of the designated area, and input data including the acquired keyword are input to the trained model to acquire a regenerated AI image; The information processing apparatus according to claim 8 , wherein the inserting means replaces the generated image with the regenerated AI image.

10. 10. The information processing apparatus according to claim 9, further comprising: updating a generated image displayed in the area designation UI to the regenerated AI image.

11. the input data includes a seed value; When a user selects a generated image displayed in the area designation UI, a user input of a keyword for regeneration is accepted in a re-designation UI; The AI ​​image acquisition means inputs input data including the seed value, the specified size, and the acquired regeneration keyword into the trained model to acquire a regenerated AI image, The information processing apparatus according to claim 8 , wherein the inserting means replaces the generated image with the regenerated AI image.

12. 12. The information processing apparatus according to claim 11, wherein the generated image displayed in the area designation UI is updated to the regenerated AI image.

13. The first keyword acquisition means includes: In response to a user's instruction to generate an AI image to be inserted into the image, a keyword extraction target designation UI accepts a user designation of a target item from which the keyword is to be extracted; 2. The information processing apparatus according to claim 1, further comprising: a text analysis unit that performs a text analysis on the target item designated by the user in the image to extract the keyword.

14. The first keyword acquisition means includes: In response to an instruction from a user to generate an AI image to be inserted into the image, In the keyword extraction range designation UI, a user designation is accepted as a range for extracting the keyword from the image, from among the same page, a designated page, a designated section, the entire document, and free selection; In a keyword extraction target designation UI, a user designation of a target item from which the keyword is to be extracted is accepted; The information processing apparatus according to claim 1 , further comprising: extracting the keywords by performing a text analysis on the target item designated by the user in the range designated by the user of the image.

15. The AI ​​image acquisition means includes: Sending an AI image generation request including the specified size and the acquired keywords to an image generation server; The information processing device according to claim 1, characterized in that the AI ​​image generated in response to the AI ​​image generation request is obtained from the image generation server.

16. 16. The information processing apparatus according to claim 15, wherein the information processing apparatus is an image forming apparatus.

17. 17. An information processing apparatus according to claim 16, further comprising an image acquisition unit for reading an original image and acquiring the image.

18. A method for controlling an information processing device, comprising: An area designation step of designating an area from an image into which an AI image is to be inserted; A first keyword acquisition step of acquiring a keyword of the AI ​​image; An AI image acquisition step of inputting input data including the acquired keyword into a trained model to acquire an AI image generated by the trained model; inserting the acquired AI image into the designated area of ​​the image; A control method comprising:

19. A program for causing a computer to function as each of the means of the information processing device according to claim 1.

Images