Information processing apparatus, control method of information processing apparatus, and storage medium

Abstract

The invention provides an information processing apparatus, a control method of the information processing apparatus, and a storage medium. Provided is a mechanism capable of easily inserting an AI image generated by a desired keyword into a desired region of the image. An information processing apparatus includes: a region specifying section that specifies, from an image, a region for inserting an AI image; a first keyword obtaining section that obtains a keyword of the AI image; an AI image obtaining section obtaining an AI image generated by inputting input data including the obtained keyword into a training model; and an insertion section that inserts the obtained AI image into a specified region of the image.

Description

Technical Field

[0001] The present invention relates to an information processing device, a control method of the information processing device, and a storage medium, and more particularly, to an information processing device for inserting an image into a document, a control method of the information processing device, and a storage medium. Background Art

[0002] Conventionally, when editing a document by using a computer, in order to insert an image into the document, the user must obtain a necessary image from the Internet or the like and then insert the image into the document.

[0003] Japanese Laid-Open Patent Publication (kokai) No. 2008-158602 discloses a mechanism that synthesizes an image owned by a user with image material data prepared on a server side and provides the obtained synthesized image to the user.

[0004] Furthermore, in recent years, a service has become known that allows a user to input (enter) keywords in document editing software and be provided with images generated by image generation artificial intelligence (image generation AI) based on the input keywords, so that images required by the user can be prepared on the spot.

[0005] However, in the case where a user inserts an image obtained from the Internet or the like or an image generated by an image generation AI into a document, the size of the image may not be a size suitable for the area in the document where the user wants to place the image. In this case, it is necessary to crop, enlarge, or reduce the image to be inserted into the document to fit the area in the document where the user wants to place the image, which is time-consuming. In addition, cropping may also destroy the original balance of the image.

[0006] Furthermore, it is well known that the synthesis of images generated by image generation AI is greatly affected by their size and aspect ratio, so it is expected that image generation AI generates images of the actual required size. Summary of the invention

[0007] The present invention provides a mechanism that can easily insert an AI image generated by a desired keyword into a desired area of ​​an image.

[0008] Therefore, the present invention provides an information processing device, comprising: an area designation unit, which designates an area from an image for inserting an AI image; a first keyword acquisition unit, which obtains a keyword of the AI ​​image; an AI image acquisition unit, which obtains an AI image generated by inputting input data including the obtained keyword into a training model; and an insertion unit, which inserts the obtained AI image into the designated area of ​​the image.

[0009] According to the present invention, an AI image generated by a desired keyword can be easily inserted into a desired area of ​​an image.

[0010] Further features of the present invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1A is a schematic diagram of a network configuration of an information processing system including a general-purpose computer as an information processing apparatus according to a first embodiment of the present invention.

[0012] Figure 1B It is shown Figure 1A The image shown is a block diagram of the hardware configuration of the server.

[0013] Figure 1C It is shown Figure 1A A block diagram of the hardware structure of a general-purpose computer is shown.

[0014] Figure 2 is a flowchart of the image insertion process in the first embodiment executed in the general-purpose computer and the image generation server.

[0015] Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E It is a diagram for explaining a specific example of the image insertion process in the first embodiment of the present invention.

[0016] Figure 4A , Figure 4B and Figure 4C 1 is a diagram for explaining a specific example of a modification of the image insertion process in the first embodiment of the present invention.

[0017] Figure 5 is a flowchart of a modified example of the image insertion process in the first embodiment executed in a general-purpose computer and an image generation server.

[0018] Figure 6 is a block diagram showing a hardware configuration of an image forming apparatus as an information processing apparatus according to a second embodiment of the present invention.

[0019] Figure 7 yes Figure 6 A cross-sectional view of the image forming apparatus shown.

[0020] Fig. 8A FIG. 1 is a diagram showing a preview display screen displayed on the operation unit of the image forming apparatus.

[0021] Figure 8B : is a figure which shows the image insertion position designation screen displayed on the operation unit.

[0022] Figure 8C: is a figure which shows the keyword input screen displayed on the operation unit.

[0023] Fig.8D : is a diagram showing a preview display screen that displays a preview of a document image after an AI image is inserted into a document.

[0024] Fig. 9 1 is a flowchart of the image insertion process in the second embodiment executed in the image forming apparatus and the image generation server.

[0025] Fig. 10A and Fig. 10B It is a diagram for explaining a specific example of the image insertion processing in the third embodiment of the present invention.

[0026] Fig.11A , Fig. 11B , Fig. 11C and Fig.11D 1 is a diagram for explaining a first modification of the image insertion process in the third embodiment of the present invention.

[0027] Fig.12 is a flowchart of a first variation of the image insertion process executed in a general-purpose computer and an image generation server.

[0028] Fig.13A and Fig. 13B 1 is a diagram for explaining a second modification of the image insertion process in the third embodiment of the present invention.

[0029] Fig.14 is a flowchart of a second variation of the image insertion process executed in the general-purpose computer and the image generation server. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below with reference to the accompanying drawings showing embodiments of the invention.

[0031] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the present invention as defined in the claims, and not all combinations of features described in the following embodiments are necessary for the solving means of the present invention.

[0032] First, the first embodiment of the present invention will be described. Figures 1A to 1C The configuration of the information processing system 1 including the general-purpose computer 102 as the information processing apparatus according to the first embodiment is described.

[0033] Figure 1A Schematic diagram of the network configuration of the information processing system 1 .

[0034] like Figure 1AAs shown, in the information processing system 1, an image generation server 101, a general-purpose computer 102, and an image forming apparatus 100 are connected to a local area network (LAN) 103. The LAN 103 may be a wired network or a wireless network.

[0035] If the configuration of the present invention has a general-purpose computer 102 or an image forming device 100 as an information processing device, the configuration of the present invention is sufficient, and it is not necessarily necessary to have both at the same time. In the first embodiment, since the general-purpose computer 102 is used as the information processing device according to the present invention, the image forming device 100 may not be included in the information processing system 1. On the other hand, in the second embodiment of the present invention described below, since the image forming device 100 is used as the information processing device according to the present invention, the general-purpose computer 102 may not be included in the information processing system 1. The general-purpose computer 102 or the image forming device 100 communicates with the image generation server 101 via the LAN 103, sends an instruction to generate an AI image, and receives the generated image, thereby obtaining the AI ​​image. It should be noted that AI is an abbreviation for artificial intelligence. In addition, in the case where the general-purpose computer 102 or the image forming device 100 has sufficient computing power for AI image generation, the image generation server 101 may not be required. In this case, the general-purpose computer 102 or the image forming device 100 (AI image acquisition unit) obtains the AI ​​image by performing AI image generation in the general-purpose computer 102 or the image forming device 100.

[0036] Since the reference Figure 7 The details of the image forming apparatus 100 are described, and thus a description thereof will be omitted here.

[0037] Figure 1B is a block diagram showing the hardware configuration of the image generation server 101 .

[0038] like Figure 1B As shown, the image generation server 101 includes: a central processing unit (CPU) 160, a random access memory (RAM) 161, a read-only memory (ROM) 162, a storage unit 163, a graphics processing unit (GPU) 166, an input device 167, a display 168 and a network interface (network I / F) 169.

[0039] The CPU 160 controls the operation of the image generation server 101, and operates based on the program stored in the RAM 161. The ROM 162 is a boot ROM, and stores a boot program for running an operating system (OS) of the image generation server 101. The storage unit 163 is a nonvolatile device such as a hard disk drive (HDD) or a solid state drive (SSD), and stores at least one or more training models 164 used in AI image generation and at least one or more image generation programs 165 used in AI image generation.

[0040] Here, in the first embodiment, the GPU 166 used as the image generation section uses an arbitrary training model as the training model 164, and uses Stable Diffusion, which is an existing program, as the image generation program 165. However, the image generation program 165 does not have to be an existing program, and may be any other program as long as it is an image generation program. The data of the training model 164 and the data of the image generation program 165 are loaded into the RAM 161 and executed by the GPU 166. Since the technology for AI image generation is a well-known technology, its details will be omitted.

[0041] In the image generation server 101, the GPU 166 inputs the parameters included in the AI ​​image generation request as input data into the training model 164 in response to the instruction from the CPU 160, and performs AI image generation processing to output an AI image. The generated AI image is sent to the image forming device 100 and / or the general-purpose computer 102 via the LAN 103, and / or is stored in the RAM 161 and / or the storage unit 163. It should be noted that the details of the AI ​​image generation request will be described below.

[0042] The input device 167 is an input device including, for example, a mouse and a keyboard, and the display 168 is a display device that displays a predetermined screen by the image generation server 101. The network I / F 169 is connected to the LAN 103, and controls input and output of various information via the network.

[0043] Figure 1C is a block diagram showing the hardware configuration of the general-purpose computer 102 .

[0044] The general purpose computer 102 includes Figure 1C The various units shown are devices (such as a personal computer (PC), a smartphone or a tablet).

[0045] like Figure 1C As shown, the general-purpose computer 102 includes a CPU 170 , a RAM 171 , a ROM 172 , a storage unit 173 , a network I / F 174 , an input device 175 , and a display 176 .

[0046] The CPU 170 controls the operation of the general-purpose computer 102, and operates based on the program stored in the RAM 171. The ROM 172 is a boot ROM, and stores a boot program for running the OS of the general-purpose computer 102.

[0047] The storage unit 173 is a nonvolatile device such as an HDD or an SSD, and stores system software, a program for controlling the operation of the general-purpose computer 102, etc. The program stored in the storage unit 173 is loaded into the RAM 171, and the CPU 170 controls the operation of the general-purpose computer 102 based on the program.

[0048] The input device 175 is an input device including, for example, a mouse and a keyboard, and the display 176 is a display device that displays a predetermined screen by the general-purpose computer 102. In the case where the general-purpose computer 102 is a tablet or a smartphone, the input device 175 and the display 176 may be integrated into a touch panel. The network I / F 174 is connected to the LAN 103, and controls input and output of various information via the network.

[0049] The general purpose computer 102 allows document data to be edited by loading document editing software stored in the storage unit 173 into the RAM 171, or by the CPU 170 executing a cloud-based document editing service executable via the LAN 103. It should be noted that the document data may be image data and does not necessarily have to include a document.

[0050] In the general purpose computer 102 including a GPU ( Figure 1C In the case where the general-purpose computer 102 has sufficient computing power to perform AI image generation, the image generation server 101 is not required, and the general-purpose computer 102 can perform AI image generation through a GPU included in the general-purpose computer 102. In this case, the (multiple) training models 164 and the (multiple) image generation programs 165 are stored in the storage unit 173 of the general-purpose computer 102.

[0051] Hereinafter, an example using a PC as the general-purpose computer 102 will be described.

[0052] Here, in the first embodiment, two examples of a method in which a user can insert a material image as an AI image into a document will be described.

[0053] Figure 3A 17 is a diagram showing a document image edited by document editing software of general-purpose computer 102. The document image is displayed on display 176 by document editing software, and the screen can be operated by input device 175. Information such as title, body, and page number is input (entered) in the document as an example. Figure 3A A first method (a first example of a method in which a user can insert a material image as an AI image into a document) is described.

[0054] In the first method, the user specifies the location where he or she wants to insert the AI ​​image in the document by coordinates on the display 176. In the first embodiment, an image generation button (not shown) is provided on the display 176 displayed by the document editing software. Figure 3A , and when the user selects the image generation button, it switches to a mode for specifying the position 300 by coordinates. After switching to this mode, the user can specify the position 300 in the document image (area specification UI) displayed on the display 176 by a left-click operation of the mouse (input device 175: area specification device) (see Figure 3A It should be noted that UI is an abbreviation for user interface.

[0055] It should be noted that this method is not limited to the method of the first embodiment (first method) as long as the method is a method in which the user can specify coordinates. For example, when the user selects Figure 3A When the user selects the coordinates of any position on the document image and then performs a right-click operation, the document editing software may display a menu including items for image generation on the display 176. In this case, when the user selects the item for image generation from the menu, the position selected by the left-click operation is designated as the coordinates for inserting the AI ​​image into the document. The user may specify the coordinates for inserting the AI ​​image into the document by other methods.

[0056] When the coordinates of the position 300 are specified, then the Figure 3B The window shown (AI image information specification UI). Figure 3B An image map of a window for specifying information about an AI image to be generated.

[0057] like Figure 3B As shown, the user inputs numerical values ​​indicating the size of the AI ​​image to be generated, namely, width information 301 and height information 302, by using the keyboard (input device 175). In addition, the user inputs characters for the keyword 303 of the AI ​​image to be generated by using the keyboard (input device 175: first keyword obtaining device). When these inputs are completed and the user selects the OK button (see Figure 3B), the CPU 170 generates an AI image generation request including the width information 301, the height information 302, and the information about the keyword 303. Then, the CPU 170 (AI image obtaining unit) transmits the AI ​​image generation request to the image generation server 101 via the LAN 103 to obtain the AI ​​image from the image generation server 102.

[0058] In the image generation server 101, the GPU 166 inputs the parameters such as the keyword 303, the width information 301, and the height information 302 included in the AI ​​image generation request sent from the general-purpose computer 102 as input data into the training model 164. Thereafter, the GPU 166 performs reasoning processing (reasoning processing based on the training model 164) using the training model 164, and generates an AI image having a size (width and height) specified by the width information 301 and the height information 302 based on the keyword 303. When the generation of the AI ​​image is completed, the CPU 160 sends the generated AI image to the general-purpose computer 102.

[0059] The AI ​​image generation request includes the above-mentioned width information 301, height information 302 and information about keywords 303, as well as parameters required to generate the AI ​​image. For example, the parameters required to generate the AI ​​image include the number of rendering steps and the scale value (scalevalue), the number of rendering steps indicating the number of times the image is updated internally in one image generation, and the scale value indicating the degree to which the keyword is considered when performing image generation. In addition, the parameters required to generate the AI ​​image also include information such as the seed value (basically a random value) used in the initial state of the AI ​​image generation and the type of sampler used in the update of the image. This information can be a fixed value, or it can be a value set by the document editing software. However, it is preferred that each time an image is generated, a random value is specified by a random number as a seed value.

[0060] The construction of the AI ​​image generation request is not limited to the construction of the AI ​​image generation request in the first embodiment. For example, the AI ​​image generation request may not include information such as the number of rendering steps described above, and the image generation server 101 may assign fixed values ​​or specified values ​​to these information. Even in this case, it is preferred that a random value is specified as a seed value by a random number each time image generation is performed.

[0061] Furthermore, in the first embodiment, if Figure 3B As shown, the keyword 303 input by the user is "black camera on the table", but other keywords can also be added to the keyword 303. For example, a keyword for generating a high-quality image, such as "best quality", etc. can be added. In addition, for example, in the case where the document being edited is a business document, a keyword such as "business" or "work" can be added as a subject of the image to be generated.

[0062] Furthermore, in addition to the keyword 303 indicating the features of the AI ​​image to be generated, the user can Figure 3B In the window shown, a negative keyword is input, which specifies the object(s) that the user does not want to appear in the AI ​​generated image. In this case, an AI image generation request including the negative keyword input by the user is sent to the image generation server 101.

[0063] Figure 3C 3 is a diagram showing a document image in which an AI image generated by the image generation server 101 is inserted into a document. The CPU 170 (insertion unit) inserts a preview image of the AI ​​image received from the image generation server 101 as an image 304 into an area starting from a position 300 on the document image. Figure 3C The image 304 shown in the figure is only an example and is not an image generated by the image generation AI, but it is actually an image generated by the image generation AI. When the document editing software obtains the AI ​​image generated by the image generation server 101, the document editing software displays the image on the screen of the display 176. Figure 3C The document image shown.

[0064] As a result, a material image (AI image) having a desired size can be inserted at a designated position in a document.

[0065] although Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E is not shown, but is selected from the user Figure 3B The OK button on the window shown in FIG. 17 is displayed on the display 176. Figure 3C During the period of the document image being shown, a message such as "AI image is being generated" may be displayed on the display 176. In addition, the user can Figure 3A to specify multiple locations in the document image where he or she wants to insert the AI ​​image, and can Figure 3B In this case, while the first requested AI image is being generated, a message such as “the first AI image is being generated” may be displayed on the display 176, and while the second requested AI image is being generated, a message such as “the second AI image is being generated” may be displayed on the display 176. As a result, when the AI ​​image generation takes time and is displayed Figure 3C When the displayed document image takes time, the user can be notified of this.

[0066] In addition, the image generation server 101 may generate multiple AI images at once. In this case, the CPU 170 may be configured to present multiple AI images generated by the image generation server 101 on the display 176 in a user-selectable manner and insert the AI ​​image selected by the user into the document.

[0067] In addition, the larger the image size of the image generated by the image generation AI, the finer the image generated by the image generation AI tends to be. Therefore, the value obtained by multiplying the image size by a certain amount can be notified to the image generation server 101 as an AI image generation request instead of the image size specified by the width information 301 and the height information 302. As a result, the GPU 166 can be caused to generate a fine AI image having an image size larger than the image size specified by the width information 301 and the height information 302. In this case, the document editing software is able to reduce the received AI image to the specified image size and insert it into the specified area of ​​the document, but the time taken by the GPU 166 to generate the AI ​​image will increase.

[0068] Figure 3D 1 is a diagram showing another document image edited by document editing software of the general-purpose computer 102. Figure 3D The second method (a second example of a method in which a user can insert a material image as an AI image into a document) is described.

[0069] In the second method, the user specifies the area in the document where he or she wants to insert the AI ​​image by using the vector 305. In the first embodiment, an image generation button (not shown) is provided on the display 176 displayed by the document editing software. Figure 3D , and when the user selects the image generation button, it switches to a mode for specifying an area. After switching to this mode, the user can drag the area by using the mouse (input device 175: area specifying device) to specify the area. Figure 3D The vector 305 shown is used to specify an area in the document image (area designation UI) displayed on the display 176.

[0070] It should be noted that this method is not limited to the method (second method) of the first embodiment as long as it is a method in which the user can specify an area. For example, when the user drags the mouse (input device 175) Figure 3D305 shown in the figure to designate an area, and then right-click the area, the document editing software can display a menu including items for image generation on the display 176. In this case, when the user selects an item for image generation from the menu, the area designated by the vector selected by the dragging-while-left-clicking operation is set as the area where the AI ​​image is to be inserted. The user can specify by other methods: specifying a vector for the area to be inserted into the document.

[0071] When the region is specified by vector 305, Figure 3E The window shown is then displayed on the display 176 . Figure 3E is a diagram showing a window for specifying information on an AI image to be generated.

[0072] Since the width and height of the AI ​​image to be generated are determined based on the vector 305, the user inputs the keyword 306 of the AI ​​image to be generated by simply using the keyboard (input device 175: first keyword obtaining device). When the input is completed and the user selects the OK button (see Figure 3E ), the CPU 170 sends an AI image generation request to the image generation server 101 via the LAN 103, the AI ​​image generation request including the width and height of the image determined based on the vector 305 and information about the keyword 306. The subsequent processing is the same as the first method, and the CPU 170 uses the preview image of the received AI image data as Figure 3C An image 304 is shown inserted into the document.

[0073] Compared to the first method, the second method allows the user to intuitively designate the area where the image is to be inserted, and is therefore expected to produce a document that better matches the user's intention.

[0074] Figure 2 1 is a flowchart of the image insertion process in the first embodiment executed in the general-purpose computer 102 and the image generation server 101 . Figure 2 The image insertion process shown is the same as that of reference Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3EThe image insertion processing corresponding to the first method and the second method has been described. The processing on the general-purpose computer 102 side (the processing of steps S201 to S203 and step S206) is performed by the CPU 170 loading the program stored in the ROM 172 into the RAM 171. In addition, the processing on the image generation server 101 side (the processing of steps S204 and S205) is performed by the CPU 160 loading the program stored in the ROM 162 into the RAM 161.

[0075] When the user selects an image generation button (not shown), the Figure 2 The image shown is inserted into the process.

[0076] In step S201, the position and size (area) of the image to be inserted into the document image is waited for by the user operation of the input device 175. Specifically, in the case of the first method, the user is waited for by the position 300 (see Figure 3A ) into the document image and the image width and height into Figure 3B In the second method, wait for the user to insert the vector 305 (see Figure 3D ) into the document image to specify the area for inserting the AI ​​image. Figure 2 , but in the case where an abnormally small size or area is specified as an error, the process may be terminated. Figure 2 In addition, in the case where an abnormally large size or area is specified, a warning may be displayed on the display 176, which asks whether to actually perform image generation even if it will take a long time. When the user specifies the position and size (area) of the image to be inserted into the document image, Figure 2 The image insertion process shown proceeds to step S202.

[0077] In step S202, the display 176 displays Figure 3B or Figure 3D The window shown in FIG. 1 is displayed, and the user is waiting to input a keyword of an image that the user wants to generate by operating the input device 175. When the user inputs the keyword into Figure 3B or Figure 3D When the window shown in Figure 2 The image insertion process shown proceeds to step S203.

[0078] In step S203, an AI image generation request including the size of the image to be generated input in step S201 and the information about the keyword of the image that the user wants to generate input in step S202 is sent to the image generation server 101. Thereafter, Figure 2The image insertion process shown proceeds to step S204.

[0079] After that, the user can continue to edit the document on the general purpose computer 102 until Figure 2 The image insertion process shown proceeds to step S206.

[0080] In step S204, the image generation server 101 causes the GPU 166 to execute the image generation program 165 and generate an AI image in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103. The AI ​​image generation request includes image size, keywords, and parameters required for image generation such as the number of rendering steps. In addition, in the case where image generation is already being processed in the image generation server 101 due to another AI image generation request, the previously received request will be processed in order, and the processing of the related request will wait until it is its turn. After that, Figure 2 The image insertion process shown proceeds to step S205.

[0081] In step S205, the image generation server 101 transmits the image data of the AI ​​image generated in step S204 to the general-purpose computer 102 via the LAN 103. Thereafter, Figure 2 The image insertion process shown proceeds to step S206.

[0082] In step S206, the AI ​​image obtained from the image generation server 101 is inserted into the position specified in step S201 on the document image, and Figure 2 The image insertion process shown ends.

[0083] In the case where multiple images are generated in response to a single AI image generation request in step S204, a set of images (multiple images) generated in step S206 may be presented to the user through the display 176, and an image selected by the user may be inserted.

[0084] As an example of another configuration, the generated image may be registered in a material image library of the document editing software instead of inserting the image in step S206. In the case of this configuration, it is not necessary to specify the position where the image is to be inserted in step S201.

[0085] Furthermore, as an example of another configuration, AI image generation may be performed within the general-purpose computer 102 , and a series of processes may be completed within the general-purpose computer 102 without using the image generation server 101 .

[0086] Furthermore, as another example of configuration, in anticipation of a situation where the user is not satisfied with the generated image, a regenerate button may be provided that allows the user to regenerate the image without having to go through Figure 3B or Figure 3E In this case, the window shown in Figure 3B or Figure 3E The contents input in or the AI ​​image generation request (hereinafter referred to as "initial request") are retained in the RAM 171 of the general-purpose computer 102. As a result, in the case of instructing regeneration, the AI ​​image generation request for regeneration can be sent to the image generation server 101 by using the contents already retained in the RAM 171.

[0087] However, it is not preferred to make the other parameters used as input data for training model 164, keyword information, and seed value included in the AI ​​image generation request for regeneration the same as those in the initial request. This is because the AI ​​image regenerated in response to such an AI image generation request for regeneration will be substantially the same as the image generated in response to the initial request. Therefore, in the AI ​​image generation request for regeneration, parameters other than the seed value (keyword information, number of rendering steps, etc.) may be the same as those in the initial request, and the seed value may be set to a value different from that in the initial request.

[0088] It should be noted that in the case of regenerating an image with the above-mentioned construction, since the seed value is changed to a value different from the value in the initial request, a relatively different image may be generated each time. Therefore, the above-mentioned construction is not suitable for a case where the user likes the image generated by the initial request to a certain extent, but wishes to regenerate the image with only minor changes. Two methods for this case will be described below. One method is a method for regenerating an image based on an AI image that has already been generated by the initial request (hereinafter referred to as an "already generated image"), which will be referred to below. Figure 4A , Figure 4B , Figure 4C and Figure 5 Described below, and another method is a method of fixing the seed value.

[0089] The method of fixing the seed value is a method of regenerating an image through an AI image generation request, which includes the same seed value as the seed value in the initial request but includes different keyword information. By using the same seed value, a similar image can be generated when performing image generation and when performing image regeneration, and image regeneration is performed by changing the keyword according to what the user wants to change. In the case of this construction, when the regeneration button is pressed, the display Figure 3B or Figure 3E The window shown has the information of the initial request entered, and when the user changes the keyword and selects the OK button, the image is regenerated.

[0090] The user can select which of these reconstruction methods to use to regenerate the image.

[0091] A method of regenerating an image based on an already generated image will be described below.

[0092] In this method, the generated images (see Figure 4A Here, the image generation of the image displayed on the display 176 will be described. Figure 3C The image 304 in the document image shown is regarded as the generated image 401 (see Figure 4A ), and generate an AI image based on the generated image 401.

[0093] like Figure 4A As shown, the document image includes information such as the title, text, page number, and image data of the generated image 401. When the user right-clicks the Figure 4A When the generated image 401 in the document image shown is displayed, the following is displayed on the display 176 Figure 4B The window shown (Reassign UI).

[0094] An item for image generation may be provided in a menu (not shown) displayed on the display 176, and when an item for image generation is selected, an image generator may be displayed. Figure 4B Alternatively, the document editing software may display a button on the display 176 indicating image generation based on an already generated image, and when the user selects the button and then selects an already generated image to become the basis, the document editing software may display Figure 4B .

[0095] Figure 4B 4 is a diagram showing a window for specifying information about an AI image to be generated based on the already generated image 401. Figure 4B In the window shown, the user enters the keyword 402 of the AI ​​image that the user wants to generate (regenerate the keyword: in this case, "antique camera on the table"), and then selects the OK button (see Figure 4B ). In response to this, the CPU 170 sends an AI image generation request to the image generation server 101 via the LAN 103, the request including information such as the width and height of the selected image, the keyword 402, and the image data of the selected image. The GPU 166 performs image generation based on the AI ​​image generation request sent from the general-purpose computer 102. Thereafter, the CPU 160 sends the image generated by the image generation server 101 to the general-purpose computer 102 via the LAN 103. The CPU 170 replaces the data of the generated image that has become the basis with the image sent from the image generation server 101.

[0096] It should be noted that instead of replacing the generated image with the newly generated image based on the already generated image, the newly generated image may be inserted in a manner superimposed on the already generated image, or may be registered in a material image library of the document editing software.

[0097] Since the technology of generating an AI image based on a specific image is well known, its detailed description will be omitted.

[0098] It should be noted that in this way, in the case of using a method for regenerating an image based on an already generated image, it is necessary to include in the AI ​​image generation request how much consideration should be given to the already generated image that becomes the basis as a parameter required for AI image generation. This value may be a fixed value, or this value may be settable by the document editing software and may be a value set in advance by the user, or this value may be set by the user in Figure 4B Specified on the window shown.

[0099] Figure 4C It is shown in Figure 4A The generated image 401 in the document image shown is replaced by the preview image of the AI ​​image received from the image generation server 101. When the document editing software obtains the AI ​​image from the image generation server 101, the document editing software updates the preview image. Figure 4A The document image shown in FIG. 1 is displayed on the screen of the display 176. Figure 4C The document image shown in .

[0100] like Figure 4C As shown in FIG. 4 , the preview image 403 of the AI ​​image generated by the image generation server 101 is placed Figure 4A The area in the document image shown is where the generated image 401 was originally located.

[0101] In the user Figure 4C If the preview image 403 in the document image shown is not satisfactory, the following button can be provided, which allows the user to modify the document image without going through Figure 4B The image data included in the AI ​​image generation request generated by the CPU 170 in response to the regeneration instruction may be the image data of the generated image 401 or the image data of the preview image 403.

[0102] Furthermore, the AI ​​image that is the basis of the preview image 403 does not have to be Figure 4A The generated image 401 shown in the figure is not inserted into the document in advance, but can be configured so that the user can specify it from the image data that can be referenced by the general-purpose computer 102. In the case of this configuration, it is necessary to display it separately on the display 176 through the document editing software. Figure 3A The document image shown or Figure 3D The document image is shown and indicates the location and size (or area) where the AI ​​image will be inserted into the document.

[0103] Figure 5 10 is a flowchart of a modified example of the image insertion process in the first embodiment executed in the general-purpose computer 102 and the image generation server 101 . Figure 5 The image insertion process shown is based on the reference Figure 4A , Figure 4B and Figure 4C The image insertion process corresponding to the method of regenerating the image of the generated image is performed. The CPU 170 loads the program stored in the ROM 172 into the RAM 171 to execute the process on the general-purpose computer 102 side (the process of steps S501 to S504 and step S507). In addition, the CPU 160 loads the program stored in the ROM 162 into the RAM 161 to execute the process on the image generation server 101 side (the process of steps S505 and S506).

[0104] In step S501, the user operation through the input device 175 is waited for to designate a generated image (in the first embodiment, Figure 4A When the user specifies the generated image, Figure 5 The image insertion process shown proceeds to step S502.

[0105] In step S502, the position of the generated image 401 in the document and information about the width and height of the image specified in step S501 are obtained. Figure 5 The image insertion process shown proceeds to step S503.

[0106] In step S503, the display 176 displays Figure 4B The window shown in FIG. 170 is displayed, and the user is waiting to input the keyword of the image that the user wants to generate on the document editing software by operating the input device 175. Figure 4B When you enter keywords in the window that appears, then Figure 5 The image insertion process shown proceeds to step S504.

[0107] In step S504, an AI image generation request including information about the width and height of the image obtained in step S502, the keyword input in step S503, and the image data of the generated image 401 specified in step S501 is sent to the image generation server 101. Thereafter, Figure 5The image insertion process shown proceeds to step S505.

[0108] After that, the user can continue to edit the document on the general purpose computer 102 until Figure 5 The image insertion process shown proceeds to step S507.

[0109] In step S505, the image generation server 101 causes the GPU 166 to execute the image generation program 165 and generate an AI image in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103. In addition, in the case where image generation is already being processed in the image generation server 101 due to another AI image generation request, the previously received request will be processed in order, and the processing of the relevant request will wait until it is its turn. After that, Figure 5 The image insertion process shown proceeds to step S506.

[0110] In step S506, the image generation server 101 transmits the image data of the image generated in step S505 to the general-purpose computer 102 via the LAN 103. Figure 5 The image insertion process shown proceeds to step S507.

[0111] In step S507, the generated image 401 specified in step S501 is replaced with the image of the image data received from the image generation server 101, and the replaced document image is displayed on the display 176 (see FIG. 17A ). Figure 4C ),and Figure 5 The image insertion process shown ends.

[0112] Through the above process, in the first embodiment, with the aid of the general-purpose computer 102, the user simply specifies the position and size of the image to be inserted into the document and inputs the keyword of the image that the user wants to generate, and the image desired by the user is inserted into the area desired by the user with the best resolution. As a result, the labor of the user in editing the document on the general-purpose computer 102 can be reduced, and a more appropriate image can be used.

[0113] Next, the second embodiment of the present invention will be described. In the first embodiment, with the aid of the general-purpose computer 102, the user has performed: specifying the position and size of the image to be inserted into the document, and inputting the keyword of the image that the user wants to generate. On the other hand, in the second embodiment, with the aid of the image forming device 100, the user performs: specifying the position and size of the image to be inserted into the document, and inputting the keyword of the image that the user wants to generate. The second embodiment will be described below with emphasis on the differences from the first embodiment. It should be noted that in the second embodiment, the same components as those in the first embodiment are represented by the same reference numerals, and their description will be omitted.

[0114] Figure 6 2 is a block diagram showing a hardware configuration of the image forming apparatus 100 .

[0115] The image forming apparatus 100 includes a control unit 110 , a scanner 130 , a printer 140 , and an operation unit 150 .

[0116] The control unit 110 includes a CPU 111 , a RAM 112 , a ROM 113 , a storage unit 114 , a device I / F 116 , an operation unit I / F 117 , an image processing unit 118 , and an image memory 119 .

[0117] The control unit 110 is connected to a scanner 130 as an image input device and a printer 140 as an image output device, and controls input and output of image information. The control unit 110 is also connected to a LAN 103, and receives print jobs, receives images generated by the image generation server 101, etc. via the LAN 103.

[0118] The CPU 111 controls the operation of the image forming apparatus 100, and operates based on the program stored in the RAM 112. The ROM 113 is a boot ROM, and stores a boot program that runs the OS of the image forming apparatus 100.

[0119] The storage unit 114 is a nonvolatile device such as an HDD or an SSD, and stores system software, image data, a program for controlling the operation of the general-purpose computer 102, etc. The program stored in the storage unit 114 is loaded into the RAM 112, and the CPU 111 controls the image forming apparatus 100 based on the program.

[0120] The network I / F 115 is connected to the LAN 103 and controls input and output of various information via the network. The device I / F 116 connects the control unit 110 to the scanner 130 as an image input device and the printer 140 as an image output device and performs synchronous / asynchronous conversion of image data.

[0121] The operation unit I / F 117 is an interface that connects the operation unit 150 and the control unit 110, and outputs image data to be displayed on the operation unit 150 to the operation unit 150. In addition, the operation unit I / F 117 transmits information input by the user from the operation unit 150 to the CPU 111. In the second embodiment, a case where the operation unit 150 is a touch panel display will be described as an example, but the operation unit 150 is not limited to the touch panel display as long as it includes a display unit for displaying an output from the control unit 110 and an acceptance unit for accepting an input from the user. For example, the operation unit 150 may be configured so that the display unit and the acceptance unit are separate units.

[0122] The image processing unit 118 performs image processing on print data received via the LAN 103, and also performs image processing on image data input to and output from the device I / F 116. The image memory 119 is a memory for temporarily storing image data to be processed by the image processing unit 118.

[0123] Figure 7 is a cross-sectional view of the image forming apparatus 100 in the second embodiment.

[0124] like Figure 7 As shown, in the image forming apparatus 100 , the scanner 130 is arranged above the printer 140 .

[0125] The printer 140 includes sheet feeding cassettes 201a, 201b and 201c, conveying rollers 202a, 202b and 202c, a printing unit 203, conveying rollers 204, 209 and 211, sheet discharge trays 205 and 208, feed rollers 206 and 207, a conveying path for double-sided printing (double-sided printing conveying path) 210 and a binding device 212.

[0126] Each of the sheet feeding cassettes 201a, 201b, and 201c stores a sheet (printing sheet). It should be noted that although the image forming apparatus 100 includes the three sheet feeding cassettes 201a, 201b, and 201c, the number of sheet feeding cassettes is not limited to three.

[0127] The conveying rollers 202a, 202b, and 202c feed the sheets stored in the corresponding sheet feeding boxes 201a, 201b, and 201c to the printing unit 203, respectively. The printing unit 203 prints an image on the fed sheet. The printing unit 203 may adopt an inkjet method of printing an image by ejecting ink onto a sheet, or may adopt an electrophotographic method of printing an image by fixing a toner onto a sheet. The sheet that has been printed by the printing unit 203 is discharged onto a sheet discharge tray 205 via the conveying rollers 204.

[0128] In the case of double-sided printing, a sheet whose front side has been printed by the printing unit 203 is conveyed once to a sheet discharge tray 208 via feed rollers 206 and 207 instead of conveying roller 204, and then conveyed to a double-sided printing conveying path 210 by reversely rotating feed rollers 207 and conveying rollers 209. Next, the sheet is conveyed to conveying rollers 211 and fed again to the printing unit 203. Thereafter, a sheet whose back side has been printed by the printing unit 203 is discharged onto a sheet discharge tray 205 via conveying rollers 204.

[0129] The binding device 212 can bind sheets output to the sheet discharge tray 205 .

[0130] Next, in the second embodiment, reference will be made to Fig. 8A , Figure 8B , Figure 8C and Fig.8D A method is described in which a user can insert a material image as an AI image into a document.

[0131] Fig. 8A is a diagram showing a preview display screen displayed on operation unit 150 .

[0132] like Fig. 8A As shown in , on the preview display screen, the document into which the user wants to insert the AI ​​image is displayed as a preview image 800, and a button 801 for switching to a mode of inserting the image into the document is provided. It should be noted that the CPU 111 (image obtaining unit) obtains the document into which the user wants to insert the AI ​​image in response to the user's operation of the operation unit 150. Specifically, the CPU 111 obtains data on one of an original image that has been read by the scanner 130 (reading unit), an image obtained by performing RIP processing on PDL data received via the LAN 103, and a document image previously stored in the storage unit 114.

[0133] The preview image 800 is an image generated based on the data of the document into which the user wants to insert the AI ​​image, which is obtained by the CPU 111. When the user selects the button 801, it switches to the mode for inserting the image into the document, and the screen displayed on the operation unit 150 changes from Fig. 8A The preview display shown switches to Figure 8B The image insertion position specification screen is shown.

[0134] Figure 8B is a diagram showing an image insertion position designation screen displayed on the operation unit 150 .

[0135] like Figure 8BAs shown in , the image insertion position designation screen is a screen for designating into which area of ​​a document to be printed the AI ​​image generated by the image generation server 101 should be inserted.

[0136] The user touches two points on the preview image 800 via the operation unit 150 to specify the region 802 where the user wants to insert the image. At this time, in the region 802, the two touched points are displayed by white circles, and the region 802 itself is displayed by a dotted line, so that the region specified by the user's operation can be easily recognized.

[0137] In addition, if Figure 8B As shown in , the image insertion position designation screen may further include an enlarged display button 806. The enlarged display button 806 is a button whose position within the preview image 800 can be moved by a drag operation performed by the user, and when the user taps the enlarged display button 806, the portion of the preview image 800 near the enlarged display button 808 is enlarged and displayed. This enlarged display allows the user to easily perform a touch operation on the above two points. The enlarged display button 806 can be provided on the display 176 to display the first embodiment. Figure 3A and Figure 3D The document image shown is displayed on the screen.

[0138] Furthermore, a margin portion in preview image 800 can be recognized, and area 802 can be displayed in the margin portion so that the user can change the size and position of area 802. As a result, even when the screen size of operation unit 150 (touch panel display) is small, the user can easily designate area 802.

[0139] When the area 802 is designated by the user's operation, the screen displayed on the operation unit 150 changes from Figure 8B The image insertion position designation screen shown switches to Figure 8C The keyword input screen is shown.

[0140] Figure 8C is a diagram showing a keyword input screen displayed on the operation unit 150 .

[0141] like Figure 8C As shown in , the keyword input screen is a screen where a user inputs a keyword of an image that the user wants to generate as an image to be inserted into a document.

[0142] When the user enters a keyword by using the numeric keypad or the software keyboard (not shown) located at the bottom of the keyword input screen and then selects the OK button (see Figure 8C), the CPU 111 (AI image acquisition unit) starts to acquire an AI image. Specifically, the CPU 111 generates a data file including information about the area 802 (see Figure 8B ) width and height information and the input keyword (see Figure 8C ) and sends the request to the image generation server 101 via the LAN 103. At this time, the data of the sent AI image generation request itself or the information included in the request is stored in the RAM 112.

[0143] In the image generation server 101, the GPU 166 generates an AI image based on the AI ​​image generation request transmitted from the image forming apparatus 100. When the generation of the AI ​​image is completed, the CPU 160 transmits the generated AI image to the image forming apparatus 100.

[0144] It should be noted that, as described above, the user can Figure 8B ) The area 802 is designated after changing the display magnification of the preview image 800. However, in this case, information about the width and height of the size of the area 802 that has been converted to the original magnification is included in the AI ​​image generation request.

[0145] Fig.8D 803 is a diagram showing a preview display screen that displays a preview of a document image after inserting an AI image into a document. An image 803 for preview display generated based on the AI ​​image received from the image generation server 101 is displayed in area 802. In the second embodiment, the image 803 may be displayed in the preview image 800 after being synthesized with the preview image 800. Fig.8D , or may be displayed as superimposed on the preview image 800. When the AI ​​image is transmitted from the image generation server 101 to the image forming apparatus 100, the CPU 111 displays the AI ​​image on the screen of the operation unit 150. Fig.8D The preview display shown in .

[0146] As a result, a material image (AI image) having a desired size can be inserted at a designated position in a document.

[0147] Fig.8D The preview display screen shown also includes a button 804 for instructing to regenerate the AI ​​image if the user is not satisfied with the image 803 .

[0148] When the user selects button 804, CPU 111 starts to obtain the regenerated AI image. Specifically, first, CPU 111 retrieves information such as the width and height of area 802 and the Figure 8C111 receives the information of the keyword input on the keyword input screen shown in the figure, and generates an AI image generation request based on this information. It should be noted that in the case where the data of the AI ​​image generation request as the initial request has been retained in the RAM 112, the data of the AI ​​image generation request as the initial request can be read out. Next, the CPU 111 sends the AI ​​image generation request that has been generated (or has been read out) to the image generation server 101, so that the image generation server 102 generates the AI ​​image again. Then, the CPU 111 updates the image 803 with the image for preview display generated based on the AI ​​image received from the image generation server 101. This process can be repeated any number of times.

[0149] exist Fig.8D On the preview display screen shown, if the user is satisfied with image 803, he or she selects button 805, and the image data of the original image of image 803 is synthesized with area 802 in the image data of the original image of preview image 800, and the obtained synthesized image is stored in storage unit 114. It should be noted that in order to save the image data of the original image of preview image 800, the synthesized image can be obtained by synthesizing a copy of the original image of preview image 800 with the data of the original image of image 803. When the user selects button 805, Fig.8D The highlighted button 801 in the figure becomes selectable by the user. Thus, the user may then select button 801 to further attempt to insert the AI ​​image into the document image, or may then select the return button (see Fig.8D ) to end the preview display on the operation unit 150.

[0150] Even in the case where the preview image 800 and the image 803 are stored in the RAM 112 , the preview image 800 and the image 803 may be stored in the image memory 119 .

[0151] It should be noted that in the case where the user selects the return button and cancels the preview display on the operation unit 150 before selecting the button 805, Fig. 8A Make any changes to the image data of the document image being previewed.

[0152] Fig. 91 is a flowchart of the image insertion processing in the second embodiment executed in the image forming apparatus 100 and the image generation server 101. The processing on the image forming apparatus 100 side (the processing of steps S900 to S904 and S907 to S909) is executed by the CPU 111 loading the program stored in the ROM 113 into the RAM 112. In addition, the processing on the image generation server 101 side (the processing of steps S905 and S906) is executed by the CPU 160 loading the program stored in the ROM 162 into the RAM 161.

[0153] In step S900, a document (original) into which the user wants to insert an AI image is placed on a document placement table (not shown) of the scanner 130, and the scanner 130 waits for a scanning instruction from the user to scan the original image. When it is detected that a document is placed on the document placement table and the user issues a scanning instruction to scan the original image, the scanner 130 scans the original image and stores the obtained document image in the storage unit 114 as a BOX document, and Fig. 9 The image insertion process shown proceeds to step S901.

[0154] In step S901, the document image stored in the storage unit 114 in step S900 is loaded into the RAM 112 or the image memory 119, and a preview image 800 is generated based on the document image and displayed on the operation unit 150. Thereafter, the user is waited to issue an instruction to generate an image, that is, the user is waited to select the button 801. Since the generation of the preview image 800 is a well-known technology, its details will be omitted. When the user issues an instruction to generate an image, Fig. 9 The image insertion process shown proceeds to step S902.

[0155] In step S902, the operation unit 150 displays Figure 8B The preview screen is displayed, and the user is waited for to specify the position and size of the AI ​​image to be inserted on the preview screen (in the second embodiment, the user is waited for to specify the area 802). When the user specifies the area 802, Fig. 9 The image insertion process shown proceeds to step S903.

[0156] In step S903, the operation unit 150 displays Figure 8C The keyword input screen shown in FIG. 1 is displayed, and the user is asked to input a keyword of an image that the user wants to generate as an AI image to be inserted. When the user inputs a keyword, Fig. 9 The image insertion process shown proceeds to step S904.

[0157] In step S904, an AI image generation request is sent to the image generation server 101, the AI ​​image generation request including the width and height of the area 802 specified in step S902 and information about the keyword of the image that the user wants to generate as the AI ​​image to be inserted, which has been input in step S903. Fig. 9 The image insertion process shown in proceeds to step S905.

[0158] After that, the user can continue to edit the document on the general purpose computer 102 until Fig. 9 The image insertion process shown proceeds to step S907.

[0159] In step S905, the image generation server 101 causes the GPU 166 to execute the image generation program 165 and generate an AI image in response to the AI ​​image generation request received from the image forming apparatus 100 via the LAN 103. In addition, in the case where image generation is already being processed in the image generation server 101 due to another AI image generation request, the previously received request will be processed in order, and the processing of the relevant request will wait until it is its turn. After that, Fig. 9 The image insertion process shown proceeds to step S906.

[0160] In step S906, the image generation server 101 transmits the image data generated in step S905 to the image forming apparatus 100 via the LAN 103. Fig. 9 The image insertion process shown proceeds to step S907.

[0161] In step S907, the image forming apparatus 100 generates an image 803 for preview display based on the image data received from the image generation server 101, and displays the image 803 by compositing or superimposing the image 803 on the area 802 specified in step S902 in the preview image displayed in step S901. Fig. 9 The image insertion process shown proceeds to step S908.

[0162] In step S908, the image forming apparatus 100 determines whether the AI ​​image to be inserted into the document has been determined. Specifically, in the case where the user selects the button 804 on the operation unit 150, in order to regenerate the AI ​​image to be inserted into the document ("No" in step S908), Fig. 9 The image insertion process shown in FIG. 1 proceeds to step S904, and causes the image generation server 101 to generate an AI image again. On the other hand, when the user selects the button 805 on the operation unit 150, since the AI ​​image to be inserted into the document has been determined ("Yes" in step S908), Fig. 9The image insertion process shown enters step S909.

[0163] In step S909, the image forming apparatus 100 synthesizes the AI ​​image determined in step S908 with the original image of the preview image 800 generated in step S901 in the area 802 specified in step S902, and stores it in the storage unit 114. Thereafter, Fig. 9 The image insertion process shown ends.

[0164] In the second embodiment, when regenerating an AI image, as in the first embodiment, a specific image in a document can be specified, and image generation can be performed based on its image data. In this case, image data obtained by cutting out an image of a designated area 802 in the image forming device 100 is used.

[0165] It should be noted that the various configurations described in the first embodiment are also applicable to the second embodiment.

[0166] Through the above process, in the second embodiment, with the aid of the image forming apparatus 100, the user simply specifies the position and size of the image to be inserted into the document and inputs a keyword of the image that the user wants to generate, and the image desired by the user is inserted into the area desired by the user in the document with the best resolution. As a result, the labor of the user in editing the document on the image forming apparatus 100 can be reduced, and a more appropriate image can be utilized.

[0167] Next, the third embodiment of the present invention will be described. In the first and second embodiments, the user has already specified the position and size of the image to be inserted into the document, and input the keyword of the image that the user wants to generate. On the other hand, in the third embodiment, the information processing device determines the position and size of the image to be inserted into the document, and the keyword of the image that should be generated. Hereinafter, the case where the information processing device according to the present invention is a general-purpose computer 102 will be described, but the information processing device according to the present invention may be an image forming device 100. It should be noted that in the third embodiment, the same components as those in the first and second embodiments are represented by the same figure marks, and their descriptions will be omitted.

[0168] Fig. 10A and Fig. 10B is a diagram for explaining a specific example of the image insertion process in the third embodiment. In the third embodiment, the CPU 170 analyzes the content of a document to extract keywords, and includes the extracted keywords in an AI image generation request as keywords of an image that a user wants to generate.

[0169] Fig. 10A1 is a diagram showing a document image edited by document editing software of the general-purpose computer 102. Information such as a title, a body, and a page number is input (entered) in the document as an example.

[0170] The user specifies the area where he or she wants to insert the AI ​​image by using the vector 1000 in the document editing software with the aid of the second method already described in the first embodiment. When the area is specified by the vector 1000, the area is displayed on the display 176. Fig. 10B The window (keyword extraction target designation UI) shown in FIG. 1 is a diagram showing a window (keyword extraction target designation UI) shown in FIG. 1 . It should be noted that the position and size that the user wants to insert the AI ​​image can be designated by the first method that has been described in the first embodiment.

[0171] Fig. 10B 1001 is a diagram showing a window for specifying an object to be analyzed for keyword extraction. The user selects an object item in a document from which the user wants to extract keywords from a group of check boxes (check box group) 1001. Fig. 10B As shown, the title and body have been selected from the check box group 1001 as target items in the document from which the user wants to extract keywords.

[0172] After that, when the user presses the OK button (see Fig. 10B ), CPU 170 extracts keywords based on the character strings included in the items selected from check box group 1001 (in the case of the third embodiment, the items selected from check box group 1001 are title and body). In the case of the third embodiment, it is expected that Fig. 10A The keywords extracted from the document shown are, for example, "social gathering, summer, seafood dishes, alcohol, Japanese cuisine".

[0173] It should be noted that the image data of the document to be analyzed may be transmitted to an external device such as the image generation server 101 and ... Fig. 10B The item selected on the window shown in the figure is analyzed by an external device to extract keywords.

[0174] Since the technology of extracting keywords from a document by text analysis is well known, its detailed description will be omitted here. However, in the case where the technology is applied to the image forming apparatus 100, the document image (see Fig. 10A ) performs optical character recognition (OCR) and then performs keyword extraction on the obtained string data.

[0175] In the third embodiment, all keywords extracted from the document by text analysis are determined as keywords of the AI ​​image to be generated, but some of the extracted keywords can be used as keywords of the AI ​​image to be generated. For example, multiple keywords extracted from the document can be obtained in association with scores indicating their importance, and the first few keywords with the highest scores can be used as keywords of the AI ​​image to be generated. Alternatively, keywords with scores equal to or greater than a predetermined specific value can be used as keywords of the AI ​​image to be generated.

[0176] After that, the CPU 170 sends an AI image generation request including the keywords of the AI ​​image extracted above to the image generation server 101, and causes the GPU 166 to generate the AI ​​image. Thereafter, when the AI ​​image is sent from the image generation server 101, the CPU 170 inserts the AI ​​image into Fig. 10A In the area designated by vector 1000 in the document image shown in .

[0177] It should be noted that a configuration may be adopted in which the CPU 170 displays a screen that displays keywords determined as keywords of the AI ​​image to be generated and requires the user to confirm the keywords before sending the AI ​​image generation request to the image generation server 101. In the case of such a configuration, all keywords extracted from the document may be displayed on the screen in a format that allows the user to identify whether they are keywords of the AI ​​image to be generated, and the user can add and / or delete keywords of the AI ​​image on the screen.

[0178] Furthermore, in the case where a keyword having a score equal to or greater than a predetermined specific value is not obtained, an error screen may be displayed, and a window prompting the user to reconsider the analysis scope may be displayed.

[0179] The strength of each keyword of the AI ​​image to be generated (the strength is included in the AI ​​image generation request sent to the image generation server 101) can be set to change the degree of difficulty of the keyword being reflected in the AI ​​image generated by the GPU 166. For example, the CPU 170 can send an AI image generation request to the image generation server 101, the AI ​​image generation request including the keyword whose strength is weighted according to the score of the keyword obtained by text analysis as the keyword of the AI ​​image to be generated.

[0180] Fig.11A , Fig. 11B , Fig. 11C and Fig.11D FIG. 1 is a diagram for explaining a first variation of the image insertion process in the third embodiment. Fig. 10A and Fig. 10B In addition to the described construction, the following reference will be made to Fig. 11C On the description window, the user can specify the scope of document analysis for extracting keywords.

[0181] Fig.11A and Fig. 11B 1 is a diagram showing a document image edited by document editing software running on the general-purpose computer 102. In the third embodiment, a document composed of two pages will be described.

[0182] The user specifies the area where he or she wants to insert the AI ​​image by using the vector 1100 in the document editing software by means of the second method already described in the first embodiment. When the area is specified by the vector 1100, the AI ​​image is displayed on the display 176. Fig. 11C Window (keyword extraction range designation UI) shown. It should be noted that the position and size that the user wants to insert the AI ​​image can be designated by means of the first method that has been described in the first embodiment.

[0183] Fig. 11C 1101 is a diagram showing a window for specifying a range of document analysis for extracting keywords (hereinafter referred to as "document analysis range"). The user selects the document analysis range from a group of check boxes (check box group) 1101. In the third embodiment, as Fig. 11C As shown, the document analysis range is constructed to allow the user to select one of five ranges ("same page", "specified page", "specified part", "entire document", and "free selection") as the document analysis range.

[0184] In the case where "Same Page" has been selected, the page where the AI ​​image is inserted is designated as the document analysis range. In the case where "Specified Page" has been selected, the page of the individually designated page number (here, the page number input into the page number input part 1101a) is designated as the document analysis range. In the case where "Specified Section" has been selected, the chapter or section indicated by the individually designated section number (here, the section number input into the section number input part 1101b) is designated as the document analysis range. In the case where "Entire Document" has been selected, the entire document is designated as the document analysis range. In the case where "Free Selection" has been selected, the user is able to designate an area of ​​the document analysis range, for example, the user designates an area of ​​the document analysis range by dragging left to surround the area that the user wants to set as the document analysis range.

[0185] Fig.11D : is a diagram showing a window for designating an object to be analyzed for keyword extraction. Fig.11D The window shown in (keyword extraction target specification UI) has the same Fig. 10BThe same structure as the window shown in FIG. 11 is shown in FIG. 1101 , and the user selects the target item in the document from which the user wants to extract keywords from a group of check boxes (check box group 1102). Finally, the analysis is performed by the check box group 1101 (see FIG. 1102 ). Fig. 11C ) includes a check box group 1102 in the document analysis scope specified by Fig.11D ) to extract keywords.

[0186] Fig.12 1 is a flowchart of a first variation of the image insertion process in the third embodiment executed in the general-purpose computer 102 and the image generation server 101. The process on the general-purpose computer 102 side (the process of steps S1201 to S1205 and step S1208) is executed by the CPU 170 loading the program stored in the ROM 172 into the RAM 171. In addition, the process on the image generation server 101 side (the process of steps S1206 and step S1207) is executed by the CPU 160 loading the program stored in the ROM 162 into the RAM 161.

[0187] When the user selects an image generation button (not shown), the Fig.12 The image shown is inserted into the process.

[0188] In step S1201, similar to step S201, the position and size (area) of the image to be inserted into the document image is waited for by the user operation of the input device 175. Specifically, wait for the user to specify the position and size (area) of the image to be inserted into the document image by Fig. 11B ) is input into the document image to specify the area for inserting the AI ​​image. When the user specifies the position and size (area) of the image to be inserted into the document image, Fig.12 The image insertion process shown proceeds to step S1202.

[0189] In step S1202, the display 176 displays Fig. 11C The window shown in the figure is displayed and waits for the user to specify the document analysis range. Depending on the configuration, a fixed setting may be used and this step may be skipped. When the user specifies the document analysis range, Fig.12 The image insertion process shown proceeds to step S1203.

[0190] In step S1203, the display 176 displays Fig.11D The window shown in the figure is displayed, and waits for the user to specify the keyword extraction object included in the document analysis range specified in step S1202. Depending on the configuration, a fixed setting may be used, or this step may be skipped. When the user specifies the keyword extraction object, Fig.12 The image insertion process shown proceeds to step S1204.

[0191] In step S1204, text analysis is performed on the keyword extraction object specified in step S1203 and included in the document analysis range specified in step S1202, and keywords of the AI ​​image to be generated are extracted. Fig.12 The image insertion process shown proceeds to step S1205.

[0192] In step S1205, an AI image generation request is sent to the image generation server 101, which includes: the size of the image to be generated input in step S1201, and the information about the keyword of the AI ​​image to be generated obtained in step S1204. Fig.12 The image insertion process shown proceeds to step S1206.

[0193] After that, the user can continue to edit the document on the general purpose computer 102 until Fig.12 The image insertion process shown proceeds to step S1208.

[0194] In step S1206, the image generation server 101 causes the GPU 166 to generate an AI image in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103. In addition, in the case where image generation is already being processed in the image generation server 101 due to another AI image generation request, the previously received request will be processed in order, and the processing of the relevant request will wait until it is its turn. After that, Fig.12 The image insertion process shown proceeds to step S1207.

[0195] In step S1207, the image generation server 101 transmits the image data of the AI ​​image generated in step S1206 to the general-purpose computer 102 via the LAN 103. Thereafter, Fig.12 The image insertion process shown proceeds to step S1208.

[0196] In step S1208, the AI ​​image obtained from the image generation server 101 is inserted into the position specified in step S1201 on the document image, and Fig.12 The image insertion process shown ends.

[0197] Fig.13A and Fig. 13B 2 is a diagram for explaining a second variation of the image insertion process in the third embodiment. In the second variation, a document image is analyzed to detect a blank area into which an AI image can be inserted, the size of the detected blank area is set to the size of the AI ​​image to be generated, and the generated AI image is inserted into the detected blank area.

[0198] Fig.13A1 is a diagram showing a document image edited by document editing software of a general-purpose computer 102. Information such as a title, a body, and a page number are input (entered) in the document as an example. In addition, there is a margin area at the bottom of this document.

[0199] In the second variant, when the user instructs to generate an AI image to be inserted into a document, the CPU 170 detects an area suitable for inserting the AI ​​image to be generated (a blank area where the AI ​​image can be inserted) from the document image, and generates an AI image generation request by using the detection result. As a result, the user does not need to specify the position and size of the AI ​​image to be inserted into the document image. In the third embodiment, a maximum rectangular area 1300 is detected as an area suitable for inserting an image, which is a blank area in the document image and is included in an area composed of pixels that are a certain distance away from certain elements such as a page edge, a body, and another image.

[0200] In addition, the user may be allowed to input a size for automatically dividing the detected rectangular area (ie, when the detected rectangular area becomes equal to or larger than the size, the area is divided). Alternatively, when the detected rectangular area is equal to or larger than a fixed value, the area may be divided.

[0201] Fig. 13B FIG. 4 is a diagram showing a window for specifying an object to be analyzed for keyword extraction. Fig. 10B The window shown is similar, and the user selects, from a group of check boxes (check box group) 1301, the target item in the document from which the user wants to extract keywords.

[0202] After that, when the user presses the OK button (see Fig. 13B ), the CPU 170 performs keyword extraction based on the character strings included in the items selected from the check box group 1301 (in the case of the third embodiment, the items selected from the check box group 1301 are the title and the body).

[0203] Afterwards, the CPU 170 sends an AI image generation request to the image generation server 101, the AI ​​image generation request including: the size of the detected area suitable for inserting the AI ​​image to be generated and the extracted keywords. As described above, in the case where the area for generating the AI ​​image is divided into a plurality of areas, a plurality of AI image generation requests corresponding to the respective areas may be sent.

[0204] Furthermore, the keyword may be specified by user input as in the first embodiment, or may be configured to allow the user to specify which range of the document is to be analyzed for keyword extraction as in the third embodiment.

[0205] Fig.14 It has been referenced Fig.13Aand Fig. 13B 14. The flowchart of the image insertion processing in the second modification of the third embodiment described above is executed in the general-purpose computer 102 and the image generation server 101. The processing on the general-purpose computer 102 side (the processing of steps S1401 to S1406, step S1409, and step S1410) is executed by the CPU 170 loading the program stored in the ROM 172 into the RAM 171. In addition, the processing on the image generation server 101 side (the processing of steps S1407 and step S1408) is executed by the CPU 160 loading the program stored in the ROM 162 into the RAM 161.

[0206] When the user selects an image generation button (not shown), the Fig.14 The image insertion process shown in .

[0207] In step S1401, the CPU 170 analyzes (detects) a blank area in the document. Fig.14 The image insertion process shown proceeds to step S1402.

[0208] In step S1402, the CPU 170 determines whether the size of the blank area detected in step S1401 is equal to or larger than a threshold value. In the case where the size of the blank area detected in step S1401 is equal to or larger than the threshold value ("YES" in step S1402), the CPU 170 determines that the blank area detected in step S1401 is a blank area large enough to insert an AI image, and Fig.14 The image insertion process shown enters step S1403. On the other hand, in the case where only a blank area whose size is smaller than the threshold is detected ("No" in step S1402), Fig.14 The image insertion process shown proceeds to step S1410.

[0209] In step S1403, the CPU 170 determines the position and size of the image to be inserted into the document based on the position and size of the blank area detected in step S1401 ( Fig.13A The rectangular area 1300 shown in FIG. 1300 is stored in RAM 171. Fig.14 The image insertion process shown proceeds to step S1404.

[0210] In step S1404, the process waits for the user to specify which item in the document is the keyword extraction target, that is, waits for the user to select Fig. 13B 1301 shown in the check box group. Depending on the configuration, a fixed setting may be used and this step may be skipped. When the user selects at least one item from the check box group 1301 and then presses the OK button (see Fig. 13B )hour, Fig.14 The image insertion process shown proceeds to step S1405.

[0211] In step S1405, the CPU 170 performs text analysis on the items in the document specified by the user in step S1404, extracts keywords, and sets the extracted keywords as keywords of the image to be generated by the image generation server 101. Fig.14 The image insertion process shown proceeds to step S1406.

[0212] In step S1406, the CPU 170 sends an AI image generation request to the image generation server 101, the AI ​​image generation request including the size of the image to be inserted into the document stored in the RAM 171 in step S1403 and the information about the keyword of the image to be generated which has been set in step S1405. Fig.14 The image insertion process shown proceeds to step S1407.

[0213] After that, the user can continue to edit the document on the general purpose computer 102 until Fig.14 The image insertion process shown proceeds to step S1409.

[0214] In step S1407, the image generation server 101 causes the GPU 166 to execute the image generation program 165 and generate an AI image in response to the AI ​​image generation request received from the general-purpose computer 102 via the LAN 103. In addition, in the case where image generation is already being processed in the image generation server 101 due to another AI image generation request, the previously received request will be processed in order, and the processing of the relevant request will wait until it is its turn. After that, Fig.14 The image insertion process shown proceeds to step S1408.

[0215] In step S1408, the image generation server 101 transmits the image data of the AI ​​image generated in step S1407 to the general-purpose computer 102 via the LAN 103. Thereafter, Fig.14 The image insertion process shown proceeds to step S1409.

[0216] In step S1409, the AI ​​image obtained from the image generation server 101 is inserted into the position on the document image stored in the RAM 171 in step S1403, and Fig.14 The image insertion process shown ends.

[0217] In step S1410, an error screen is displayed, which notifies the user that a sufficiently large area for inserting the AI ​​image into the document is not found, and the image generation server 101 does not generate the AI ​​image, and Fig.14 The image insertion process shown ends.

[0218] Although details are omitted, various configurations described in the first and second embodiments are also applicable to the third embodiment.

[0219] Through the above process, in the third embodiment, the CPU 170 determines the area where the AI ​​image should be inserted and the keywords to be used when generating the AI ​​image based on the document image, and based on this information, the image generation server 101 generates the AI ​​image and inserts it into the document. As a result, in the third embodiment, the labor of the user to edit the document can be further reduced compared with the first and second embodiments.

[0220] It should be noted that in the embodiments of the present invention, it is also possible to implement such a process in which a program for implementing one or more functions is supplied to a computer of a system or device via a network or a storage medium, and a system control unit in the system or device reads out and executes the program. The system control unit may include one or more processors or circuits, and in order to read out and execute executable instructions, the system control unit may include a plurality of isolated system control units or a network of a plurality of isolated processors or circuits.

[0221] The processor or circuit may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC) and / or a field programmable gate array (FPGA). In addition, the processor or circuit may include a digital signal processor (DSP), a data flow processor (DFP) or a neural processing unit (NPU).

[0222] Other embodiments

[0223] The embodiments of the present invention may also be implemented by reading and executing computer executable instructions (e.g., one or more programs) recorded on a storage medium (also more completely referred to as a "non-transitory computer-readable storage medium") to perform one or more functions in the above-mentioned embodiments, and / or a computer of a system or device including one or more circuits (e.g., an application-specific integrated circuit (ASIC)) for performing one or more functions in the above-mentioned embodiments, and the embodiments of the present invention may be implemented by, for example, reading and executing the computer executable instructions from the storage medium by the computer of the system or device to perform one or more functions in the above-mentioned embodiments, and / or controlling the one or more circuits to perform one or more functions in the above-mentioned embodiments. The computer may include one or more processors (e.g., a central processing unit (CPU), a microprocessing unit (MPU)), and may include a network of separate computers or separate processors to read and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium. The storage medium may include, for example, a hard disk, a random access memory (RAM), a read-only memory (ROM), a memory of a distributed computing system, an optical disk (such as a compact disc (CD), a digital versatile disc (DVD), or a Blu-ray disc (BD) TM ), flash memory devices, memory cards, etc.

[0224] The embodiments of the present invention may also be implemented by providing software (program) for performing the functions of the above-described embodiments to a system or device via a network or various storage media, and a computer or a central processing unit (CPU) or a microprocessing unit (MPU) of the system or device reads and executes the program.

[0225] While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the following claims is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures and functions.

[0226] This application claims the benefit of Japanese Patent Application No. 2023-147574, filed September 12, 2023, which is hereby incorporated by reference herein in its entirety.

Claims

1. An information processing device, comprising: an area designating unit that designates an area for inserting an AI image from an image; A first keyword obtaining unit, which obtains a keyword of the AI ​​image; An AI image obtaining section that obtains an AI image generated by inputting input data including the obtained keyword into a training model; and An inserting unit inserts the obtained AI image into a designated area of ​​the image.

2. The information processing device according to claim 1, wherein: In response to an instruction from a user to generate an AI image to be inserted into an image, displaying an image on an area designation UI and accepting a user designation of an area in the displayed image through the area designation UI, and A user input of information regarding the AI ​​image is accepted through the AI ​​image information designation UI.

3. The information processing device according to claim 2, wherein: The area designation section The position of the area is determined based on the coordinates specified by the user for the image displayed on the area specifying UI, and The size of the area input by the user through the AI ​​image information designation UI is obtained.

4. The information processing device according to claim 2, wherein: The area specifying section determines the position and size of the area based on a vector specified by a user with respect to an image displayed on the area specifying UI.

5. The information processing device according to claim 2, wherein: The area designation UI performs an enlarged display of a portion of an image in response to a user operation.

6. The information processing device according to claim 1, wherein: The area specifying section specifies an area based on a blank area detected by analyzing an image.

7. The information processing device according to claim 2, wherein: The first keyword obtaining unit obtains a keyword input by a user through an AI image information designation UI.

8. The information processing device according to claim 2, wherein: The AI ​​image is displayed as a generated image in a designated area of ​​the image displayed on the area designation UI.

9. The information processing device according to claim 8, wherein: The input data includes a seed value, When a user selects the generated image displayed on the area designation UI, the AI ​​image acquisition unit acquires an AI image regenerated by inputting input data including a seed value different from the seed value, a size of the designated area, and the acquired keyword into a training model, The inserting unit replaces the generated image with the regenerated AI image.

10. The information processing device according to claim 9, wherein: The generated image displayed on the area designation UI is updated to the newly generated AI image.

11. The information processing device according to claim 8, wherein: The input data includes a seed value, When the user selects an image that has already been generated and is displayed on the area designation UI, the user input of a regeneration keyword is accepted through the redesignation UI. The AI ​​image obtaining section obtains an AI image regenerated by inputting input data including a seed value, a size of a designated area, and the obtained regeneration keyword into a training model, and The inserting unit replaces the generated image with the regenerated AI image.

12. The information processing device according to claim 11, wherein: The generated image displayed on the area designation UI is updated to the newly generated AI image.

13. The information processing device according to any one of claims 1 to 12, wherein: The first keyword obtaining unit In response to an instruction from a user to generate an AI image to be inserted into an image, accepting user designation of an object item from which keywords are to be extracted through a keyword extraction object designation UI, and Keywords are extracted by performing text analysis on user-specified object items of the image.

14. The information processing device according to any one of claims 1 to 12, wherein: The first keyword obtaining unit In response to an instruction from a user to generate an AI image to be inserted into an image, accepting, through a keyword extraction range designation UI, a user designation of one of the same page, a designated page, a designated portion, the entire document, and a free selection as a range for extracting keywords from the image, The user specifies a target item from which keywords are to be extracted through the keyword extraction target specifying UI, and Keywords are extracted by performing text analysis on a user-specified object item in a user-specified range of an image.

15. The information processing device according to claim 1, wherein: The AI ​​image acquisition unit Sending an AI image generation request including the size of the designated area and the obtained keyword to the image generation server, and An AI image generated in response to the AI ​​image generation request is obtained from the image generation server.

16. The information processing device according to claim 15, wherein: The information processing apparatus is an image forming apparatus.

17. The information processing device according to claim 16, further comprising: The image acquisition unit acquires an image by reading the original image with the reading unit.

18. A method for controlling an information processing device, comprising: A region specifying step, specifying a region from the image for inserting the AI ​​image; The first keyword obtaining step is to obtain keywords of the AI ​​image; an AI image obtaining step of obtaining an AI image generated by inputting input data including the obtained keyword into a training model; and The inserting step inserts the obtained AI image into a designated area of ​​the image.

19. A non-transitory computer-readable storage medium storing a program for causing a computer to execute a control method for an information processing device, the control method comprising: A region specifying step, specifying a region from the image for inserting the AI ​​image; The first keyword obtaining step is to obtain keywords of the AI ​​image; an AI image obtaining step of obtaining an AI image generated by inputting input data including the obtained keyword into a training model; and The inserting step inserts the obtained AI image into a designated area of ​​the image.

Images