Image processing apparatus

Abstract

To provide an image processing apparatus that appropriately adds an image desired by a user to a document image.SOLUTION: A text information extraction unit 12 performs character recognition processing on a document image to acquire, from the document image, text data of a text described in the document image. A prompt generation unit 13 generates a prompt for a predetermined image generation model on the basis of the acquired text data. A generated image acquisition unit 14 acquires a generated image corresponding to the generated prompt. A document image editing unit 15 adds the acquired generated image into the document image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] Image generation AI services such as Stable Diffusion generate images that correspond to input text (prompts).

[0003] One information processing method uses an image generation AI to generate an advertising image based on advertising information (company name, product name, etc.) (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2024-43680 Summary of the Invention [Problem to be solved by the invention]

[0005] In the image generation AI services and image generation AI described above, users must input appropriate text (prompts) to obtain the image they desire. When a user wants to add an automatically generated image to a document image, depending on the user's level of proficiency, there is a possibility that the appropriate text (prompt) will not be selected and the appropriate image will not be added.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an image processing apparatus that can appropriately add a user-desired image to a document image. [Means for solving the problem]

[0007] The image processing device of the present invention comprises a document image acquisition unit that acquires a document image, a text information extraction unit that performs character recognition processing on the document image to acquire text data of the text described in the document image from the document image, a prompt generation unit that generates a prompt for a predetermined image generation model based on the acquired text data, a generated image acquisition unit that acquires a generated image corresponding to the generated prompt, and a document image editing unit that adds the acquired generated image to the document image. [Effects of the Invention]

[0008] According to the present invention, an image processing device that appropriately adds an image desired by a user to a document image is provided.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a document image. [Figure 3] FIG. 3 is a diagram illustrating text extracted from the document image shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of text data used to generate a prompt, out of the text data of the lines shown in FIG. [Figure 5] FIG. 5 is a diagram showing an example of a generated image generated using the character string "hamburger" specified by the user as a prompt. [Figure 6] FIG. 6 shows an example of a generated image generated using the character string “hamburger” specified by the user and the character strings “Lunch,” “menu,” “one item,” and “plus” selected from text data extracted from a document image as prompts. [Figure 7]FIG. 7 is a diagram showing an example of a document image to which a generated image has been added. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] Fig. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. The image processing device shown in Fig. 1 is an information processing device such as a personal computer, or an electronic device such as a digital camera or an image forming device (scanner, multifunction peripheral, etc.), and includes an arithmetic processing device 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, etc.

[0013] The arithmetic processing device 1 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 2 into the RAM and executing it on the CPU. The arithmetic processing device 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0014] The storage device 2 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processes described below. The storage device 2 also stores setting data and the like.

[0015] The communication device 3 is a device that performs data communication with external devices, such as a network interface or a peripheral device interface. The display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display. The input device 5 is a device that detects user operations, such as a keyboard or a touch panel.

[0016] The internal device 6 is a device that executes a predetermined function of the image processing device. For example, if the image processing device is an image forming device, the internal device 6 includes an image reading device that optically reads a document image from a document, a printing device that prints an image on a print sheet, etc.

[0017] Here, the processing device 1 operates as the above-mentioned processing units, namely, a document image acquisition unit 11, a text information extraction unit 12, a prompt generation unit 13, a generated image acquisition unit 14, a document image editing unit 15, and an output processing unit 16.

[0018] The document image acquisition unit 11 acquires a document image (image data) of a certain document from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a flyer, etc. Figure 2 is a diagram showing an example of a document image. The document image 101 shown in Figure 2 includes multiple lines of text.

[0019] The text information extraction unit 12 performs character recognition processing on the document image to acquire text data of the text written in the document image 101 from, for example, a document image 101 as shown in Fig. 2. The prompt generation unit 13 generates a prompt (input character string) for a predetermined image generation model based on the text data acquired by the text information extraction unit 12.

[0020] Fig. 3 is a diagram illustrating text extracted from the document image shown in Fig. 2. Here, the text information extraction unit 12 (a) identifies the position where the text is written along with the above-mentioned text data for each piece of text in the document image, and (b) classifies the text data by line based on the position, for example, as shown in Fig. 3. Here, for example, as shown in Fig. 3, line bounding boxes 121 to 125 for each line are identified.

[0021] Furthermore, the text information extraction unit 12 also identifies the text characteristics in the document image, and the prompt generation unit 13 selects text data from the acquired text data based on the identified text characteristics, and generates a prompt for a predetermined image generation model based on the selected text data.

[0022] Here, the text feature is, for example, the size of the text, and the prompt generation unit 13 selects the text data of the line with the largest text size from the acquired text data, and generates a prompt for a predetermined image generation model based on the selected text data. Here, words (such as nouns) in the text data of that line are extracted and included in the prompt. For example, the height of the line bounding boxes 121 to 125 in Figure 3 is set to the size of the text.

[0023] The text characteristics may also include at least one of the text's character type, font, color, weight, slant, character spacing, and line spacing.

[0024] 3, an image addition area 141 to which a generated image should be added is set in the document image 101. The image addition area 141 may be written in advance in the document image 101, or may be set based on a user operation on the input device 5. The prompt generation unit 13 generates a prompt for a predetermined image generation model based on text data of a line of the acquired text data that is within a predetermined distance from the image addition area 141. Here, words (such as nouns) in the text data of that line are extracted and included in the prompt.

[0025] Fig. 4 is a diagram showing an example of text data used to generate a prompt from the text data of the lines shown in Fig. 3. For example, as shown in Fig. 4, the text data of the T line, which is the largest, and the text data of the lines closest to image addition area 141 are used to generate a prompt. In other words, of the acquired text data, text data other than the text data of the lines selected as described above is not used to generate a prompt.

[0026] For example, in the case shown in Figure 4, from the selected line text data "Lunch," "any item from the menu," and "+plus," "Lunch," "Menu," "Item," and "plus" are included in the prompt.

[0027] The prompt generation unit 13 may include a character string desired by the user in the prompt. For example, a character string indicating an object to be included in the generated image, such as “hamburger,” is specified based on a user operation on the input device 5.

[0028] Furthermore, the prompt generation unit 13 may generate a prompt from the above-mentioned character string using a large-scale language model such as a Generative Pretrained Transformer (GPT). For example, the prompt generation unit 13 may access an external server using the communication device 3, input the above-mentioned character string into the large-scale language model on the external server, and acquire the text of the prompt from the large-scale language model on the external server.

[0029] The generated image acquisition unit 14 acquires a generated image corresponding to the generated prompt.

[0030] For example, the generated image acquisition unit 14 may be provided with a learning device (image generation model) for machine-learned images, and the learning device may generate a generated image corresponding to the generated prompt, or the communication device 3 may be used to access an external server, input a prompt into the learning device (image generation model) on the external server, and acquire a generated image from the learning device (image generation model) on the external server.

[0031] Fig. 5 is a diagram showing an example of a generated image generated using the character string "hamburger" specified by the user as a prompt. Fig. 6 is a diagram showing an example of a generated image generated using the character string "hamburger" specified by the user and the character strings "Lunch," "Menu," "One dish," and "Plus" selected from text data extracted from a document image as prompts. Note that the generated images shown in Figs. 5 and 6 were actually generated using an image generation model and are actually color images.

[0032] For example, from the words "hamburger," "Lunch," "menu," "dish," and "plus," the large-scale language model generates multiple prompts, such as "A lunch menu with hamburgers as the main dish," "In addition to the hamburger menu, there are also plenty of side dishes and drinks," and "All of the hamburger menu items are very filling," and multiple generated images corresponding to these multiple prompts are obtained, as shown in Figure 6.

[0033] The document image editing unit 15 adds the acquired generated image to the document image. Here, an image addition area is set in the document image, and the document image editing unit 15 pastes the generated image into the image addition area. The image addition area may be specified by an instruction object in the document image, or may be specified by a user operation on the input device 5. When pasting the generated image into the image addition area, the generated image may be resized or trimmed, if necessary, so that the shape of the generated image matches the shape of the image addition area.

[0034] FIG. 7 is a diagram showing an example of a document image to which a generated image has been added. When the generated image acquisition unit 14 acquires multiple generated images corresponding to the above-described prompt as described above, the document image editing unit 15 may select a generated image specified by the user from the multiple acquired generated images and add the selected generated image to the document image. In this case, for example, the document image editing unit 15 displays a list of multiple generated images as shown in FIGS. 5 and 6 on the display device 4, selects a generated image from the list based on a user operation on the input device 5, and pastes the selected generated image 151 into the image addition area 141, as shown in FIG. 7, for example.

[0035] The output processing unit 16 outputs the document image edited by the document image editing unit 15 (printing, data transmission, data storage, etc.).

[0036] Next, the operation of the image processing device will be described.

[0037] First, the document image acquisition unit 11 acquires image data of a document image from the storage device 2, the communication device 3, the internal device 6, etc. The text information extraction unit 12 performs character recognition processing on the acquired document image to acquire text data of the text described in the document image from the document image.

[0038] Next, the prompt generating unit 13 generates a prompt (input character string) for a predetermined image generation model based on the text data acquired by the text information extracting unit 12, as described above.

[0039] Then, generated image acquisition unit 14 acquires a generated image corresponding to the generated prompt, and document image editing unit 15 adds the acquired generated image to the document image. Output processing unit 16 outputs the document image after being edited by document image editing unit 15.

[0040] As described above, according to the embodiment, the text information extraction unit 12 performs character recognition processing on the document image to acquire text data of the text described in the document image from the document image. The prompt generation unit 13 generates a prompt for a predetermined image generation model based on the acquired text data. The generated image acquisition unit 14 acquires a generated image corresponding to the generated prompt. The document image editing unit 15 adds the acquired generated image to the document image.

[0041] This allows the image desired by the user to be appropriately added to the document image based on the text information in the document image.

[0042] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0043] For example, in the above embodiment, the importance (a numerical value greater than 0 and less than or equal to 1) of a character string to be included in a prompt may be derived according to a predetermined rule based on the text characteristics identified by the text information extraction unit 12, and the prompt may be generated by specifying the importance together with the character string in the large-scale language model. In this case, a threshold value for importance may be set in advance, and the large-scale language model may generate a prompt using character strings whose importance is equal to or greater than the threshold, without using character strings whose importance is less than the threshold. As the rule, for example, a specific color (a conspicuous color such as red) may be set to have a high importance, or the importance may be set according to the density (brightness) of the text or the size of the text. In this way, information that the user is paying attention to is extracted based on the characteristics of the text described in the document image, a prompt based on that information is generated, and ultimately a generated image based on that information is obtained. [Industrial Applicability]

[0044] The present invention is applicable to image processing devices such as multifunction peripherals. [Explanation of symbols]

[0045] 11 Document image acquisition unit 12 Text information extraction section 13 Prompt Generation 14 Generated image acquisition unit 15 Document Image Editing Department

Claims

1. a document image acquisition unit that acquires a document image; a text information extraction unit that performs character recognition processing on the document image to acquire text data of text described in the document image from the document image; a prompt generator that generates a prompt for a predetermined image generation model based on the acquired text data; a generated image acquisition unit that acquires a generated image corresponding to the generated prompt; a document image editing unit that adds the acquired generated image to the document image; An image processing device comprising:

2. The text information extraction unit (a) identifies the text data and a position where the text is written from the document image, and (b) classifies the text data by line based on the position, the prompt generation unit generates a prompt for a predetermined image generation model based on text data of a line within a predetermined distance from an image addition area to which the generated image is to be added, among the acquired text data; 2. The image processing device according to claim 1, wherein:

3. The text information extraction unit (a) identifies, from the document image, the text data and a position where the text is written, and further identifies features of the text in the document image; (b) classifies the text data by line based on the position; the prompt generation unit selects text data from the acquired text data based on features of the text, and generates a prompt for a predetermined image generation model based on the selected text data; 2. The image processing device according to claim 1, wherein:

4. the text characteristic is the size of the text; the prompt generation unit selects text data of a line having the largest text size from the acquired text data, and generates a prompt for a predetermined image generation model based on the selected text data; 4. The image processing device according to claim 3, wherein:

5. 4. The image processing apparatus according to claim 3, wherein the features of the text include at least one of the character type, font, color, thickness, slant, character spacing, and line spacing of the text.

6. 6. The image processing device according to claim 1, wherein the prompt generating unit includes a character string desired by the user in the prompt.

7. the generated image acquisition unit acquires a plurality of generated images corresponding to the generated prompt; the document image editing unit selects a generated image designated by a user from the acquired plurality of generated images, and adds the selected generated image to the document image; 2. The image processing device according to claim 1, wherein:

Images