Image processing device

Abstract

In the present invention, an additional object detection unit (12) detects an additional object additionally written into a manuscript, identifies an addition command and a processing target region designation command included in the detected additional object, and identifies the processing target region designation command corresponding to the addition command. A processing execution unit (14) executes, on the corresponding processing target region, processing of a processing type corresponding to the identified addition command. The processing type is image generation processing. The image generation processing involves: (a) extracting text information from an image of the processing target region; (b) generating an image corresponding to the extracted text information to acquire a generated image; and (c) inserting the generated image into the processing target region.

Description

Image Processing Device

[0001] The present invention relates to an image processing device.

[0002] One automatic coloring device colors a line drawing based on hint information, which is information that specifies colors using dots, line segments, etc. (see, for example, Patent Document 1).

[0003] International Publication No. 2018 / 203374

[0004] However, although the above-mentioned automatic coloring device can color objects in a target image, it is difficult for the user to add a new image object desired by the user.

[0005] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an image processing device that can automatically generate and add a new image object desired by a user to a target image.

[0006] The image processing device according to the present invention includes a target image acquisition unit that acquires a document image of a document, an additional object detection unit that (a) detects an additional object written on the document, (b) identifies a retouch command and a processing target area designation command included in the detected additional object, and (c) identifies the processing target area designation command corresponding to the retouch command, a processing type identification unit that identifies a processing type corresponding to the identified retouch command, and a processing execution unit that uses the image processing device or an external server to execute a process of the processing type corresponding to the identified retouch command on a processing target area designated by the processing target area designation command corresponding to the retouch command. The processing type is a process of (a) extracting text information from an image of the processing target area, (b) generating an image corresponding to the extracted text information to obtain a generated image, and (c) inserting the generated image into the processing target area.

[0007] According to the present invention, an image processing device can be provided that can automatically generate and add a new image object desired by a user to a target image.

[0008] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0009] FIG. 1 is a block diagram illustrating a configuration of an image processing device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a document and an example of a retouch command and an ROI command as additional objects additionally written on the document. FIG. 3 is a diagram illustrating the association of detected retouch commands and ROI commands. FIG. 4 is a diagram illustrating an example of a document image read from a document to which retouch commands and ROI commands have been added in an image generation process. FIG. 5 is a diagram illustrating an example of image generation process. FIG. 6 is a diagram illustrating an example of a document image after the image generation process has been executed. FIG. 7 is a diagram illustrating an example of a display of a document image after a generated image has been inserted. FIG. 8 is a flowchart illustrating the operation of the image processing device illustrated in FIG. 1. FIG. 9 is a flowchart illustrating details of the execution of the process in FIG. 8. FIG. 10 is a diagram illustrating an example of a document image read from a document to which retouch commands and ROI commands have been added in an image generation process according to a second embodiment. FIG. 11 is a diagram illustrating an example of image generation process according to the second embodiment. FIG. 12 is a diagram illustrating an example of a document image after the image generation process according to the second embodiment has been executed.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] Embodiment 1.

[0012] 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Fig. 1 is an information processing apparatus such as a personal computer, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction peripheral, etc.), and includes a processing unit 1, a storage unit 2, a communication unit 3, a display unit 4, an input unit 5, an internal unit 6, etc.

[0013] The arithmetic processing device 1 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 2 into the RAM and executing it on the CPU. The arithmetic processing device 1 may also include an application specific integrated circuit (ASIC) that functions as a specific processing unit.

[0014] The storage device 2 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processes described below. The storage device 2 stores setting data 2a. The setting data 2a includes parameters for the learning device described below, data on the correspondence between revision commands and processing types, and the like.

[0015] The communication device 3 is a device that performs data communication with external devices, such as a network interface, a peripheral device interface, etc. The display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display, etc. The input device 5 is a device that detects user operations, such as a keyboard, a touch panel, etc.

[0016] The internal device 6 is a device that executes a predetermined function of the image processing device. For example, if the image processing device is an image forming device, the internal device 6 includes an image reading device that optically reads an original image from an original, a printing device that prints an image on printing paper, etc.

[0017] Here, the processing device 1 operates as the above-mentioned processing units, namely, a target image acquisition unit 11, an additional object detection unit 12, a processing type identification unit 13, a processing execution unit 14, an output processing unit 15, an error processing unit 16, and a command registration unit 17.

[0018] The target image acquisition unit 11 acquires (image data of) a document image from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a slip.

[0019] The additional object detection unit 12 (a) detects additional objects that have been written in addition to the above-mentioned manuscript, (b) identifies the retouching command and processing target area designation command (ROI command described below) contained in the detected additional objects, and (c) identifies the processing target area designation command corresponding to the retouching command (i.e., the corresponding processing target area).

[0020] The additional object detection unit 12 may detect the additional object written in addition to the original document in an original document image including the additional object, or may detect the additional object written in addition to the original document in a difference image between the original document image including the additional object and the original document image not including the additional object. Furthermore, if the original document image is a monochrome image and the additional object is added by hand with a colored writing implement such as a pen, the additional object may be detected by extracting the writing color (chromatic color) of the writing implement from the original document image including the additional object. For example, the writing color is specified by coordinate values ​​in the HSV color space and registered in advance.

[0021] FIG. 2 shows an example of a manuscript and an example of a retouch command and an ROI command as additional objects written on the manuscript. For example, a retouch command 111 and ROI commands 112a and 112b are retouched on a manuscript 101 as shown in FIG. 2. For example, this retouching is performed by hand using a writing implement such as a marker of a predetermined color. The ROI commands 112a and 112b are processing target area designation commands. The retouch command 111 shown in FIG. 2 is the character "OCR." The ROI commands 112a and 112b shown in FIG. 2 are a pair of brackets, and are used to designate the processing target area of ​​the processing specified by the retouch command 111.

[0022] Here, the retouch command is not text information but image information, and is at least one of text, symbols, and graphics. The processing target area designation command is also image information, and is at least one of text, symbols, and graphics.

[0023] In other words, in order to extract the addition command, OCR processing is not performed on the addition object detected by the addition object detection unit 12, and the addition object is directly classified as an addition command based on the image information (image features) of the addition object.

[0024] The additional object detection unit 12 may classify detected additional objects into retouching commands and processing target area specification commands through object detection using a machine-learned learner. This learner may be a deep neural network or the like built into the image processing device, and may be, for example, a deep neural network trained in advance by an existing machine learning method, with additional objects for each manuscript image as input data and classifications (command IDs) of retouching commands and ROI commands as output data. When recognizing retouching commands through object detection, even if a user makes a minor error, such as a spelling mistake, when entering a retouching command into a manuscript, the command is likely to be recognized as the user's intended retouching command without the need to refer to a dictionary, and the user's intended retouching command can be recognized quickly and accurately.

[0025] The region-of-interest (ROI) specification command includes a pair of mutually different first and second commands (ROI command 112a and 112b). The region-of-interest (ROI) specification command is a bounding box of the pair of first and second ROI commands (ROI command 112a and 112b). Here, the first command (ROI command 112a) is an open bracket, and the second command (ROI command 112b) is a closed bracket.

[0026] In the detection of the additional objects described above, the bounding boxes (circumscribing rectangles) 121, 122a, and 122b of the additional objects are derived, and in the identification of the processing type described above, the additional objects having specific image features are recognized as retouching commands 111, and the additional objects having specific image features are recognized as ROI commands 112a and 112b (pairs of images showing the upper left corner and the lower right corner of the processing target area).

[0027] FIG. 3 is a diagram illustrating the association of detected retouch commands and ROI commands. Then, for example, as shown in FIG. 3 , the bounding boxes 122 of the ROI commands 112 a and 112 b are set as the processing target region, and the retouch command 111 and the processing target region are associated with each other based on the distance D between the center of gravity of the bounding box 122 (processing target region) and the center of gravity of the bounding box 121 of the retouch command 111. That is, the processing specified by the retouch command 111 may be executed on this processing target region. For example, the retouch command 111 and the processing target region may be associated with each other if the distance D is equal to or less than a predetermined value. Furthermore, when there are multiple processing target regions and multiple retouch commands, the processing target region (processing target region designation command) with the shortest distance between the center position of the retouch command (the center position of the bounding box of the retouch command) and the center position of the processing target region is identified as the processing target region (processing target region designation command) corresponding to that retouch command.

[0028] 1 , the process type identification unit 13 identifies a process type corresponding to the identified retouch command from among a predetermined plurality of process types. For example, a process ID indicating a process type is associated with the command ID of the retouch command and registered in advance, and the process type identification unit 13 identifies the process type of the process ID corresponding to the command ID of the detected retouch command as the process type corresponding to the retouch command.

[0029] In this embodiment, a plurality of processing types that can be specified by a plurality of different revision commands are set in the setting data 2a.

[0030] The plurality of processing types includes an image generation process that (a) extracts text information from an image in a processing target area, (b) generates an image corresponding to the extracted text information to obtain a generated image, and (c) inserts the generated image into the processing target area.

[0031] Fig. 4 is a diagram showing an example of a document image read from a document to which a retouch command and an ROI command for the image generation process have been added, Fig. 5 is a diagram showing an example of the image generation process, and Fig. 6 is a diagram showing an example of a document image after the image generation process has been executed.

[0032] In this embodiment, the retouching command corresponding to the image generation process is the retouching command "@AI." For example, as shown in FIG. 4 , a retouching command 211 and an ROI command 212 are added, and further, hint information 213 is added to the processing target area specified by the ROI command 212. Here, in the image generation process, as shown in FIG. 5 , for example, (a) OCR processing is performed on the image of the processing target area to extract the hint information 213 as text information (character code string), (b) a generated image 222 is generated by performing image generation corresponding to the extracted text information using a machine-learned learning device such as a diffusion model, (c) the retouching command 211, the ROI command 212, and the hint information 213 are erased, for example, by filling in the background color, and (d) the generated image 222 is inserted into the processing target area 221 (after scaling the generated image as necessary), as shown in FIG. 6 .

[0033] Here, the hint information 213 is handwritten text information, and is a word or text that indicates an object in the generated image. For example, the hint information 213 in Fig. 4 is text information indicating a Christmas tree.

[0034] Furthermore, the above-mentioned multiple processing types include image insertion, character information extraction, structured data generation, and the like.

[0035] For example, the "image insertion" process type corresponding to the retouching command "ERASE" executes a process of overwriting the processing target area with the background color (i.e., erasing the objects in the processing target area). For example, the "image insertion" process type corresponding to the retouching command "MASK" executes a process of overwriting the processing target area with a specific color (such as black) (masking process). For example, the "image insertion" process type corresponding to the retouching command "STAMP" executes a process of fitting a specific image (a seal image) that has been specified in advance into the processing target area.

[0036] For example, the "text information extraction" process type corresponding to the "OCR" retouch command executes OCR processing on the image of the processing target area. For example, the "text information extraction" process type corresponding to the "FILENAME" retouch command executes a process of saving image data of the original document, using the text information (text data) obtained by OCR processing on the image of the processing target area as a file name. The TAG process type is a process of generating character strings (text information) included in the processing target area by OCR processing when the original document image is converted into PDF format data, and setting the text information as a search tag.

[0037] Also, for example, in the process type "structured data generation process" corresponding to the revision command "TABLE," a process is executed to generate structured data (tabular data) from the image of the processing target area.

[0038] 1 , the processing execution unit 14 uses the image processing device or an external server to execute processing of the processing type corresponding to the identified retouch command (command ID) on the processing target area specified by the processing target area specification command corresponding to the retouch command. When executing the processing using an external server, the processing execution unit 14 transmits a processing request and data required for the processing to the external server via the communication device 3, and receives the processing result from the external server via the communication device 3.

[0039] When executing the image generation process described above, the process execution unit 14 assigns parameters to a learning device that has been machine-learned using a diffusion model and generates an image using the learning device. The parameters specify the style of the generated image (hand-drawn style, photo-realistic style, etc.), and may be specified in advance as a user setting or may be specified in response to a user operation on the input device 5.

[0040] In this embodiment, the process execution unit 14 selects whether to execute the above-mentioned process using the image processing device itself or an external server based on the identified process type. Specifically, the process execution unit 14 causes the external server to execute predetermined high-load processes, and executes predetermined low-load processes in the image processing device itself without using the external server.

[0041] The output processing unit 15 also outputs (prints, transmits data, etc.) the document image after the above processing, outputs the data obtained by the above processing, and displays the results of the above processing on the display device 4.

[0042] FIG. 7 is a diagram showing an example of a display of an original image after a generated image has been inserted. For example, as shown in FIG. 7 , the output processing unit 15 displays an original image 301 after the generated image has been inserted on the display device 4, and further displays an operation menu 302 along with the original image 301. The operation menu 302 is a menu from which the user can select items: (a) printing the original image after the generated image has been inserted, (b) data output of the original image after the generated image has been inserted, (c) discarding the original image after the generated image has been inserted, and regenerating the generated image. Each item in the operation menu 302 is displayed as a soft key, and a process (e.g., print, data output, discard, regenerate) corresponding to the soft key pressed by the user is executed. Specifically, when a soft key in the operation menu 302 is pressed, the output processing unit 15 identifies the process corresponding to the pressed soft key and causes the process execution unit 14 to execute the identified process.

[0043] Furthermore, if an error occurs in the detection of the added object or the identification of the processing type, the error processing unit 16 identifies the cause and type of the error and displays them on the display device 4. The command registration unit 17 registers retouching commands desired by the user.

[0044] Next, a description will be given of the operation of the image processing apparatus shown in Fig. 1. Fig. 8 is a flowchart illustrating the operation of the image processing apparatus shown in Fig. 1.

[0045] When the target image acquisition unit 11 acquires a document image (with retouching commands, ROI commands, etc. written on it) (step S1), the additional object detection unit 12 detects additional objects that have been written on the document and identifies the retouching commands and ROI commands from the detected additional objects (step S2).

[0046] When a revision command is identified, the process type identifying unit 13 identifies the process type corresponding to the revision command and the process to be executed (step S3).

[0047] The processing execution unit 14 then identifies the processing target region (position and size) specified by the ROI command for each identified retouch command (step S4), extracts an image of the identified processing target region from the document image (step S5), and executes the identified processing for each identified retouch command on the document image (step S6). After the processing is completed, the output processing unit 15 displays the processing results on the display device 4 and performs output processing (printing, data transmission, data storage, etc.) of the processed document image and the generated data.

[0048] The details of the process executed in step S6 will now be described with reference to a flowchart of FIG.

[0049] The process execution unit 14 determines whether or not the retouch command "@AI" is present in the retouch commands identified in the document image (step S11).

[0050] If there is a retouch command "@AI", the process execution unit 14 executes OCR processing on the image of the processing target area corresponding to the retouch command "@AI" to extract hint information as text information (step S12).

[0051] Next, the processing execution unit 14 sets parameters for the diffusion model (step S13), generates an image using the diffusion model (step S14), and inserts the generated image into the processing target area (step S15).

[0052] Thereafter, if there is another retouching command identified in the document image, the process execution unit 14 executes a process corresponding to that retouching command (step S16).

[0053] In this way, based on the retouch command "@AI", an image is generated based on the hint information written in the processing target area specified by the ROI command in the original image, and the newly generated image is added to the processing target area.

[0054] As described above, according to the first embodiment, the target image acquisition unit 11 acquires a document image of a document. The additional object detection unit 12 (a) detects an additional object written on the document, (b) identifies a retouch command and a processing target area designation command included in the detected additional object, and (c) identifies a processing target area designation command corresponding to the retouch command. The process type identification unit 13 identifies a process type corresponding to the identified retouch command. The process execution unit 14 uses the image processing device or an external server to execute a process of the process type corresponding to the identified retouch command on the processing target area designated by the processing target area designation command corresponding to the retouch command. The process type is an image generation process, which (a) extracts text information from an image of the processing target area, (b) generates an image corresponding to the extracted text information to obtain a generated image, and (c) inserts the generated image into the processing target area.

[0055] This allows a new image object desired by the user to be automatically generated and added to the target image.

[0056] Embodiment 2.

[0057] In the second embodiment, the image generation process described above is a process of (a) extracting text information from a processing target area, (b) performing image generation corresponding to the extracted text information and an image object in the processing target area to obtain a generated image, and (c) inserting the generated image into the processing target area. More specifically, the image generation process is a process of (a) extracting text information from an image of a first processing target area among the processing target areas, (b) performing image generation corresponding to the extracted text information and an image object in a second processing target area among the processing target areas to obtain a generated image, and (c) inserting the generated image into the second processing target area.

[0058] The image object in the second processing target area is a schematic diagram showing the composition of the generated image, and is added by hand to the document, for example.

[0059] Fig. 10 is a diagram showing an example of a document image read from a document to which a touch-up command and an ROI command have been added in the image generation process according to embodiment 2. Fig. 11 is a diagram showing an example of the image generation process according to embodiment 2. Fig. 12 is a diagram showing an example of a document image obtained by executing the image generation process according to embodiment 2.

[0060] In the second embodiment, similarly to the first embodiment, a retouch command "@AI" is used as a retouch command corresponding to the image generation process. For example, as shown in Fig. 10 , a retouch command 411, an ROI command 412 specifying a first processing target region, and an ROI command 413 specifying a second processing target region are added (handwritten in this case), and further, first hint information 414 is added to the first processing target region specified by the ROI command 412, and second hint information 415 is added (handwritten in this case) to the second processing target region specified by the ROI command 413.

[0061] The ROI command 412 specifying the first processing target region and the ROI command 413 specifying the second processing target region may have the same shape or different shapes.

[0062] Furthermore, if the shapes of the ROI commands 412 and 413 are different from each other, the ROI commands 412 and 413 may be classified into an ROI command specifying a first processing target region and an ROI command specifying a second processing target region according to their shapes.

[0063] Furthermore, among ROI commands 412 and 413, an ROI command whose processing target area contains text information may be detected as an ROI command specifying a first processing target area, and an ROI command whose processing target area contains an image object such as a figure or picture may be detected as an ROI command specifying a second processing target area.

[0064] Furthermore, among the ROI commands 412 and 413, the ROI command located above the retouch command 411 may be detected as the ROI command specifying the first processing target area, and the ROI command located below the retouch command 211 may be detected as the ROI command specifying the second processing target area.

[0065] Here, in the image generation process, as shown in FIG. 11 , for example, (a) first hint information 414 is extracted as text information (character code string) by performing OCR processing on the image of the first processing target area, (b) a machine-learned learner using a diffusion model or the like is used to generate an image corresponding to the extracted text information and second hint information 415 (an image object within the second processing target area), thereby generating a generated image 422, (c) the retouch command 411, ROI command 412, ROI command 413, and hint information 414 and 415 are erased, for example, by filling in with a background color, and (d) the generated image 422 is inserted into the second processing target area 421 (after scaling the generated image as necessary), as shown in FIG. 12 .

[0066] Here, the first hint information 414 is handwritten text information, which is a word or text indicating an object in the generated image. The second hint information 415 (i.e., an image object in the second processing target area) is a schematic diagram indicating the composition of the generated image. For example, in FIG. 10 , the hint information 414 is text information indicating Christmas trees, and the hint information 415 is a schematic diagram indicating the number of Christmas trees. In other words, the above-mentioned learning device is trained by machine learning using such text information and schematic diagrams as input data.

[0067] Next, the operation of the image processing device according to the second embodiment will be described.

[0068] In embodiment 2, if it is determined that the retouching command "@AI" is present among the retouching commands identified in the original image, the processing execution unit 14 performs OCR processing on the image of the first processing target area corresponding to the retouching command "@AI" to extract first hint information as text information, and extracts second hint information as an image object from the second processing target area corresponding to the retouching command "@AI" (step S12).

[0069] Next, the processing execution unit 14 sets parameters of the diffusion model (step S13), generates an image using the diffusion model based on the first hint information and the second hint information (step S14), and inserts the generated image into the processing target area (step S15).

[0070] The other configurations and operations of the image processing device according to the second embodiment are the same as those of the first embodiment, and therefore the description thereof will be omitted.

[0071] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0072] The present invention is applicable to, for example, an image processing apparatus that executes processing desired by a user on an original image.

Claims

1. An image processing device comprising: a target image acquisition unit that acquires a document image of a document; an additional object detection unit that (a) detects additional objects written on the document, (b) identifies a retouch command and a processing target area designation command contained in the detected additional objects, and (c) identifies the processing target area designation command corresponding to the retouch command; a processing type identification unit that identifies a processing type corresponding to the identified retouch command; and a processing execution unit that uses the image processing device or an external server to execute processing of the processing type corresponding to the identified retouch command on a processing target area designated by the processing target area designation command corresponding to the retouch command, wherein the processing type is a process of (a) extracting text information from an image of the processing target area, (b) performing image generation corresponding to the extracted text information to obtain a generated image, and (c) inserting the generated image into the processing target area.

2. The image processing device according to claim 1, wherein the image of the processing target area includes handwritten hint information, the processing execution unit extracts the handwritten hint information as the text information, and the hint information is words or text indicating an object in the generated image.

3. The image processing device according to claim 1, wherein the processing execution unit assigns parameters to a machine-learned learning device and generates the image using the learning device, and the parameters specify a style of the generated image.

4. An image processing device as described in claim 1, further comprising an output processing unit which (a) displays the original image after the generated image has been inserted, and (b) displays, together with the original image after the generated image has been inserted, a menu which enables the user to select printing the original image after the generated image has been inserted, outputting the data of the original image after the generated image has been inserted, discarding the original image after the generated image has been inserted, and regenerating the generated image.

5. An image processing device as described in claim 1, characterized in that the processing type is a process of (a) extracting text information from the processing target area, (b) performing image generation corresponding to the extracted text information and an image object in the processing target area to obtain a generated image, and (c) inserting the generated image into the processing target area.

6. An image processing device as described in claim 5, characterized in that the processing type is a process of: (a) extracting text information from an image of a first processing target area among the processing target areas; (b) performing image generation corresponding to the extracted text information and an image object in a second processing target area among the processing target areas to obtain a generated image; and (c) inserting the generated image into the second processing target area.

7. An image processing apparatus according to claim 6, wherein said image object is a schematic diagram showing the composition of said generated image.

Images