Image processing device

Abstract

To obtain an image processing device capable of executing desired processing to a desired part in an original with an intuitive operation by a user even if information on processing at generation of the original of a processing object is not contained.SOLUTION: An objective image acquisition part 11 acquires an original image of an original, an additional object detection part 12 detects an additional object additionally written to the original. A processing type specification part 13 specifies a processing type corresponded to a retouch command contained in the detected additional object from a plurality of predetermined processing types. A processing execution part 14 uses the image processing device or an external server to execute processing of the specified processing type for a processing object region specified on the basis of the detected additional object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] One image processing device identifies a processing item specified by a specified color in the processing list for a document image obtained by scanning a document containing the processing items "delete," "acquire," and "restrict," and executes the identified processing item on the portion of the document image that is surrounded or filled with the specified color (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-27243 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the case of the above-mentioned image processing device, it is necessary to include the above-mentioned processing list in the document to be processed in advance. The user must include the processing list in the document when creating the document, or additionally print the processing list on the document after creating the document, and further specify the processing with a marker or the like. If the processing list is included in the document when creating the document, objects unrelated to the original contents of the document are printed in the document, which is not preferable. Also, if the processing list is additionally printed on the document after creating the document, it is necessary to execute the printing and then specify the processing with a marker or the like, which is time-consuming for the user and not practical.

[0005] The present invention has been made in consideration of the above problems, and aims to provide an image processing device that allows a user to intuitively perform desired processing on desired parts of a document, without having to include information related to the processing when creating the document to be processed. [Means for solving the problem]

[0006] The image processing device of the present invention includes a target image acquisition unit that acquires a document image of a document, an added object detection unit that detects an additional object that has been additionally written on the document, a processing type identification unit that identifies a processing type from among a predetermined number of processing types that corresponds to an addition command contained in the detected additional object, and a processing execution unit that uses the image processing device or an external server to execute processing of the identified processing type on a processing target area identified based on the detected additional object. Effect of the Invention

[0007] According to the present invention, an image processing device is obtained that allows a user to intuitively execute desired processing on a desired portion of a document, even if information related to the processing is not included when the document to be processed is created.

[0008] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing a configuration of an image processing device according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a diagram showing an example of a document, and an example of a revision command and an ROI command as additional objects additionally written on the document. [Diagram 3] FIG. 3 is a diagram for explaining the association between the detected revision command and the ROI command. [Figure 4]FIG. 4 is a diagram showing examples of processing types and revision commands. [Diagram 5] FIG. 5 is a flowchart illustrating the operation of the image processing device shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] Fig. 1 is a block diagram showing the configuration of an image processing device according to an embodiment of the present invention. The image processing device shown in Fig. 1 is an information processing device such as a personal computer, or an electronic device such as a digital camera or an image forming device (scanner, multifunction machine, etc.), and includes an arithmetic processing device 1, a storage device 2, a communication device 3, a display device 4, an input device 5, an internal device 6, etc.

[0012] The arithmetic processing device 1 includes a computer, and executes programs on the computer to operate as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like, and operates as a specific processing unit by loading a program stored in the ROM or storage device 2 into the RAM and executing the program on the CPU. The arithmetic processing device 1 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0013] The storage device 2 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processes described below. Setting data 2a is stored in the storage device 2. The setting data 2a includes parameters of a learning device described below, and data on the correspondence between revision commands and processing types.

[0014] The communication device 3 is a device that performs data communication with an external device, such as a network interface, a peripheral device interface, etc. The display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display, etc. The input device 5 is a device that detects user operations, such as a keyboard, a touch panel, etc.

[0015] The internal device 6 is a device that executes a predetermined function of the image processing device. For example, if the image processing device is an image forming device, the internal device 6 includes an image reading device that optically reads an original image from an original, a printing device that prints an image on a printing paper, and the like.

[0016] Here, the calculation processing device 1 operates as the above-mentioned processing units, namely, a target image acquisition unit 11, an additional object detection unit 12, a processing type identification unit 13, a processing execution unit 14, an output processing unit 15, an error processing unit 16, and a command registration unit 17.

[0017] The target image acquisition unit 11 acquires (image data of) an original image of a certain original from the storage device 2, the communication device 3, the internal device 6, etc., and stores it in a RAM, etc. For example, this original is a print product output from a printing device, and this original image is read from the original by an image reading device. This original is, for example, a slip.

[0018] The additional object detection unit 12 detects additional objects that are additionally written on the above-mentioned document.

[0019] The additional object detection unit 12 may detect the additional object additionally written on the above-mentioned manuscript in the manuscript image including the above-mentioned additional object, or may detect the additional object additionally written on the above-mentioned manuscript in a difference image between the manuscript image including the above-mentioned additional object and the original manuscript image not including the above-mentioned additional object. Also, when the original manuscript image is a monochrome image and the additional object is added by handwriting with a color writing implement such as a pen, the additional object may be detected by extracting the writing color (chromatic color) of the writing implement from the manuscript image including the above-mentioned additional object. For example, the writing color is specified by a coordinate value in the HSV color space.

[0020] The process type identifying unit 13 identifies, from among a plurality of predetermined process types, a process type that corresponds to a retouching command included in the detected added object.

[0021] Here, the revision command is not character information but image information, and is at least one of image information of characters, symbols, and figures.

[0022] In this embodiment, the added object detection unit 12 and the processing type identification unit 13 detect the added object and identify the processing type by object detection using a machine-learned learning device. Note that this learning device is a deep neural network or the like built into the image processing device, and is machine-learned in advance by an existing machine learning method, using, for example, a large number of document images to which retouching commands and ROI (Region Of Interest) commands have been added as input data and classifications of the retouching commands and ROI commands as output data.

[0023] In other words, in order to extract the addition command, OCR (Optical Character Recognition) processing is not performed on the addition object detected by the addition object detection unit 12, and the addition object is directly classified as an addition command based on the image information (image features) of the addition object.

[0024] FIG. 2 is a diagram showing an example of a document, and an example of a revision command and an ROI command as additional objects additionally written on the document.

[0025] For example, a revision command 111 and an ROI command 112 are revised on a document 101 as shown in Fig. 2. For example, this revision is done by handwriting using a writing implement such as a marker of a predetermined color.

[0026] 2 is the character string “OCR.” Also, the ROI command 112 shown in Fig. 2 is a pair of brackets, and is used to specify a processing target area for the processing specified by the revision command 111.

[0027] In the detection of the added object described above, the bounding boxes (circumscribing rectangles) 121, 122 of the added object are detected, and in the identification of the processing type described above, an added object having specific image features is recognized as an add command 111, and an added object having specific image features is recognized as an ROI command 112 (a pair of an image indicating the upper left corner of the processing target area and an image indicating the lower right corner of the processing target area).

[0028] 3 is a diagram for explaining the association of the detected retouch command and ROI command. Then, for example, as shown in FIG. 3, the bounding box 122 of the ROI command 112 is set as the processing target area, and the retouch command 111 and the processing target area are associated with each other based on the distance D between the center of gravity of the bounding box 122 (processing target area) and the center of gravity of the bounding box 121 of the retouch command 111. In other words, the process specified by the retouch command 111 is executed on this processing target area. For example, when the distance D is equal to or less than a predetermined value, the retouch command 111 and the processing target area are associated with each other.

[0029] Fig. 4 is a diagram showing examples of processing types and revision commands In this embodiment, for example, as shown in Fig. 4, a plurality of processing types that can be specified by a plurality of different revision commands are set in the setting data 2a.

[0030] The multiple processing types include image insertion, character information extraction, data structuring, image generation, and the like.

[0031] For example, as shown in FIG. 4, the process type "insert image" corresponding to the retouch command "ERASE" executes a process of overwriting the processing target area with the background color (i.e., erasing the objects in the processing target area).

[0032] For example, as shown in FIG. 4, the process type "insert image" corresponding to the retouch command "MASK" executes a process (mask process) of overwriting the processing target area with a specific color (such as black).

[0033] For example, as shown in FIG. 4, the process type "text information extraction" corresponding to the retouching command "OCR" executes OCR processing of type on the image in the processing target area.

[0034] For example, as shown in FIG. 4, the process type "character information extraction" corresponding to the retouch command "AIOCR" executes OCR processing of handwritten characters on the image in the processing target area.

[0035] For example, as shown in Figure 4, the processing type "Extract Character Information" corresponding to the retouching command "FILENAME" executes a process to save image data of a manuscript, etc., using the character information (text data) obtained by OCR processing of the image in the processing target area as the file name.

[0036] For example, as shown in Figure 4, the processing type "data structuring" corresponding to the retouching command "TABLE" extracts a table from the image of the area to be processed and saves the data in the table as CSV data or the like.

[0037] For example, as shown in Figure 4, the processing type "image generation" corresponding to the retouching command "@AI" executes a process in which an image related to character information (text data) obtained by OCR processing of the image in the processing target area is generated using a diffusion model or the like, and the image is fitted into the processing target area.

[0038] The processing execution unit 14 executes the processing of the specified processing type on the processing target area specified based on the detected additional object using the image processing device or an external server. When executing the processing using an external server, the processing execution unit 14 transmits a processing request and data required for the processing to the external server via the communication device 3, and receives the processing result from the external server via the communication device 3.

[0039] In this embodiment, the process execution unit 14 selects whether to execute the above-mentioned process using the image processing device itself or an external server based on the identified process type. Specifically, the process execution unit 14 causes the external server to execute the process for high-load processes, and executes the process for low-load processes in the image processing device itself without using the external server.

[0040] For example, when the identified type of processing is any one of the above-mentioned image insertion, OCR processing of a printed character target, and data structuring, the processing execution unit 14 selects to execute the above-mentioned processing using the image processing device.

[0041] Furthermore, for example, when the identified process type is either the above-mentioned image generation or OCR process for handwritten characters, the process execution unit 14 selects to execute the above-mentioned process using an external server.

[0042] The output processing unit 15 outputs (prints, transmits data, etc.) the document image after the above-mentioned processing, outputs the data obtained by the above-mentioned processing, and displays the results of the above-mentioned processing on the display device 4.

[0043] If an error occurs during the above-mentioned detection of the additional object and the specification of the processing type, the error processing unit 16 specifies the cause and type of the error and displays them on the display device 4.

[0044] The command registration unit 17 performs the registration of revision commands desired by the user.

[0045] Next, a description will be given of the operation of the image processing device shown in Fig. 1. Fig. 5 is a flow chart illustrating the operation of the image processing device shown in Fig. 1.

[0046] When the target image acquisition unit 11 acquires a document image (with the retouch command and ROI command written) (step S1), the additional object detection unit 12 detects additional objects that have been written in addition to the above-mentioned document (step S2), and the processing type identification unit 13 identifies the processing target area and attempts to classify the additional objects into retouch commands, and determines whether the retouch command is identified (i.e., whether the processing specified by the retouch command can be executed) (step S3).

[0047] When an add command is identified, the processing type identification unit 13 identifies the processing type and processing corresponding to the add command (step S4), and the processing execution unit 14 executes the processing of the processing type corresponding to the add command on the processing target area identified based on the detected added object, as described above (step S5).

[0048] Then, the process execution unit 14 displays the process result on the display device 4 (step S6), and performs output process (printing, data transmission, data storage, etc.) of the processed document image and the generated data (step S7).

[0049] On the other hand, if detection of an added object or specification of an addition command fails in step S3, the error processor 16 performs an error analysis and displays the cause of the error, the type of the error, etc. on the display device 4 (step S8).

[0050] As described above, according to the embodiment, the target image acquisition unit 11 acquires a document image of a document, the additional object detection unit 12 detects an additional object additionally written on the document, and the process type identification unit 13 identifies a process type from among a plurality of predetermined process types that corresponds to a retouch command included in the detected additional object. The process execution unit 14 uses the image processing device or an external server to execute a process of the identified process type on a processing target area identified based on the detected additional object.

[0051] This allows the user to intuitively execute a desired process on a desired portion of a document, without needing to include information regarding the process when creating the document to be processed.

[0052] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, such changes and modifications are intended to be included within the scope of the claims. [Industrial Applicability]

[0053] The present invention is applicable, for example, to an image processing apparatus that executes processing desired by a user on an original image. [Explanation of symbols]

[0054] 11. Target image acquisition section 12 Additional object detection section 13 Processing type specification section 14 Processing Execution Unit

Claims

1. In the image processing device, a target image acquisition unit for acquiring a document image of a document; an additional object detection unit that detects an additional object additionally written on the document; a process type identification unit that identifies a process type corresponding to a retouching command included in the detected additional object from among a plurality of predetermined process types; a processing execution unit that executes, using the image processing device or an external server, a processing of the identified processing type on a processing target area that is identified based on the detected additional object; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the touch-up command is image information rather than text information.

3. 3. The image processing apparatus according to claim 2, wherein the touch-up command is at least one of image information of a character, a symbol, and a graphic.

4. 4. The image processing device according to claim 2, wherein the additional object detection unit and the processing type identification unit detect the additional object and identify the processing type by object detection using a machine learning learning device.

5. 2 . The image processing apparatus according to claim 1 , wherein the process execution unit selects whether to execute the process using the image processing apparatus itself or the external server, based on the identified process type.

6. the processing execution unit, when the specified processing type is any one of image insertion, OCR processing of a typed object, and data structuring, selects to execute the processing using the image processing device; 6. The image processing device according to claim 5,

7. the process execution unit, when the specified process type is either image generation or OCR process for handwritten characters, selects to execute the process using the external server; 7. The image processing device according to claim 5 or 6,

Images