Image processing device

Abstract

A target image acquisition unit (11) acquires a document image of a document. An additional object detection unit (12) (a) detects an additional object additionally written to the document, (b) identifies a retouch command and a processing target region designation command included in the detected additional object, and (c) identifies a processing target region designation command corresponding to the retouch command. A processing type identification unit (13) identifies a processing type corresponding to the identified retouch command among a plurality of prescribed processing types. A processing execution unit (14) executes the processing of the processing type corresponding to the identified retouch command for the processing target region designated by the processing target region designation command corresponding to the retouch command.

Description

Image Processing Device

[0001] The present invention relates to an image processing device.

[0002] One image processing device identifies the processing items specified by a specified color in the processing list for a document image obtained by scanning a document containing the processing items "delete," "acquire," and "restrict," and executes the identified processing items on the parts of the document image that are surrounded or filled in with the specified color (see, for example, Patent Document 1).

[0003] JP 2009-27243 A

[0004] However, in the case of the image processing device described above, it is necessary to include the above-mentioned processing list in advance in the document to be processed. The user must either include the processing list in the document when creating the document, or additionally print the processing list on the document after creating the document, and then specify the processing using a marker or the like. Including the processing list in the document when creating the document is undesirable because objects unrelated to the original document content are printed in the document. Furthermore, if the processing list is additionally printed on the document after creating the document, the user must first print the list and then specify the processing using a marker or the like, which is time-consuming and unrealistic for the user.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing device that allows a user to intuitively perform desired processing on a desired part of a document, without having to include information related to the processing when creating the document to be processed.

[0006] The image processing device of the present invention comprises a target image acquisition unit that acquires a document image of a document, an additional object detection unit that (a) detects additional objects written on the document, (b) identifies retouch commands and processing target area designation commands contained in the detected additional objects, and (c) identifies the processing target area designation command corresponding to the retouch command, a processing type identification unit that identifies a processing type corresponding to the identified retouch command from among a plurality of specified processing types, and a processing execution unit that uses the image processing device or an external server to execute processing of the processing type corresponding to the identified retouch command on the processing target area designated by the processing target area designation command corresponding to the retouch command.

[0007] According to the present invention, an image processing device is provided that allows a user to intuitively perform desired processing on a desired part of a document, even if information regarding the processing is not included when creating the document to be processed.

[0008] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0009] FIG. 1 is a block diagram illustrating a configuration of an image processing device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a document and an example of a retouch command and an ROI command as additional objects written on the document. FIG. 3 is a diagram illustrating the association of detected retouch commands and ROI commands. FIG. 4 is a diagram illustrating an example of a case where multiple processing target regions are specified by a processing target region specification command. FIG. 5 is a diagram illustrating the association between first commands and second commands when multiple processing target regions are specified by a processing target region specification command (1 / 2). FIG. 6 is a diagram illustrating another example of a case where multiple processing target regions are specified by a processing target region specification command. FIG. 7 is a diagram illustrating the association between first commands and second commands when multiple processing target regions are specified by a processing target region specification command (2 / 2). FIG. 8 is a diagram illustrating an example of a case where multiple retouch commands are located at approximately the same distance from the center position of a certain processing target region. FIG. 9 is a diagram illustrating an example of an editing screen. FIG. 10 is a flowchart illustrating the operation of the image processing device illustrated in FIG. 1. FIG. 11 is a block diagram illustrating the configuration of an image processing device according to a second embodiment of the present invention. FIG. 12 is a diagram illustrating an example of a document on which retouch commands have been entered. FIG. 13 is a diagram showing an example of a list of registered rules. FIG. 14 is a diagram showing an example of a preview display of another document image to which revision commands 111-1 and 111-2 and ROI commands 112-1 and 112-2 have been added based on the rules. FIG. 15 is a diagram showing an example of another document. FIG. 16 is a diagram showing an example of a preview display (initial state) of the document image of the document shown in FIG. 15. FIG. 17 is a diagram showing an example of a preview display (after editing) of the document image of the document shown in FIG. 15. FIG. 18 is a diagram showing an example of a list display to which a new rule has been added. FIG. 19 is a diagram showing an example of a rule registration screen. FIG. 20 is a diagram showing an example of a revision command candidate list. FIG. 21 is a diagram showing an example of a preview display (after the revision command has been changed) of the document image of the document shown in FIG. 12. FIG. 22 is a flowchart for explaining the operation of the image processing device according to the second embodiment.FIG. 23 is a diagram illustrating the association of detected retouch commands and ROI commands in the third embodiment. FIG. 24 is a diagram illustrating an example in which multiple processing target regions are specified along with multiple retouch commands. FIG. 25 is a diagram illustrating an example in which multiple processing target regions are specified along with one retouch command. FIG. 26 is a diagram illustrating an example in which multiple retouch commands are specified for one processing target region (a pair of processing target region specification commands). FIG. 27 is a diagram illustrating an example of argument commands. FIG. 28 is a flowchart illustrating the operation of the image processing device according to the third embodiment. FIG. 29 is a diagram illustrating an example of a document according to the fourth embodiment. FIG. 30 is a diagram illustrating an example of structured data generation processing corresponding to the retouch command "TABLE". FIG. 31 is a diagram illustrating an example of data addition processing. FIG. 32 is a diagram illustrating an example of argument commands for the retouch command "TABLE". FIG. 33 is a diagram illustrating an example of argument commands for the retouch command "CSV". FIG. 34 is a diagram illustrating an example of processing in which one structured data item is generated from multiple document images. FIG. 35 is a flowchart illustrating the operation of the image processing device according to the fourth embodiment. FIG. 36 is a flowchart illustrating the details of the process execution in FIG.

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0011] Embodiment 1.

[0012] 1 is a block diagram showing the configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus shown in Fig. 1 is an information processing apparatus such as a personal computer, or an electronic device such as a digital camera or an image forming apparatus (scanner, multifunction peripheral, etc.), and includes a processing unit 1, a storage unit 2, a communication unit 3, a display unit 4, an input unit 5, an internal unit 6, etc.

[0013] The arithmetic processing device 1 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a central processing unit (CPU), read-only memory (ROM), random access memory (RAM), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 2 into the RAM and executing it on the CPU. The arithmetic processing device 1 may also include an application specific integrated circuit (ASIC) that functions as a specific processing unit.

[0014] The storage device 2 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processes described below. The storage device 2 stores setting data 2a. The setting data 2a includes parameters for the learning device described below, data on the correspondence between revision commands and processing types, and the like.

[0015] The communication device 3 is a device that performs data communication with external devices, such as a network interface, a peripheral device interface, etc. The display device 4 is a device that displays various information to the user, such as a display panel such as a liquid crystal display, etc. The input device 5 is a device that detects user operations, such as a keyboard, a touch panel, etc.

[0016] The internal device 6 is a device that executes a predetermined function of the image processing device. For example, if the image processing device is an image forming device, the internal device 6 includes an image reading device that optically reads an original image from an original, a printing device that prints an image on printing paper, etc.

[0017] Here, the processing device 1 operates as the above-mentioned processing units, namely, a target image acquisition unit 11, an additional object detection unit 12, a processing type identification unit 13, a processing execution unit 14, an output processing unit 15, an error processing unit 16, and a command registration unit 17.

[0018] The target image acquisition unit 11 acquires (image data of) a document image from the storage device 2, communication device 3, internal device 6, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a slip.

[0019] The additional object detection unit 12 (a) detects additional objects that have been written in addition to the above-mentioned manuscript, (b) identifies the retouching command and processing target area designation command (ROI command described below) contained in the detected additional objects, and (c) identifies the processing target area designation command corresponding to the retouching command (i.e., the corresponding processing target area).

[0020] The additional object detection unit 12 may detect the additional object written in addition to the original document in an original document image including the additional object, or may detect the additional object written in addition to the original document in a difference image between the original document image including the additional object and the original document image not including the additional object. Furthermore, if the original document image is a monochrome image and the additional object is added by hand with a colored writing implement such as a pen, the additional object may be detected by extracting the writing color (chromatic color) of the writing implement from the original document image including the additional object. For example, the writing color is specified by coordinate values ​​in the HSV color space.

[0021] The process type specifying unit 13 specifies the process type corresponding to the specified revision command from among a predetermined plurality of process types.

[0022] Here, the touch-up command is not character information but image information, and is image information of at least one of characters, symbols, and figures.

[0023] In this embodiment, the added object detection unit 12 and the processing type identification unit 13 detect added objects and identify processing types by object detection using a machine-learned learning device. Note that this learning device is a deep neural network or the like built into the image processing device, and is trained in advance by an existing machine learning method, for example, using a large number of document images to which retouching commands and ROI (Region of Interest) commands have been added as input data and classifications of the retouching commands and ROI commands (command IDs) as output data.

[0024] In other words, in order to extract the addition command, OCR (Optical Character Recognition) processing is not performed on the addition object detected by the addition object detection unit 12, but the addition object is directly classified into an addition command based on the image information (image features) of the addition object and registered in advance.

[0025] In this way, because revision commands are recognized through object detection, even if a user makes a minor error, such as a spelling mistake, when entering a revision command into a manuscript, the command is easily recognized as the user's intended revision command, without the need to refer to a dictionary or the like, and the user's intended revision command is recognized quickly and accurately.

[0026] FIG. 2 is a diagram showing an example of a manuscript and examples of a revision command and an ROI command as additional objects additionally written on the manuscript.

[0027] For example, a retouch command 111 and ROI commands 112a and 112b are added to a document 101 as shown in Fig. 2. For example, the retouch command 111 is handwritten using a writing implement such as a marker of a predetermined color. The ROI commands 112a and 112b are commands for specifying a region to be processed.

[0028] The retouch command 111 shown in Fig. 2 is the character string "OCR." The ROI commands 112a and 112b shown in Fig. 2 are a pair of brackets, and are used to specify a processing target region for the processing specified by the retouch command 111.

[0029] In this embodiment, the setting data 2 a includes a plurality of processing types that can be specified by a plurality of different retouching commands, such as image insertion, character information extraction, data structuring, and image generation.

[0030] For example, the "image insertion" process corresponding to the "ERASE" retouch command overwrites the target area with a background color (i.e., erases objects within the target area). For example, the "image insertion" process corresponding to the "MASK" retouch command overwrites the target area with a specific color (e.g., black) (masking process). For example, the "text information extraction" process corresponding to the "OCR" retouch command executes OCR processing on the image of the target area. For example, the "data structuring" process corresponding to the "TABLE" retouch command extracts a table from the image of the target area and saves the data in the table as CSV data or the like. For example, the "image generation" process corresponding to the "@AI" retouch command generates an image related to the character information (text data) obtained by OCR processing on the image of the target area using a diffusion model or the like, and then embeds the image in the target area.

[0031] The region-of-interest (ROI) specification command includes a pair of mutually different first and second commands (ROI command 112a and 112b). The region-of-interest (ROI) specification command is a bounding box of the pair of first and second ROI commands (ROI command 112a and 112b). Here, the first command (ROI command 112a) is an open bracket, and the second command (ROI command 112b) is a closed bracket.

[0032] In the detection of the additional objects described above, the bounding boxes (circumscribing rectangles) 121, 122a, and 122b of the additional objects are detected, and in the identification of the processing type described above, the additional objects having specific image features are recognized as retouching commands 111, and the additional objects having specific image features are recognized as ROI commands 112a and 112b (a pair of an image indicating the upper left corner of the processing target area and an image indicating the lower right corner of the processing target area).

[0033] 3 is a diagram illustrating the association of detected retouch commands and ROI commands. For example, as shown in FIG. 3, the bounding boxes 122 of the ROI commands 112a and 112b are set as the processing target region, and the retouch command 111 and the processing target region are associated with each other based on the distance D between the center of gravity of the bounding box 122 (processing target region) and the center of gravity of the bounding box 121 of the retouch command 111. In other words, the processing specified by the retouch command 111 is executed on this processing target region. For example, if the distance D is equal to or less than a predetermined value, the retouch command 111 and the processing target region are associated with each other.

[0034] 4 is a diagram showing an example in which multiple processing target regions are specified by processing target region specification commands. For example, as shown in FIG. 4, when multiple processing target regions are specified by multiple pairs of ROI commands 112a-1 to 112a-3 and 112b-1 to 112b-3, the processing target region with the shortest distance between the center position of the retouching command (the center of gravity of the bounding box of the retouching command) and the center position of the processing target region is determined to be the processing target region specified by the processing target region specification command corresponding to that retouching command.

[0035] FIG. 5 is a diagram (1 / 2) explaining the correspondence between the first command (ROI command 112a-i) and the second command (ROI command 112b-i) when multiple processing target regions are specified by the processing target region specification command.

[0036] In this embodiment, as shown in FIG. 5, for example, a second command (ROI command 112b-i) that is within a specific angle range (here, an angle range that is in the positive main scanning direction and the positive sub-scanning direction) from a first command (ROI command 112a-i) is considered to be the second command that pairs with the first command.

[0037] 5, if there is only one second command (ROI command 112b-i) within a specific angular range as viewed from the first command (ROI command 112a-i), that second command (ROI command 112b-i) is considered to be the second command (ROI command 112b-i) paired with that first command (ROI command 112a-i). For example, in the case shown in FIG. 5, ROI command 112b-3 is the only second command within a specific angular range as viewed from ROI command 112a-3, so ROI command 112b-3 is considered to be the second command paired with ROI command 112a-3.

[0038] 5, after ROI command 112b-3 and ROI command 112a-3 are paired, ROI command 112b-2 is the only undetermined (unpaired) second command within a specific angle range from ROI command 112a-2, so ROI command 112b-2 is set as the second command paired with ROI command 112a-2.After that, ROI command 112b-1 is the only undetermined (unpaired) second command within a specific angle range from ROI command 112a-1, so ROI command 112b-1 is set as the second command paired with ROI command 112a-1.

[0039] Fig. 6 is a diagram showing another example in which multiple processing target regions are specified by a processing target region specification command. Fig. 7 is a diagram (2 / 2) explaining the correspondence between the first command (ROI command 112a-i) and the second command (ROI command 112b-i) in the case in which multiple processing target regions are specified by a processing target region specification command.

[0040] For example, as shown in FIG. 6, when a plurality of first commands (ROI commands 112a-i) and a plurality of second commands (ROI commands 112b-i) are identified as processing target region designation commands, if there are a plurality of second commands (ROI commands 112b-i) within a specific angular range as viewed from the first commands (ROI commands 112a-i) (in other words, if there is no first command (ROI command 112a-i) with only one second command (ROI command 112b-i) within a specific angular range), the second command (ROI command 112b-i) that is the closest from the first command (ROI command 112a-i) among the plurality of second commands (ROI commands 112b-i) is determined to be the second command (ROI command 112b-i) that pairs with the first command (ROI command 112a-i).

[0041] For example, in the case shown in Fig. 6, ROI commands 112b-4 and 112b-5 are present within a specific angle range relative to ROI command 112a-4, as shown in Fig. 7. Therefore, of ROI commands 112b-4 and 112b-5, ROI command 112b-4, which is closest to ROI command 112a-4, is designated as the second command paired with ROI command 112a-4 (first command). Then, ROI command 112b-5 is designated as the second command paired with ROI command 112a-5 (first command).

[0042] In this way, the first command and the second command of the processing target area designation commands are associated with each other.

[0043] Furthermore, if there are multiple retouching commands at approximately the same distance from the center position of a certain processing target area, the retouching command whose center position is above (i.e., in the negative sub-scanning direction) the processing target area (center position) is considered to be the retouching command corresponding to that processing target area.

[0044] 8 is a diagram showing an example of a case where multiple retouch commands are located at approximately the same distance from the center of a certain processing target area. For example, in the case shown in FIG. 8, retouch commands 111-1 and 111-2 are located at approximately the same distance from the center of the processing target area specified by ROI commands 112a-6 and 112b-6. In this case, retouch command 111-1, which is located above the processing target area, is considered to be the retouch command corresponding to this processing target area.

[0045] Furthermore, in this embodiment, the additional object detection unit 12 (a) classifies the additional object into one of a plurality of predetermined commands, and (b) displays the detection results and classification results of the additional object on a predetermined display device 4.

[0046] 9 is a diagram showing an example of an editing screen. The additional object detection unit 12 displays an editing screen such as that shown in FIG. 9 on the display device 4, and displays the detection results and classification results of the additional objects on the editing screen. Here, the detection results of the additional objects are shown as bounding boxes 211, and the classification results (classes 212, i.e., the names of the retouching commands) are displayed for the bounding boxes 211. In the case shown in FIG. 9, three additional objects are detected, one of which is classified as a retouching command (OCR), one of which is classified as the first command (ROI[), and one of which is classified as the second command (ROI]).

[0047] The edit screen also displays the class, position, and processing number of a command selected by a user operation (here, a touch operation).

[0048] The edit screen further includes a class change key 221. The class change key 221 is a soft key for manually changing the class of a selected command. When the class change key 221 is pressed, the additional object detection unit 12 displays a dialog for changing the class of the selected command, and changes the class of the selected command based on a user operation on the dialog.

[0049] The edit screen also includes a delete key 222. The delete key 222 is a key for canceling the detection of a selected command if the command is erroneously detected. When the delete key 222 is pressed, the additional object detection unit 12 deletes the detection results and classification results of the selected command.

[0050] In this way, the additional object detection unit 12 uses a graphical user interface to modify the above-mentioned detection results and classification results in accordance with user operations on the input device 5 (such as a touch panel).

[0051] 1 , the processing execution unit 14 uses the image processing device or an external server to execute processing of the processing type corresponding to the identified retouch command (command ID) on the processing target area specified by the processing target area specification command corresponding to the retouch command. When executing the processing using an external server, the processing execution unit 14 transmits a processing request and data required for the processing to the external server via the communication device 3, and receives the processing result from the external server via the communication device 3.

[0052] In this embodiment, the process execution unit 14 selects whether to execute the above-mentioned process using the image processing device itself or an external server based on the identified process type. Specifically, the process execution unit 14 causes the external server to execute high-load processes, and executes low-load processes in the image processing device itself without using the external server.

[0053] The output processing unit 15 outputs the document image after the above processing (printing, data transmission, etc.), outputs the data obtained by the above processing, and displays the results of the above processing on the display device 4. If an error occurs in the detection of the added object or the identification of the processing type, the error processing unit 16 identifies the cause and type of the error and displays them on the display device 4. The command registration unit 17 registers retouching commands desired by the user.

[0054] Next, a description will be given of the operation of the image processing apparatus shown in Fig. 1. Fig. 10 is a flowchart illustrating the operation of the image processing apparatus shown in Fig. 1.

[0055] When the target image acquisition unit 11 acquires a document image (including retouching commands and ROI commands) (step S1), the additional object detection unit 12 detects additional objects written on the document (step S2) and identifies retouching commands and ROI commands in the detected additional objects. At this time, the above-mentioned editing screen may be displayed so that the user can edit the detection and classification results of the retouching commands and ROI commands.

[0056] The additional object detection unit 12 identifies the processing target area based on the ROI command, attempts to classify the additional object into an add command, and determines whether the add command is identified (i.e., whether the processing specified by the add command can be executed) (step S3).

[0057] When an add command is identified, the processing type identification unit 13 identifies the processing type and processing corresponding to the add command (step S4), and the processing execution unit 14 executes the processing of the processing type corresponding to the add command on the processing target area identified based on the detected added object as described above (step S5).

[0058] The processing execution unit 14 then displays the processing results on the display device 4 (step S6), and performs output processing (printing, data transmission, data storage, etc.) of the processed document image and generated data (step S7).

[0059] On the other hand, if the detection of the added object or the identification of the retouching command fails in step S3, the error processing unit 16 performs an error analysis and displays the cause of the error, the type of the error, etc. on the display device 4 (step S8).

[0060] As described above, according to the first embodiment, the target image acquisition unit 11 acquires a document image of a document. The additional object detection unit 12 (a) detects additional objects written on the document, (b) identifies a retouch command and a processing target area designation command included in the detected additional objects, and (c) identifies a processing target area designation command corresponding to the retouch command. The process type identification unit 13 identifies a process type corresponding to the identified retouch command from among a plurality of predetermined process types. The process execution unit 14 uses the image processing device or an external server to execute a process of the process type corresponding to the identified retouch command on the processing target area designated by the processing target area designation command corresponding to the retouch command.

[0061] This allows the user to intuitively execute desired processing on desired portions of a document, without including information about the processing when creating the document to be processed.

[0062] Embodiment 2.

[0063] 11 is a block diagram showing the configuration of an image processing device according to embodiment 2 of the present invention. In embodiment 2, a processing device 1 operates as the above-mentioned processing units, namely, a target image acquisition unit 11, an additional object detection unit 12, a process type identification unit 13, a process execution unit 14, an output processing unit 15, an error processing unit 16, a command registration unit 17, and a rule management unit 18.

[0064] The rule management unit 18 (a) registers the above-mentioned touch-up command and the corresponding processing target area as a rule, and (b) adds the touch-up command and processing target area registered as the rule to another document image.

[0065] In this case, the rule management unit 18 (a) registers the above-mentioned retouch command and the position of the processing target area, and (b) adds the retouch command and the processing target area to the registered position in another document image.

[0066] Fig. 12 is a diagram showing an example of a document to which revision commands have been entered. Fig. 13 is a diagram showing an example of a display list of registered rules. Fig. 14 is a diagram showing an example of a preview display of another document image to which revision commands 311-1 and 311-2 and ROI commands 312-1 and 312-2 have been added based on the rules.

[0067] For example, as shown in Fig. 12, an original image of an original 301 to which touch-up commands 311-1 and 311-2 and ROI commands 312-1 and 312-2 have been added is acquired, and the touch-up commands and ROI commands (regions to be processed) are detected and registered as rule "OCR#1" in the list shown in Fig. 13. Then, as shown in Fig. 14, rule "OCR#1" is applied to another original image 401 obtained from another original, and touch-up commands 311-1 and 311-2 and ROI commands 312-1 and 312-2 are added. At this time, the original other than the original 301 does not contain any touch-up commands or ROI commands, and the original image 401 (the other original image) read from that original does not originally contain any touch-up commands or ROI commands.

[0068] In this embodiment, the rule management unit 18 may (a) display a list of edit commands registered as rules on a specified display device 4, for example, as shown in Figure 13, (b) select an edit command from the list in accordance with user operation, and (c) add the selected edit command and the corresponding processing target area to another original image.

[0069] In this embodiment, the rule management unit 18 may move the retouch command and the processing target area to a position designated by a user operation in the other document image 401 described above.

[0070] Fig. 15 is a diagram showing an example of another original. Fig. 16 is a diagram showing an example of a preview display (initial state) of the original image of the original shown in Fig. 15. Fig. 17 is a diagram showing an example of a preview display (after editing) of the original image of the original shown in Fig. 15.

[0071] For example, an original image of a manuscript 501 to which no retouching commands or ROIs have been added, as shown in Figure 15, is obtained, and the original image 601 is displayed, for example, as shown in Figure 16.Furthermore, a default rule or a rule specified by the user (here, the rule "OCR#1") is applied, and retouching commands 311-1, 311-2 and ROI commands 312-1, 312-2 (areas to be processed) of that rule are added and displayed.

[0072] If the format (layout) of the original image (original) at the time of rule registration is different from the format (layout) of the original image 601 (original 501), the retouch commands 311-1, 311-2 and ROI commands 312-1, 312-2 (areas to be processed) may not be added in the appropriate positions, as shown in Figure 16, for example.

[0073] Therefore, for example, the user can drag a desired retouch command and ROI command (region to be processed) (retouch commands 311-1 and 311-2 and ROI commands 312-1 and 312-2 in FIG. 16) to a desired position (for example, the position shown in FIG. 17). In other words, the rule management unit 18 detects this dragging with the input device 5 and moves the retouch command and ROI command to the end position of the dragging.

[0074] In addition, when a document image including a retouch command and a processing target area is displayed on a predetermined display device 4, the rule management unit 18 registers the displayed retouch command and processing target area as a rule.

[0075] The rule management unit 18 may register the rule edited as described above as a new rule in accordance with a user operation. Fig. 18 is a diagram showing an example of a list displaying a new rule added. As shown in Figs. 16 and 17, the revision command and ROI command whose positions have been changed may be registered as a new rule "B", for example, as shown in Fig. 18.

[0076] 19 is a diagram showing an example of a rule registration screen. The rule management unit 18 displays, for example, a rule registration screen as shown in Fig. 19 on the display device 4 and identifies the name of the rule specified by the user. For example, the rule registration screen shown in Fig. 19 includes a rule name input key 701 and a rule name candidate list 702.

[0077] The rule name input key 701 is a soft key, and when the rule name input key 701 is pressed, a dialog in which the rule name can be input is displayed, and the text input from the keyboard in the dialog is specified as the name of the rule to be registered.

[0078] The rule name candidate list 702 also includes items such as the name of the revision command ("OCR" in FIG. 19), text obtained by OCR processing of the processing target area ("XXX-0010" in FIG. 19), and text obtained by OCR processing of the document image ("Invoice" in FIG. 19). Each item is displayed as a soft key, and the pressed item is identified as the name of the rule to be registered.

[0079] In this case, the rule management unit 18 may edit the displayed retouch command and processing target area in accordance with a user operation and register the edited retouch command and processing target area as a rule. Here, the command type of the retouch command and the position of the retouch command and processing target area as described above can be edited. The rule management unit 18 may also delete the retouch command and processing target area specified by a user operation.

[0080] Fig. 20 is a diagram showing an example of a revision command candidate list, Fig. 21 is a diagram showing an example of a preview display (after revision commands have been changed) of the document image of the document shown in Fig. 12 .

[0081] For example, as shown in Fig. 20, the rule management unit 18 displays a command name 711 of the edit command specified by a user operation among the displayed edit commands, and also displays an edit command candidate list 721. Here, the edit command candidate list 721 includes the edit commands "MASK" and "ERASE." Then, as shown in Fig. 21, for example, the rule management unit 18 changes the edit command in the document image to the edit command selected by a user operation from the edit command candidate list 721. In Fig. 21, the edit command has been changed from "OCR" to "MASK."

[0082] Next, a description will be given of the operation of the image processing device according to embodiment 2. Fig. 22 is a flowchart illustrating the operation of the image processing device according to embodiment 2.

[0083] When the target image acquisition unit 11 acquires a document image (step S21), the additional object detection unit 12 attempts to detect additional objects that have been additionally written on the document (step S22).

[0084] The rule management unit 18 determines whether or not there are additional objects (step S23), and if there are additional objects, determines whether or not to display a preview of the document image according to a predetermined setting (step S24). If the rule management unit 18 determines that a preview of the document image is to be displayed, the preview of the document image is displayed (step S25), as shown in, for example, FIGS.

[0085] Here, it is possible to change the retouch command as shown in FIG. 20, or to change the positions of the retouch command and ROI command (region to be processed) as shown in FIGS.

[0086] The rule management unit 18 then determines whether to register the retouch command and ROI command as a rule (step S26) in accordance with the user's operation on the input device 5. If it determines that the retouch command and ROI command should be registered as a rule, the rule management unit 18 registers the retouch command and ROI command as a rule (step S27). Note that this rule is included in the setting data 2a.

[0087] Then, the processing type identification unit 13 identifies the processing target area, attempts to classify the added object into an add command, and determines whether the add command is identified (i.e., whether the processing specified by the add command can be executed) (step S28).

[0088] When a retouch command is identified, the process type identification unit 13 identifies the process type and process corresponding to the retouch command, and the process execution unit 14 executes the process of the process type corresponding to the retouch command on the processing target area identified based on the detected added object as described above, and displays the processing result on the display device 4 (step S29).The process execution unit 14 then performs output processing (printing, data transmission, data storage, etc.) of the processed document image and the generated data (step S30).

[0089] On the other hand, if the detection of the added object or the identification of the retouching command fails in step S28, the error processing unit 16 performs an error analysis and displays the cause of the error, the type of the error, etc. on the display device 4 (step S31).

[0090] On the other hand, if it is determined in step S23 that there are no additional objects in the acquired document image, the rule management unit 18 selects a default rule from the registered rules and adds the retouch command and ROI command of that rule to the document image at the position specified by that rule (step S32). The default rule may be the most recently used rule or a previously specified rule.

[0091] The rule management unit 18 then determines whether or not to display a preview of the document image in accordance with a predetermined setting (step S33). If the rule management unit 18 determines that a preview of the document image is to be displayed, the rule management unit 18 displays a preview of the document image as shown in, for example, FIGS. 14, 16, and 20 (step S34).

[0092] Here, it is possible to change the retouch command as shown in FIG. 20, or to change the positions of the retouch command and ROI command (region to be processed) as shown in FIGS.

[0093] Then, the rule management unit 18 applies the currently displayed retouch command and ROI command in accordance with the user's operation on the input device 5 (step S35).

[0094] Thereafter, the process type identification unit 13 identifies the process target area, attempts to classify the added object into a retouch command, and determines whether a retouch command is identified (step S28). Then, the process from step S28 onward is executed in the same manner.

[0095] The other configurations and operations of the image processing device according to the second embodiment are the same as those of any of the other embodiments, and therefore a description thereof will be omitted.

[0096] As described above, according to the second embodiment, the target image acquisition unit 11 acquires a document image of a document. The added object detection unit 12 detects an added object written on the document. The process type identification unit 13 identifies, from among a plurality of predetermined process types, a process type corresponding to a retouch command included in the detected added object. The process execution unit 14 uses the image processing device or an external server to execute a process of the identified process type on a process target area identified based on the detected added object. The rule management unit 18 then (a) registers the retouch command and the process target area as a rule, and (b) adds the retouch command and the process target area registered as the rule to another document image.

[0097] This eliminates the need for the user to add similar commands to multiple pages, allowing the user to easily execute the same process on each of the multiple pages of document images.

[0098] Embodiment 3.

[0099] In the third embodiment, the added object detection unit 12 classifies detected added objects into retouching commands and processing target area designation commands by object detection using a machine-learned learner. Here, the retouching commands are not text information but image information, and are at least one of text, symbols, and figures. Similarly, the processing target area designation commands are also image information, and are at least one of text, symbols, and figures.

[0100] This learning device is a deep neural network built into the image processing device, and is trained in advance using an existing machine learning method, for example, with additional objects for each original image as input data and classifications (command IDs) of retouching commands and ROI commands as output data.

[0101] In other words, in order to extract the addition command, OCR (Optical Character Recognition) processing is not performed on the addition object detected by the addition object detection unit 12, and the addition object is directly classified as an addition command based on the image information (image features) of the addition object.

[0102] The region-of-interest (ROI) specification command may include a pair of mutually different first and second commands (ROI command 112a and 112b), as shown in Fig. 2. The region-of-interest (ROI) specification command is the bounding box of the pair of first and second ROI commands (ROI command 112a and 112b). Here, the first command (ROI command 112a) is an open bracket, and the second command (ROI command 112b) is a closed bracket.

[0103] In detecting the additional objects described above, bounding boxes (circumscribed rectangles) 121, 122a, and 122b of the additional objects are derived, as shown in FIG. 3, for example. In identifying the processing type described above, an additional object having a specific image feature is recognized as a retouch command 111, and an additional object having a specific image feature is recognized as ROI commands 112a and 112b (a pair of an image indicating the upper left corner of the processing target area and an image indicating the lower right corner of the processing target area).

[0104] FIG. 23 is a diagram illustrating the association of detected retouch commands and ROI commands. Then, as shown in FIGS. 3 and 23 , the bounding boxes 122 of the ROI commands 112 a and 112 b are set as the processing target region, and the retouch command 111 and the processing target region are associated with each other based on the distance D between the center of gravity of the bounding box 122 (processing target region) and the center of gravity of the bounding box 121 of the retouch command 111. That is, the processing specified by the retouch command 111 may be executed on this processing target region. For example, the retouch command 111 and the processing target region may be associated with each other if the distance D is equal to or less than a predetermined value. Furthermore, when there are multiple processing target regions and multiple retouch commands, the processing target region (processing target region designation command) with the shortest distance between the center position of the retouch command (the center position of the bounding box of the retouch command) and the center position of the processing target region is identified as the processing target region (processing target region designation command) corresponding to that retouch command.

[0105] The learning device is trained by machine learning using multiple training datasets in which additional objects in training manuscript images are used as input data and classifications of retouching commands and target area designation commands for the additional objects are paired with output data. For example, the training manuscript images in the multiple training datasets include training manuscript images in which the retouching commands and target area designation commands partially overlap. Furthermore, for example, the training manuscript images in the multiple training datasets include training manuscript images in which the (virtual) bounding boxes of the retouching commands and the (virtual) bounding boxes of the target area designation commands partially overlap.

[0106] By using such a training data set, even if the edit command 111 and the ROI command 112b are close to each other, as shown in Figure 23, the edit command 111 and the ROI command 112b are separated from each other and identified separately.

[0107] Furthermore, because revision commands are recognized through object detection, even if a user makes a minor error, such as a spelling mistake, when entering a revision command into a manuscript, the command is easily recognized as the user's intended revision command, without the need to refer to a dictionary, and the user's intended revision command is recognized quickly and accurately.

[0108] Furthermore, the process type identification unit 13 identifies a process type corresponding to the identified retouch command from among a predetermined plurality of process types. For example, a process ID indicating a process type is associated with the command ID of the retouch command and registered in advance, and the process type identification unit 13 identifies the process type of the process ID corresponding to the command ID of the detected retouch command as the process type corresponding to the retouch command.

[0109] In this embodiment, the setting data 2 a includes a plurality of processing types that can be specified by a plurality of different retouching commands, such as image insertion, character information extraction, data structuring, and image generation, as described above.

[0110] For example, the "image insertion" process type corresponding to the retouching command "ERASE" executes a process of overwriting the processing target area with the background color (i.e., erasing the objects in the processing target area). For example, the "image insertion" process type corresponding to the retouching command "MASK" executes a process of overwriting the processing target area with a specific color (such as black) (masking process). For example, the "image insertion" process type corresponding to the retouching command "STAMP" executes a process of fitting a specific image (a seal image) that has been specified in advance into the processing target area.

[0111] 24 is a diagram showing an example in which multiple processing target regions are specified along with multiple retouch commands. For example, as shown in FIG. 24, multiple processing target regions may be specified by multiple pairs of ROI commands 812-1 to 812-3 for multiple retouch commands 811-1 to 811-3. The closest ROI command 812-1 among the ROI commands 812-1 to 812-3 is associated with the retouch command 811-1, the closest ROI command 812-2 among the ROI commands 812-1 to 812-3 is associated with the retouch command 811-2, and the closest ROI command 812-4 among the ROI commands 812-1 to 812-3 is associated with the retouch command 811-3. In the case of FIG. 24, the retouching command "STAMP" is executed on the processing target area specified by ROI command 812-1, and the retouching command "MASK" is executed on the processing target areas specified by ROI commands 812-2 and 812-3.

[0112] 25 is a diagram showing an example in which multiple processing target regions are specified along with one retouch command. For example, as shown in FIG. 25, multiple processing target regions may be specified for one retouch command 831 using multiple pairs of ROI commands 832-1 to 832-3. In other words, if there is no other retouch command located closer to the multiple processing target regions specified by the multiple pairs of ROI commands 832-1 to 832-3 than the retouch command 831, the multiple pairs of ROI commands 832-1 to 832-3 are associated with the single retouch command 831. In the case of FIG. 25, the processing of the retouch command "MASK" is executed for the three processing target regions specified by the ROI commands 832-1 to 832-3.

[0113] For example, the process type "text information extraction" corresponding to the retouch command "FILENAME" executes a process of saving image data of a document using text information (text data) obtained by OCR processing of the image in the processing target area as a file name. The process type TAG is a process of generating a character string (text information) included in the processing target area by OCR processing when the document image is converted into PDF format data, and setting the text information as a search tag.

[0114] FIG. 26 is a diagram showing an example in which multiple retouching commands are specified for one processing target area (a pair of processing target area designation commands). For example, as shown in FIG. 26, one processing target area may be specified by a pair of ROI commands 842 for multiple retouching commands 841-1 and 841-2. In other words, if there is no other processing target area closer to the multiple retouching commands 841-1 and 841-2 than the processing target area designated by the ROI command 842, the pair of ROI commands 842 (one processing target area) is associated with the multiple retouching commands 841-1 and 841-2. In the case of FIG. 26, processing of the retouching command "FILENAME" and processing of the retouching command "TAG" are executed for the one processing target area designated by the ROI command 842. Specifically, the text "TRAVELER INFORMATION" is obtained by OCR processing of the processing target area, a file of the document image is generated with the file name "TRAVELER INFORMATION.pdf", and the tag "TRAVELER INFORMATION" is included in the file as attribute information.

[0115] In addition, the processing execution unit 14 uses the image processing device or an external server to perform processing of the processing type corresponding to the identified retouch command (command ID) on the processing target area specified by the processing target area designation command corresponding to the retouch command.

[0116] In this case, the processing execution unit 14 (a) erases the retouching command from the document image, and (B) identifies the processing target area in the document image after the retouching command has been erased based on the processing target area designation command. As a result, even if the retouching command is close to the processing target area designation command, the processing target area is identified without being affected by the retouching command, and the processing target area is accurately identified.

[0117] Furthermore, when a specific retouching command is used, an argument command can be added. Here, the argument command is a number entered on the document together with the ROI command and specifies the processing order of multiple pairs of ROI commands (multiple processing target regions). The argument command is converted from image information into a numerical value by OCR processing. The added object detection unit 12 (a) identifies the argument command along with the retouching command and processing target region designation command included in the detected added object using the above-described object detection, and (b) identifies the processing target region designation command corresponding to the retouching command and the argument command corresponding to the processing target region designation command. Then, the processing execution unit 14 executes the processing corresponding to the retouching command on the processing target region designated by the processing target region designation command in the order specified by the identified argument commands.

[0118] FIG. 27 is a diagram showing an example of argument commands. In the case of FIG. 27, there are multiple pairs of processing target area designation commands 852-1 to 852-3 (multiple processing target areas) for a retouch command 851 (FILENAME), and argument commands 853-1 to 853-3 are entered for each pair of processing target area designation command 852-i (each processing target area). In this case, the argument commands 853-1 to 853-3 specify the processing order as follows (in ascending numerical order): the processing target area of ​​processing target area designation command 852-1, the processing target area of ​​processing target area designation command 852-2, and the processing target area of ​​processing target area designation command 852-3. Specifically, the text "Taro" is obtained by OCR processing of the processing target area of ​​processing target area designation command 852-1, the text "Suzuki" is obtained by OCR processing of the processing target area of ​​processing target area designation command 852-2, and the text "28" is obtained by OCR processing of the processing target area of ​​processing target area designation command 852-3, and the file name "Taro_Suzuki_28.pdf" is generated by concatenating "Taro," "Suzuki," and "28." Here, the delimiter "_" is used, but other delimiters may be used, or no delimiter may be used.

[0119] Next, a description will be given of the operation of the image processing device according to embodiment 3. Fig. 28 is a flowchart illustrating the operation of the image processing device according to embodiment 3.

[0120] When the target image acquisition unit 11 acquires a document image (containing retouching commands, ROI commands, etc.) (step S41), the additional object detection unit 12 detects additional objects added to the document (step S42), and identifies retouching commands and ROI commands from the detected additional objects by object detection as described above. When a retouching command is identified, the process type identification unit 13 identifies a process type and process corresponding to the retouching command (step S43).

[0121] Then, the processing execution unit 14 erases the retouch command, identifies the processing target region (position and size) specified by the ROI command (step S44), extracts an image of the identified processing target region from the document image (step S45), and executes the identified processing on the document image (step S46). After completing the processing, the processing execution unit 14 displays the processing results on the display device 4 and performs output processing (printing, data transmission, data storage, etc.) of the processed document image and the generated data.

[0122] The other configurations and operations of the image processing device according to the third embodiment are the same as those of any of the other embodiments, and therefore a description thereof will be omitted.

[0123] As described above, according to the third embodiment, the target image acquisition unit 11 acquires a document image of a document. The additional object detection unit 12 (a) detects an additional object written on the document, (b) identifies a retouch command and a processing target area designation command included in the detected additional object, and (c) identifies a processing target area designation command corresponding to the retouch command. The process type identification unit 13 identifies a process type corresponding to the identified retouch command from among a plurality of predetermined process types. The process execution unit 14 uses the image processing device or an external server to perform a process of the process type corresponding to the identified retouch command on the processing target area designated by the processing target area designation command corresponding to the retouch command. The additional object detection unit 12 then classifies the detected additional object into a retouch command and a processing target area designation command by object detection using a machine-learned learner.

[0124] This allows the processing target area designation command and the retouching command to designate the processing target area to be detected appropriately even if they are close to each other in the original image.

[0125] Embodiment 4.

[0126] In the fourth embodiment, the plurality of processing types described above includes a structured data generation process. This structured data generation process is a process for generating structured data from an image of the processing target area described above. In this embodiment, the revision commands corresponding to the structured data generation process include the revision command "TABLE" and the revision command "CSV."

[0127] FIG. 29 is a diagram showing an example of a manuscript. FIG. 30 is a diagram showing an example of structured data generation processing corresponding to the revision command "TABLE." For example, in the structured data generation processing corresponding to the revision command "TABLE," a table is extracted from an image of the processing target area, and the image data of the table is converted into tabular data such as CSV data using OCR processing or the like as appropriate. Note that the conversion from the image data of the table to tabular data is performed using an existing method. For example, the revision command "TABLE" and an ROI command are entered into the manuscript shown in FIG. 29, and then the image of the manuscript is read, resulting in the manuscript image shown in FIG. 30. Then, through the structured data generation processing, tabular data as structured data shown in FIG. 30 is obtained from the manuscript image.

[0128] FIG. 31 is a diagram illustrating an example of a data addition process. The above-described multiple process types include a data addition process. This data addition process generates data (text, numerical values, etc.) from an image of a processing target area using OCR processing and adds the generated data to structured data (such as the table data described above) generated by the structured data generation process. The retouch command corresponding to the data addition process is different from the retouch command corresponding to the structured data generation process. In this embodiment, the retouch command corresponding to the data addition process is an "ID" retouch command. For example, the retouch command "TABLE" and an ROI command are entered into the manuscript shown in FIG. 29 , and then the retouch command "ID" and an ROI command are entered. The manuscript image is then scanned, resulting in the manuscript image shown in FIG. 31 . Then, the structured data generation process obtains tabular data as structured data shown in FIG. 31 from the manuscript image. In this tabular data, data corresponding to the edit command "TABLE" and the ROI command (here, the text "January") is added to each row of the tabular data (same as the tabular data shown in Figure 30) corresponding to the edit command "TABLE" and the ROI command.

[0129] Also, for example, in the structured data generation process corresponding to the add command "CSV," OCR processing is performed on images of multiple processing target areas corresponding to multiple add commands "CSV," and a process is executed to generate CSV data from the data (text, numbers, etc.) obtained by the OCR processing.

[0130] Furthermore, argument commands can be used along with the retouching commands corresponding to the structured data generation process. For example, an argument command is a single character (e.g., a number, a Greek letter, etc.). The process corresponding to the argument command is set according to the retouching command used together.

[0131] 32 is a diagram showing an example of an argument command for the edit command "TABLE." For example, as shown in FIG. 32, by using (adding) the argument command "1" for the edit command "TABLE," the rows and columns of the tabular data are swapped.

[0132] 33 is a diagram showing an example of argument commands for the "CSV" edit command. Furthermore, as shown in FIG. 33, for example, the order can be specified by using (adding) argument commands "1," "2," "3," ... for the "CSV" edit command. That is, as shown in FIG. 33, for example, CSV data is generated that includes the data obtained in response to the "CSV" edit command with argument command "1" and the data obtained in response to the "CSV" edit command with argument command "2" in this order.

[0133] Furthermore, the processing execution unit 14 may (a) store the retouching commands and processing target area designation commands identified for the original image, and (b) apply the stored retouching commands and processing target area designation commands to the original image subsequent to the original image. In this case, the structured data generation process generates one piece of structured data from the image of the processing target area in the original image and the image of the processing target area in the subsequent original image.

[0134] FIG. 34 shows an example of a process for generating one piece of structured data from multiple document images. In this case, a retouching command and a processing target area designation command are added to the first document, and no retouching command or processing target area designation command is added to the subsequent documents. Then, as shown in FIG. 34 , for example, a retouching command and a processing target area designation command are identified from a first document image 901 obtained from the first document, and the retouching command and the processing target area designation command of the first document image 901 are applied to subsequent document images 902 and 903 obtained from the subsequent documents. Then, the structured data obtained from the first document image 901 is sequentially linked to the structured data obtained from the subsequent document images 902 and 903.

[0135] Specifically, the process type corresponding to the retouch command "TABLE" in the first document image 901 is identified, and the position and size of the processing target area corresponding to the retouch command "TABLE" are identified. Similarly, the process type corresponding to the retouch command "ID" in the first document image 901 is identified, and the position and size of the processing target area corresponding to the retouch command "ID" are identified.

[0136] Next, in the subsequent original images 902 and 903, processing corresponding to the retouch command "TABLE" is executed on the processing target area corresponding to the retouch command "TABLE" in the first original image 901, and processing corresponding to the retouch command "ID" is executed on the processing target area corresponding to the retouch command "ID" in the first original image 901.

[0137] Then, the structured data obtained from the first original image 901 is linked to the structured data obtained from the subsequent original image 902, and further linked to the structured data obtained from the subsequent original image 903.

[0138] In the subsequent original images 902 and 903, the corresponding processing is performed on the processing target area corresponding to the retouch command "TABLE." Therefore, if the table in original image 903, for example, is larger than the processing target area corresponding to the retouch command "TABLE" in the first original image 901, the processing corresponding to the retouch command "TABLE" is performed only on the portion of the table that is included in the processing target area. In other words, in the case of Figure 34, five lines of data are obtained for the table in original image 903, but the remaining two lines of data are not obtained.

[0139] Next, a description will be given of the operation of the image processing device according to embodiment 4. Fig. 35 is a flowchart illustrating the operation of the image processing device according to embodiment 4.

[0140] When the target image acquisition unit 11 acquires a document image (containing retouching commands, ROI commands, etc.) (step S61), the additional object detection unit 12 detects additional objects written on the document and identifies retouching commands and ROI commands from the detected additional objects (step S62). If a retouching command is identified, the process type identification unit 13 identifies the process type corresponding to the retouching command and the process to be executed (step S63).

[0141] The processing execution unit 14 then identifies the processing target region (position and size) specified by the ROI command for each identified retouch command (step S64), extracts an image of the identified processing target region from the document image (step S65), and executes the identified processing for each identified retouch command on the document image (step S66). After completing the processing, the processing execution unit 14 displays the processing results on the display device 4 and performs output processing (printing, data transmission, data storage, etc.) of the processed document image and generated data (such as the above-mentioned structured data).

[0142] Here, the details of the process execution in step S66 will be described. Fig. 36 is a flowchart illustrating the details of the process execution in Fig. 35.

[0143] The process execution unit 14 determines whether or not the revision command "TABLE" is present among the revision commands identified in the document image (step S71).

[0144] If there is a revision command "TABLE", the process execution unit 14 generates structured data from the image of the processing target area corresponding to the revision command "TABLE" (step S72).

[0145] Next, the process execution unit 14 determines whether or not the retouch command "ID" is present in the retouch commands identified in the document image (step S73).

[0146] If there is an add command "ID", the processing execution unit 14 generates data from the image of the processing target area corresponding to the add command "TABLE" and adds the data to the structured data generated in step S12 (step S74).

[0147] Thereafter, if there is another retouching command identified in the document image, the process executing unit 14 executes a process corresponding to that retouching command (step S75).

[0148] In this way, the table in the original image is converted into structured data in table format based on the touch-up command.

[0149] As described above, according to the fourth embodiment, the target image acquisition unit 11 acquires a document image of a document. The additional object detection unit 12 (a) detects an additional object written on the document, (b) identifies a retouch command and a processing target area designation command included in the detected additional object, and (c) identifies a processing target area designation command corresponding to the retouch command. The process type identification unit 13 identifies a process type corresponding to the identified retouch command from among a plurality of predetermined process types. The process execution unit 14 uses the image processing device or an external server to execute a process of the process type corresponding to the identified retouch command on the processing target area designated by the processing target area designation command corresponding to the retouch command. The plurality of process types includes a structured data generation process that generates structured data from an image of the processing target area.

[0150] This allows the user to specify processing by entering revision commands, allowing structured data to be generated from desired portions of a manuscript through intuitive operations.

[0151] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0152] For example, in the second embodiment, a rule may be applied to an original image that already contains touch-up commands and processing target areas, so that further touch-up commands and processing target areas can be added.

[0153] The present invention is applicable to, for example, an image processing apparatus that executes processing desired by a user on an original image.

Claims

1. An image processing device comprising: a target image acquisition unit that acquires a document image of a document; an additional object detection unit that (a) detects additional objects written on the document, (b) identifies a retouch command and a processing target area designation command contained in the detected additional objects, and (c) identifies the processing target area designation command corresponding to the retouch command; a processing type identification unit that identifies a processing type that corresponds to the identified retouch command from among a plurality of processing types; and a processing execution unit that uses the image processing device or an external server to execute processing of the processing type that corresponds to the identified retouch command on the processing target area designated by the processing target area designation command that corresponds to the retouch command.

2. An image processing device as described in claim 1, characterized in that when multiple processing target areas are specified by the processing target area designation command, the processing target area with the shortest distance between the center position of the retouching command and the center position of the processing target area is designated as the processing target area designated by the processing target area designation command corresponding to the retouching command.

3. The image processing device according to claim 1, wherein the processing target area designation command includes a pair of a first command and a second command that are different from each other, and the processing target area is a bounding box of the pair of the first command and the second command.

4. An image processing apparatus according to claim 1, wherein the second command within a specific angle range as viewed from the first command is regarded as the second command that forms a pair with the first command.

5. An image processing device as described in claim 4, characterized in that when a plurality of the first commands and a plurality of the second commands are identified as the processing target area designation command, if there is only one second command within the specific angle range from the first command, the second command is treated as the second command that pairs with the first command.

6. An image processing device as described in claim 4, characterized in that when a plurality of the first commands and a plurality of the second commands are identified as the processing target area designation command, and there are a plurality of the second commands within the specific angle range from the first command, the one of the plurality of second commands that is the shortest distance from the first command is determined to be the second command that pairs with the first command.

7. The image processing apparatus according to claim 1, wherein the touch-up command is not character information but is image information of at least one of characters, symbols, and figures.

8. The image processing device according to claim 1, characterized in that the additional object detection unit (a) classifies the additional object into one of a plurality of predetermined commands, (b) displays the detection results and classification results of the additional object on a predetermined display device, and (c) modifies the detection results and the classification results in accordance with user operations on a predetermined input device.

9. An image processing device as described in claim 1, further comprising a rule management unit that (a) registers the retouching command and the processing target area as rules, and (b) adds the retouching command and the processing target area registered as rules to other original images.

10. An image processing device as described in claim 9, characterized in that the rule management unit (a) registers the positions of the retouching command and the processing target area, and (b) adds the retouching command and the processing target area to the registered positions in the other original image.

11. The image processing device described in claim 9, characterized in that the rule management unit (a) displays the other original image to which the retouching command and the processing target area have been added on a specified display device, and (b) moves the retouching command and the processing target area in the other original image to a position specified by user operation.

12. The image processing device described in claim 9, characterized in that the rule management unit (a) displays a list of the revision commands registered as rules on a specified display device, (b) selects a revision command from the list in accordance with a user operation, and (c) adds the selected revision command and the corresponding processing target area to the other original image.

13. The image processing device according to claim 9, characterized in that the rule management unit (a) displays the original image including the retouching command and the processing target area on a specified display device, and (b) registers the displayed retouching command and the processing target area as the rule.

14. The image processing device described in claim 13, characterized in that the rule management unit (a) displays the original image including the touch-up command and the processing target area on a specified display device, (b) edits the displayed touch-up command and the processing target area in accordance with user operation, and (c) registers the edited touch-up command and the processing target area as the rule.

15. The image processing device according to claim 1, characterized in that the added object detection unit classifies the detected added object into the retouching command and the processing target area designation command by object detection using a machine learning learning device.

16. The image processing device described in claim 15, characterized in that the learning device is machine-trained using a plurality of training data sets in which an additional object in a training manuscript image is used as input data and a classification of an addition command and a processing target area designation command in the additional object is paired with output data, and the training manuscript images in the plurality of training data sets include training manuscript images in which the addition command and the processing target area designation command partially overlap.

17. The image processing device of claim 15, wherein the learning device is machine-trained using a plurality of training data sets in which an additional object in a training manuscript image is used as input data and a classification of an add-on command and a processing target area designation command in the additional object is paired with output data, and the training manuscript images in the plurality of training data sets include training manuscript images in which a bounding box of the add-on command and a bounding box of the processing target area designation command partially overlap.

18. An image processing device as described in claim 15, characterized in that the processing execution unit (a) erases the retouching command from the original image, and (B) identifies the processing target area in the original image after the retouching command has been erased based on the processing target area designation command.

19. The image processing device of claim 15, wherein the additional object detection unit (a) identifies the retouching command, the processing target area designation command, and an argument command contained in the detected additional object, and (b) identifies the processing target area designation command corresponding to the retouching command and identifies the argument command corresponding to the processing target area designation command, and the processing execution unit executes processing of the processing type on the processing target area designated by the processing target area designation command in the order specified by the argument commands.

20. An image processing device as described in claim 15, characterized in that when there are multiple processing target areas and multiple retouching commands, the processing target area with the shortest distance between the center position of the retouching command and the center position of the processing target area is designated as the processing target area specified by the processing target area designation command corresponding to the retouching command.

21. An image processing device as described in claim 15, characterized in that the processing target area designation command includes a pair of a first command and a second command that are different from each other, and the processing target area is a bounding box of the pair of the first command and the second command.

22. The image processing apparatus according to claim 1, wherein the plurality of processing types includes a structured data generation process for generating structured data from an image of the processing target area.

23. The image processing device of claim 22, wherein the multiple processing types include a data addition process that generates data from an image of the processing target area and adds the generated data to the structured data generated by the structured data generation process, and the retouching command corresponding to the data addition process is different from the retouching command corresponding to the structured data generation process.

24. The image processing device of claim 22, characterized in that the processing execution unit (a) stores the retouching command and the processing target area designation command identified for the original image, and (b) applies the stored retouching command and the processing target area designation command to an original image subsequent to the original image, and the structured data generation process generates one piece of structured data from an image of the processing target area in the original image and an image of the processing target area in the subsequent original image.

25. An image processing device as described in claim 22, characterized in that the retouching command corresponding to the structured data generation process is accompanied by an argument command, and the processing type corresponding to the argument command is a process of exchanging rows and columns in the structured data.

26. The image processing device according to claim 22, wherein the retouching command corresponding to the structured data generation process is accompanied by an argument command, and the processing type corresponding to the argument command is a process for specifying a data order in the structured data.

27. An image processing apparatus according to claim 22, wherein said structured data is data in a table format.

Images