Image processing system

Abstract

To obtain an image processing system that appropriately detects instruction objects specifying replacement processing targets within a target image.SOLUTION: An instruction object detection unit 22 detects, via object detection using a machine-learned learning device, a first instruction object and a second instruction object among handwritten objects additionally written on a manuscript within a target image 101. An instruction identifying unit 23 identifies a processing target object specified by the detected first instruction object and identifies data specified by the detected second instruction object. A replacement processing unit 24 replaces the processing target object in the target image with its data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing system. [Background technology]

[0002] An image processing device identifies the processing items specified by a specified color in the processing list for a document image obtained by scanning a document containing the processing items "delete," "acquire," and "restrict," and then executes the identified processing items on the parts of the document image that are surrounded or filled in with the specified color (see, for example, Patent Document 1).

[0003] Furthermore, an image processing device (a) separates handwritten portions from an original image of a document, (b) determines whether the handwritten portions are an enclosing line surrounding a target area or a classification symbol for classifying the target area, and (c) classifies the image of the target area by the classification symbol (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-27243 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-151722 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, it is conceivable to identify a target area designated by a handwritten pointing object such as a surrounding line and replace the target area with other content. In this case, if the pointing object is handwritten, there is a possibility that the pointing object will not be properly detected in the target image due to fluctuations in the shape of the pointing object.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an image processing system that can appropriately detect a pointing object that designates a target for replacement processing in a target image. [Means for solving the problem]

[0007] The image processing system according to the present invention includes a target image acquisition unit that acquires a document image of a document as a target image, a pointing object detection unit that detects, by object detection using a machine-learned learner, a first pointing object that designates a target object and a second pointing object that designates data to be used in a replacement process for the target object among handwritten objects additionally written on the document in the target image, a pointing object identification unit that identifies the target object designated by the detected first pointing object and the data designated by the detected second pointing object, a replacement processing unit that performs a replacement process for the identified target object in the target image with the identified data, and a user editing processing unit that performs editing on the target image after the replacement process based on a user operation. The user editing processing unit (a) changes the target object or the data in accordance with the user operation, and (b) edits the target image corresponding to the change. [Effects of the Invention]

[0008] According to the present invention, an image processing system is provided that can appropriately detect a pointing object that designates a replacement processing target in a target image.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a target image including a first pointing object and a second pointing object. [Figure 3] FIG. 3 is a diagram illustrating an object identified in the target image shown in FIG. [Figure 4] FIG. 4 is a diagram showing an example of the target image after the replacement process. [Figure 5] FIG. 5 is a flowchart illustrating the operation of the image processing system shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention, which includes a processing server 1 and an image forming device 2 (such as a multifunction peripheral) that can communicate with each other via a network.

[0013] The processing server 1 includes a processing unit 11 , a storage unit 12 , and a communication unit 13 .

[0014] The arithmetic processing device 11 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 12 into the RAM and executing it on the CPU. The arithmetic processing device 11 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0015] The storage device 12 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processing described below. The storage device 12 also stores system setting data. The system setting data includes parameters of the learning device described below.

[0016] The communication device 13 is a device that performs data communication with an external device such as the image forming device 2, and is, for example, a network interface or a peripheral device interface.

[0017] On the other hand, the image forming device 2 is equipped with an image reading device, a communication device, an operation panel, etc., and generates an original image from an original document as a target image using the image reading device, and transmits the target image to the processing server 1 using the communication device and receives the processed target image.

[0018] In the processing server 1, the arithmetic processing device 11 operates as the above-mentioned processing units, that is, a target image acquisition unit 21, a pointing object detection unit 22, a pointing identification unit 23, a replacement processing unit 24, a user edit processing unit 25, and an output processing unit 26.

[0019] The target image acquisition unit 21 acquires (image data of) a document image of a document as a target image from the storage device 12, communication device 13, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a slip, a flyer, etc.

[0020] The instruction object detection unit 22 detects a first instruction object that specifies the object to be processed and a second instruction object that specifies data to be used for replacement processing of the object to be processed among handwritten objects that are additionally written on the original in the target image by object detection using a machine-learned learner (YOLO (You Only Look Once), SSD (Single Shot MultiBox Object Detector), Faster R-CNN (Regions with Convolutional Neural Networks), etc.).

[0021] Note that handwritten objects are separate, individual foreground images that do not include the background. Furthermore, by performing machine learning using training data including manuscript images in which various handwritten pointing objects are superimposed on printed objects, pointing objects can be properly detected even when they are superimposed on printed objects in the target image. Furthermore, if the size of the manuscript image differs from the size of the input data to this learning device (object detection model), the manuscript image is converted to match the size of the input data to the learning device (object detection model). For example, if the manuscript image size is 2480 pixels x 3507 pixels, which is A4 size at 300 dpi, and the input data size is 640 pixels x 640 pixels, the resolution of the manuscript image is converted to 640 pixels x 640 pixels. Furthermore, the output data from this learning device includes, for each detected object, the class classification (whether it is a first pointing object, a second pointing object, or another object), a bounding box (position and size), and a confidence level of the class classification (a value between 0 and 1).

[0022] FIG. 2 is a diagram showing an example of a target image including a first pointing object and a second pointing object.

[0023] The first and second pointing objects have different shapes. In this embodiment, the first pointing object 111 is a strikeout line (two straight lines, for example), and the second pointing object 112 is an encircling line (any closed curve, such as a circle), as shown in FIG. 2.

[0024] 2, for example, a first pointing object 111 is written so as to be superimposed on an object 113 to be replaced in the target image 101. Also, as shown in FIG. 2, for example, data 114 to be used for replacement ("20%" in FIG. 2) is written in the area indicated by the second pointing object 112 in the target image 101 (here, the area inside the encircling line).

[0025] The learning device performs machine learning using training data of first and second pointing objects of multiple colors, multiple line widths, and multiple shapes, respectively, for the first and second pointing objects. This allows the first and second pointing objects to be properly detected regardless of the writing implement used to write them and the variations in shape due to handwriting.

[0026] Fig. 3 is a diagram illustrating objects identified in the target image shown in Fig. 2. Fig. 4 is a diagram illustrating an example of the target image after the replacement process.

[0027] 3, the instruction specifying unit 23 specifies a processing target object 113 specified by a detected first instruction object 111, and specifies replacement data 114 specified by a detected second instruction object 112. The replacement processing unit 24 performs replacement processing on the specified processing target object 113 in the target image 101 with the specified data 114, for example, as shown in FIG.

[0028] Specifically, as shown in FIG. 3, the above-described object detection indicates a bounding box 111a of the first instruction object 111 (i.e., the position and size of the object 111), a bounding box 112a of the second instruction object 112 (i.e., the position and size of the object 112), a bounding box 113a of the processing target object 113 (i.e., the position and size of the object 113), and a bounding box 114a of the data 114 (i.e., the position and size of the object as data 114), and based on the relative positional relationships among these, the instruction identification unit 23 identifies the processing target object 113 specified by the first instruction object 111, identifies the data 114 specified by the second instruction object 112, and also associates the first instruction object 111 with the second instruction object 112 in the target image 101.

[0029] In this embodiment, the instruction identification unit 23 performs character recognition processing on the image area designated by the second instruction object to identify the text data written in that image area, and the replacement processing unit 24 erases the target object in the target image and adds an object based on the identified text data to the position of the target object (i.e., places the object as the foreground image). At this time, the added object (specifically, the text as the object) is generated so as to have the same font type, color, and size as the target object.

[0030] Specifically, as shown in FIG. 4, for example, the replacement processing unit 24 converts the processing target object 113 and data 114 from a raster image to text data (here, "15%" and "20%") through character recognition processing, erases the first instruction object 111, the second instruction object 112, the processing target object 113, and the data 114 (specifically, erases all foreground images within the bounding boxes 111a, 112a, 113a, and 114a), converts the data 114 from text data to a raster image, and adds it to the position of the bounding box 113a.

[0031] Furthermore, user edit processing unit 25 performs editing on the target image after the replacement process based on a user operation. Specifically, user edit processing unit 25 (a) changes the object or data to be processed in accordance with the user operation, and (b) edits the target image in accordance with the change.

[0032] Furthermore, the output processing unit 26 outputs the target image after the above-described processing. For example, the output processing unit 26 transmits image data of the target image to the image forming device 2, causes the image forming device 2 to print the target image, or stores the target image in a predetermined storage device (such as the image forming device 2 or another server).

[0033] Next, the operation of the image processing system will be described with reference to Fig. 5, which is a flowchart illustrating the operation of the image processing system shown in Fig. 1.

[0034] The target image acquisition unit 21 of the processing server 1 acquires a target image (step S1). Here, the above-mentioned first and second instruction objects and data 114 are additionally written by hand on a document printed by the image forming device 2, and then the image of the document is read to obtain a document image, and the document image (image data) is sent from the image forming device 2 to the processing server 1 as a target image.

[0035] When the target image is acquired in the processing server 1, the pointing object detection unit 22 detects the first pointing object 111 and the second pointing object 112, as well as the processing target object 113 and the data 114, in the target image 101 by object detection (step S2).

[0036] Based on the result of the object detection, the instruction identification unit 23 identifies the processing target object 113 designated by the detected first instruction object 111 and the data 114 designated by the detected second instruction object 112 (step S3). That is, at this stage, the correspondence between the first instruction object 111 and the processing target object 113, the correspondence between the second instruction object 112 and the data 114, and the correspondence between the first instruction object 111 and the second instruction object 112 are identified.

[0037] Then, based on the correspondence identified by the instruction identification unit 23, the replacement processing unit 24 performs replacement processing on the identified processing target object 113 in the target image 101 with the identified data 114 described above (step S4).

[0038] When the replacement process is completed, user edit processing unit 25 transmits the target image after the replacement process to image forming apparatus 2 for display. Image forming apparatus 2 displays the target image after the replacement process on a display device of an operation panel or the like (step S5).

[0039] The user visually checks the displayed target image after the replacement process, determines whether the replacement was performed appropriately, and performs a user operation indicating the result of the determination on the image forming device 2 (the input device of the operation panel). If the user determines that the replacement was not performed appropriately, the user performs a user operation for editing on the image forming device 2 (the input device of the operation panel). The image forming device 2 notifies the user editing processing unit 25 of the processing server 1 of the user operation.

[0040] Based on the notification, user edit processing unit 25 determines whether the replacement has been performed appropriately (step S6). If it is determined that the replacement has been performed appropriately, output processing unit 26 outputs the target image after the replacement process.

[0041] On the other hand, if it is determined that the replacement was not performed appropriately, user edit processing unit 25 (a) changes target object 113 or data 114 in accordance with the notified user operation, and (b) edits the target image in accordance with the change (step S7). Thereafter, output processing unit 26 outputs the edited target image.

[0042] For example, an editing key (soft key) is displayed adjacent to the data 114 added by the replacement process in the target image after the replacement process, and when the editing key is pressed, a dialog is displayed, and a user operation is performed in the dialog to input the editing content, and the object 113 to be processed or the data 114 is changed in accordance with the user operation.

[0043] For example, as shown in FIG. 4, if "15%" is replaced with "20%," and the user wants to change it to another value (such as "35%"), or if "26%" is mistakenly recognized as "20%" during character recognition processing, the user inputs an appropriate value (the above-mentioned "35%" or "26%), and the user edit processing unit 25 changes the portion of data 114 in the target image to an image of that value.

[0044] Also, for example, as shown in Figures 3 and 4, if the object to be processed 113 should be recognized as "15%" but is mistakenly recognized as "15", and the replacement process results in, for example, "15%" becoming "20%%," then the range of the object to be processed 113 (i.e., the range of the object specified by the first instruction object 111) being changed to "15%" (expanded in this case) by user operation, and the user editing processing unit 25 causes the replacement processing unit 24 to perform the replacement process again based on the changed object to be processed 113, thereby appropriately replacing "15%" with "20%".

[0045] As described above, according to the above embodiment, the pointing object detection unit 22 detects, by object detection using a machine-learned learner, a first pointing object 111 that specifies a target object and a second pointing object 112 that specifies data to be used in a replacement process for the target object, among handwritten objects additionally written on the document in the target image 101. The pointing specification unit 23 specifies a target object 113 specified by the detected first pointing object 111, and also specifies data 114 specified by the detected second pointing object 112. The replacement processing unit 24 replaces the specified target object 113 in the target image 101 with the specified data 114.

[0046] This allows object detection to properly detect the instruction objects 111 and 112 that specify the replacement processing targets in the target image 101 without any particular limitation on the writing implements used to write the first instruction object 111 and the second object 112 on the manuscript.

[0047] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0048] For example, in the above embodiment, user edit processing unit 25 may be provided in image forming apparatus 2 instead of processing server 1. Furthermore, processing server 1 may be built into image forming apparatus 2.

[0049] Furthermore, in the above embodiment, the replacement process is performed after the data 114 is converted into text data, but instead, the replacement process may be performed on the data 114 as image data (raster image data). That is, in the above example, characters (numbers, etc.) are written by hand as the data 114, but the data 114 may also be handwritten figures, etc. [Industrial Applicability]

[0050] The present invention is applicable to, for example, an image processing system that executes a replacement process desired by a user on an original image. [Explanation of symbols]

[0051] 1 Processing Server 2. Image forming device 21 Target image acquisition unit 22 Pointing object detection unit 23 Instruction identification part 24 Replacement processing section 25 User edit processing section

Claims

1. a target image acquisition unit that acquires a document image of a document as a target image; a pointing object detection unit that detects, by object detection using a machine-learned learner, a first pointing object that designates a processing target object and a second pointing object that designates data to be used in a replacement process for the processing target object, among handwritten objects additionally written on the document in the target image; an instruction specifying unit that specifies the processing target object specified by the detected first instruction object and that specifies the data specified by the detected second instruction object; a replacement processing unit that performs a replacement process on the identified processing target object in the target image with the identified data; An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein the first pointing object and the second pointing object have different shapes.

3. the first instruction object is a strikethrough; the second pointing object is an encircling line; 3. The image processing system according to claim 2, wherein:

4. 4. The image processing system according to claim 1, wherein the learning device is machine-learned using a plurality of original images including the first pointing object and the second pointing object of a plurality of colors, a plurality of line widths, and a plurality of shapes as training data for the first pointing object and the second pointing object, respectively.

5. the learning device outputs, for each detected object, at least a class classification including the first pointing object and the second pointing object and a bounding box for each detected object; the instruction identification unit identifies the processing target object designated by the detected first instruction object based on the bounding box, and also identifies the data designated by the detected second instruction object; 4. The image processing system according to claim 1, wherein:

6. the instruction specifying unit performs character recognition processing on the image area specified by the second instruction object to specify text data written in the image area; the replacement processing unit erases the processing target object in the target image and adds an object based on the identified text data to a position of the processing target object; 4. The image processing system according to claim 1, wherein:

Images