Image processing system

Abstract

To obtain an image processing system in which a block definition object specifying the processing object is appropriately detected in an object image.SOLUTION: An object detection unit 22 detects a block definition object additionally handwritten onto the manuscript in an object image with object detection using a machine-learned classifier, and derives a confidence score for each detected block definition object. An area extraction part 23 identifies pairs of block definition objects, each specifying a rectangular area, among the detected block definition objects based on confidence levels, then extracts rectangular areas based on the positions of the identified pairs of block definition objects. A processing execution part 24 performs predetermined processing on the extracted rectangular areas in the object image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing system. [Background technology]

[0002] An image processing device identifies the processing items specified by a specified color in the processing list for a document image obtained by scanning a document containing the processing items "delete," "acquire," and "restrict," and then executes the identified processing items on the parts of the document image that are surrounded or filled in with the specified color (see, for example, Patent Document 1).

[0003] Furthermore, an image processing device (a) separates handwritten portions from an original image of a document, (b) determines whether the handwritten portions are an enclosing line surrounding a target area or a classification symbol for classifying the target area, and (c) classifies the image of the target area by the classification symbol (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-27243 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-151722 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, it is conceivable to identify a target area of ​​any shape designated by a handwritten area designation object such as an encircling line, and then perform predetermined image processing on that target area. In this case, if the area designation object is handwritten, there is a possibility that the area designation object will not be properly detected in the target image due to fluctuations in the shape of the area designation object.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an image processing system that can appropriately detect an area designating object that designates a processing target in a target image. [Means for solving the problem]

[0007] The image processing system of the present invention includes a target image acquisition unit that acquires a document image of a document as a target image; an object detection unit that detects area-designating objects that have been additionally handwritten on the document in the target image by object detection using a machine-learned learner and derives a confidence level for each of the detected area-designating objects; an area extraction unit that identifies, from the detected area-designating objects, pairs of area-designating objects that designate one rectangular area based on the confidence levels and extracts rectangular areas based on the positions of the identified pair of area-designating objects; and a processing execution unit that executes a predetermined process on the rectangular areas extracted in the target image. [Effects of the Invention]

[0008] According to the present invention, an image processing system is provided that can appropriately detect an area designating object that designates a processing target in a target image.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a target image including an area designation object. [Figure 3] FIG. 3 is a diagram showing examples of area designation objects of a plurality of shape types. [Figure 4]FIG. 4 is a diagram illustrating the area designation object identified in the target image shown in FIG. [Figure 5] FIG. 5 is a diagram illustrating the representative angles of area designation objects of each shape type. [Figure 6] FIG. 6 is a diagram showing an example of a plurality of first area-designating objects and a plurality of second area-designating objects detected in a target image. [Figure 7] FIG. 7 is a diagram showing an example of a rectangular area specified from the area specifying object shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of an original image newly generated by rearranging an image of a rectangular area extracted from a target image. [Figure 9] FIG. 9 is a diagram showing another example of an original image newly generated by rearranging an image of a rectangular area extracted from a target image. [Figure 10] FIG. 10 is a flowchart illustrating the operation of the image processing system shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention, which includes a processing server 1 and an image forming device 2 (such as a multifunction peripheral) that can communicate with each other via a network.

[0013] The processing server 1 includes a processing unit 11 , a storage unit 12 , and a communication unit 13 .

[0014] The arithmetic processing device 11 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 12 into the RAM and executing it on the CPU. The arithmetic processing device 11 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0015] The storage device 12 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processing described below. The storage device 12 also stores system setting data. The system setting data includes parameters of the learning device described below.

[0016] The communication device 13 is a device that performs data communication with an external device such as the image forming device 2, and is, for example, a network interface or a peripheral device interface.

[0017] On the other hand, the image forming device 2 is equipped with an image reading device, a communication device, an operation panel, etc., and generates an original image from an original document as a target image using the image reading device, and transmits the target image to the processing server 1 using the communication device and receives the processed target image.

[0018] In the processing server 1, the arithmetic processing device 11 operates as the above-mentioned processing units, that is, a target image acquisition unit 21, an object detection unit 22, an area extraction unit 23, a processing execution unit 24, a user edit processing unit 25, and an output processing unit 26.

[0019] The target image acquisition unit 21 acquires (image data of) a document image of a document as a target image from the storage device 12, communication device 13, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a slip, a flyer, etc.

[0020] The object detection unit 22 detects area-specified objects that have been added to a document by handwriting in a target image by object detection using a machine-learned learner (such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Object Detector), or Faster R-CNN (Regions with Convolutional Neural Networks)). As output data, the learner derives, for each detected object, a class classification (whether it is an area-specified object or another object), a bounding box (position and size), and a confidence level of the class classification (a value between 0 and 1).

[0021] Note that the handwritten area specification object is an independent foreground image other than the background. Furthermore, if the size of the original image differs from the size of the input data for this learning device (object detection model), the original image is converted to the size of the input data for the learning device (object detection model). For example, if the size of the original image is 2480 pixels x 3507 pixels, which is A4 size at 300 dpi, and the size of the input data is 640 pixels x 640 pixels, the resolution of the original image is converted to 640 pixels x 640 pixels.

[0022] FIG. 2 is a diagram showing an example of a target image including area designation objects. FIG. 3 is a diagram showing examples of area designation objects of a plurality of shape types. For example, the target image 101 shown in FIG. 2 includes area designation objects 111UL and 111LR. As area designation objects that can be detected by the object detection unit 22, area designation objects 111LL, 111UL, 111UR, and 111LR of shape types as shown in FIG. 3 are set. In other words, the learner performs machine learning using area designation objects having these shape types.

[0023] 2, area designation objects 111UL and 111LR are paired, and area designation objects 111LL and 111UR are also paired, with each pair designating a rectangular area, and the image of that rectangular area is the subject of image processing, which will be described later. In other words, one area designation object in the pair is a first area designation object that designates a first of the four corners of the rectangular area, and the other area designation object in the pair is a second area designation object that designates a second corner diagonally opposite to the first corner.

[0024] Fig. 4 is a diagram illustrating area-specified objects identified in the target image shown in Fig. 2. For example, as shown in Fig. 4, the object detection unit 22 identifies bounding boxes 112-1 to 112-4 of the area-specified objects 111UL, 111LR (that is, the positions and sizes of the objects 111UL, 111LR) by the above-mentioned object detection.

[0025] The learning device described above is machine-learned using a plurality of document images containing pairs of area designation objects for designating a single rectangular area as training data. The area designation objects (here, the four types of area designation objects 111LL, 111UL, 111UR, and 111LR) in the plurality of document images in the training data each have a plurality of colors, a plurality of line widths, and a plurality of shapes. This allows the area designation objects to be properly detected regardless of the writing implement used to write the area designation objects on the document and even if there is variation in the shape due to handwriting, and also allows the area designation objects to be properly detected corresponding to pairs of area designation objects with various positional relationships.

[0026] The area extraction unit 23 identifies a pair of area designating objects that designate one rectangular area from among the detected area designating objects based on the above-mentioned certainty factor, and extracts a rectangular area based on the positions of the identified pair of area designating objects.

[0027] Specifically, the learning device of the object detection unit 22 detects the first area-designated object and the second area-designated object separately, and the area extraction unit 23 identifies pairs of the first area-designated object and the second area-designated object, and extracts rectangular areas based on the positions of the first area-designated object and the second area-designated object in the identified pairs.

[0028] Here, the area extraction unit 23 extracts an image within the bounding box of the area-specified object from the target image, binarizes the image to generate a binary image, detects contours (boundary lines between the object and the background) in the binary image, identifies the bounding box of the contours, and regards the contour with the largest area of ​​the detected contours as the contour of the area-specified object, and sets this bounding box as the bounding box of the area-specified object. Note that the binarization at this time is performed using existing methods such as Otsu's binarization method or k-means (n=2).

[0029] Fig. 5 is a diagram illustrating the representative corner of the area designation object of each shape type. Specifically, for example, as shown in Fig. 5, for the area designation object of each shape type, one corner 113 of the four corners of the bounding box 112 is determined as the representative corner, and the area extraction unit 23 specifies a rectangular area by taking the representative corner of the first area designation object as the first corner and the representative corner of the second area designation object as the second corner diagonal to the first corner, and extracts (an image of) that rectangular area.

[0030] Furthermore, when identifying a plurality of pairs of area designating objects that designate a plurality of rectangular areas from the detected area designating objects, the area extraction unit 23 (a) identifies a plurality of combination patterns for the plurality of pairs from the plurality of first area designating objects and the plurality of second area designating objects based on the positions of the first area designating object and the second area designating object, (b) selects, from the plurality of combination patterns, a combination pattern that has the greatest sum of the certainty factors of the first area designating object and the plurality of second area designating objects that make up the combination pattern, and (c) identifies a plurality of pairs using the selected combination pattern.

[0031] Fig. 6 is a diagram showing an example of a plurality of first area-designating objects and a plurality of second area-designating objects detected in a target image. Fig. 7 is a diagram showing an example of rectangular areas identified from the area-designating objects shown in Fig. 6. In Fig. 6, area-designating objects 111UL-1 and 111UL-2 are first area-designating objects, and area-designating objects 111LR-1, 111LR-2, and 111LR-3 are second area-designating objects, and the bounding boxes and certainty factors of each have been derived by the object detection unit 22.

[0032] 6, the area extraction unit 23 first identifies a combination pattern of a plurality of pairs as follows. Note that, depending on the shape type of a certain area designation object, the angle range of another area designation object that is paired with a certain area designation object is limited to a specific 90 degrees. Here, for an area designation object 111UL-i (i=1, 2), the area designation objects that are paired with the area designation object 111UL-i are limited to those that exist to the right and below the area designation object 111UL-i, and for an area designation object 111LR-j (j=1, 2, 3), the area designation objects that are paired with the area designation object 111LR-j are limited to those that exist to the left and above the area designation object 111LR-j.

[0033] Pattern #1: A pair of area-specified object 111UL-1 and area-specified object 111LR-1, and a pair of area-specified object 111UL-2 and area-specified object 111LR-2

[0034] Pattern #2: A pair of area-specified object 111UL-1 and area-specified object 111LR-1, and a pair of area-specified object 111UL-2 and area-specified object 111LR-3

[0035] Pattern #3: A pair of the area-specified object 111UL-1 and the area-specified object 111LR-2, and a pair of the area-specified object 111UL-2 and the area-specified object 111LR-3

[0036] Pattern #4: A pair of area-specified object 111UL-1 and area-specified object 111LR-3, and a pair of area-specified object 111UL-2 and area-specified object 111LR-2

[0037] The region extraction unit 23 then calculates the sum of the certainty factors of each pattern. The sum of the certainty factors of pattern #1 is 3.61 (= 0.92 + 0.90 + 0.88 + 0.91), the sum of the certainty factors of pattern #2 is 3.05 (= 0.92 + 0.90 + 0.88 + 0.35), the sum of the certainty factors of pattern #3 is 3.06 (= 0.92 + 0.91 + 0.88 + 0.35), and the sum of the certainty factors of pattern #4 is 3.06 (= 0.92 + 0.35 + 0.88 + 0.91).

[0038] Therefore, in this case, the area extraction unit 23 selects pattern #1 with the largest sum, identifies a pair of area designation object 111UL-1 and area designation object 111LR-1, and a pair of area designation object 111UL-2 and area designation object 111LR-2, and identifies and extracts rectangular areas 121-1 and 121-2 for these pairs, as shown in FIG. 7, for example. Note that the area designation objects are appropriately deleted from the extracted image. For example, even if area designation object 111LR-3 is obtained due to a false detection, the certainty factor is low, so an appropriate pair is identified based on the certainty factor.

[0039] Furthermore, the processing execution unit 24 executes a predetermined process on the rectangular area extracted from the target image 101 .

[0040] Fig. 8 is a diagram showing an example of an original image newly generated by rearranging an image of a rectangular area extracted from a target image, and Fig. 9 is a diagram showing another example of an original image newly generated by rearranging an image of a rectangular area extracted from a target image.

[0041] 8, the processing execution unit 24 rearranges the image 131 extracted from the rectangular area 121 identified in the target image 101 to generate a new original image 141. At this time, the image 131 is arranged (arranged) in the original image 141 at a position and size specified in advance (for example, left edge aligned in the main scanning direction, center aligned in the sub-scanning direction, and enlarged by 170%).

[0042] The above-mentioned positions and sizes may be determined by the user (user operation or user setting). For example, the position in the main scanning direction may be selected from multiple options such as center alignment or left edge alignment, the position in the sub-scanning direction may be selected from multiple options such as center alignment or top edge alignment, and the size may be selected from an enlargement rate of 100% to 200%, each selected by user operation or user setting. The enlargement rate may be set to the maximum value that fits within the size of the print sheet at the time of printing. Also, while FIG. 8 shows multiple (two) rectangular areas 121, a new original image may be generated by extracting and rearranging one rectangular area from one target image, as shown in FIG. 9. Also, the orientation of the newly generated original image may be rotated to match the orientation of the print sheet at the time of printing, as shown in FIG. 9.

[0043] Furthermore, user edit processing unit 25 performs editing based on a user operation on the processing results (such as the above-mentioned new document image 141) by processing execution unit 24. Specifically, output processing unit 26 displays the processing results to the user, and user edit processing unit 25 edits the processing results in accordance with the user operation.

[0044] Furthermore, the output processing unit 26 outputs the above-mentioned processing results. For example, the output processing unit 26 transmits the processing results to the image forming device 2, causes the image forming device 2 to print the processing results, or stores the processing results in a predetermined storage device (such as the image forming device 2 or another server).

[0045] Next, the operation of the image processing system will be described with reference to Fig. 10, which is a flowchart illustrating the operation of the image processing system shown in Fig. 1.

[0046] The target image acquisition unit 21 of the processing server 1 acquires a target image (step S1). Here, the above-mentioned area designation object 111 is additionally written by hand on a document printed by the image forming device 2, and then the image of the document is read to obtain a document image, and the document image (image data) is sent from the image forming device 2 to the processing server 1 as the target image.

[0047] When the target image is acquired in the processing server 1, the object detection unit 22 detects the area-specifying object 111 in the target image 101 by object detection (step S2).

[0048] Based on the detected area-specifying object 111, the area extracting unit 23 identifies the rectangular area handwritten by the user in the target image 101, and extracts an image of the rectangular area from the target image 101 (step S3).

[0049] Then, the processing execution unit 24 performs a predetermined process (such as generating a new document image as described above) based on the image of the extracted rectangular area (step S4).

[0050] Thereafter, the output processing unit 26 displays the processing result to the user, for example, on the image forming device 2 (step S5).

[0051] The user visually checks the displayed processing results, determines whether the image processing has been performed appropriately by the processing execution unit 24, and performs a user operation indicating the determination result on the image forming device 2 (the input device of the operation panel). If the user determines that the image processing has not been performed appropriately, the user performs a user operation for editing on the image forming device 2 (the input device of the operation panel). The image forming device 2 notifies the user editing processing unit 25 of the processing server 1 of the user operation.

[0052] Based on the notification, user edit processing unit 25 determines whether the image processing has been performed appropriately (step S6). If it is determined that the image processing has been performed appropriately, output processing unit 26 outputs the processing results as necessary.

[0053] On the other hand, if it is determined that the image processing has not been performed appropriately, the user edit processing unit 25 edits the processing result (such as the new document image described above) in accordance with the user's operation (step S7), after which the output processing unit 26 outputs the edited processing result.

[0054] As described above, according to the above embodiment, the object detection unit 22 detects area-specifying objects that have been added by hand to a document in a target image through object detection using a machine-learned learner, and derives a confidence level for each of the detected area-specifying objects. The area extraction unit 23 identifies pairs of area-specifying objects that specify a single rectangular area from the detected area-specifying objects based on the confidence levels, and extracts rectangular areas based on the positions of the identified pair of area-specifying objects. The processing execution unit 24 executes a predetermined process on the rectangular areas extracted in the target image.

[0055] This allows object detection to appropriately detect an area designation object that designates an image processing target in a target image without any particular limitations on the writing implement used to write the area designation object on the document.

[0056] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0057] For example, in the first and second embodiments, the user edit processing unit 25 may be provided in the image forming apparatus 2 instead of the processing server 1. The processing server 1 may also be built into the image forming apparatus 2. [Industrial Applicability]

[0058] The present invention is applicable to, for example, an image processing system that executes image processing on a user-specified portion of an original image. [Explanation of symbols]

[0059] 21 Target image acquisition unit 22 Object detection unit 23 Region extraction part 24 Processing execution unit

Claims

1. a target image acquisition unit that acquires a document image of a document as a target image; an object detection unit that detects area-specifying objects that are additionally written by hand on the document in the target image by object detection using a machine-learned learner, and derives a confidence level for each of the detected area-specifying objects; an area extraction unit that identifies a pair of area designating objects that designate one rectangular area from among the detected area designating objects based on the degree of certainty, and extracts a rectangular area based on the positions of the identified pair of area designating objects; a processing execution unit that executes a predetermined process on the rectangular area extracted from the target image; An image processing system comprising:

2. the area designation objects include a first area designation object that designates a first corner of the four corners of the rectangular area and a second area designation object that designates a second corner diagonally opposite to the first corner; the learning device distinguishes between and detects the first area-specifying object and the second area-specifying object; the area extraction unit identifies a pair of the first area designating object and the second area designating object, and extracts the rectangular area based on positions of the first area designating object and the second area designating object of the identified pair; 2. The image processing system according to claim 1, wherein:

3. 3. The image processing system according to claim 2, wherein the area extraction unit extracts the rectangular area by setting one of four corners of a bounding box of the first area-specifying object as the first corner and one of four corners of a bounding box of the second area-specifying object as the second corner.

4. 3. The image processing system according to claim 2, wherein, when identifying a plurality of pairs of area designating objects that designate a plurality of rectangular areas from the detected area designating objects, the area extraction unit: (a) identifies a plurality of combination patterns for the plurality of pairs from the plurality of first area designating objects and the plurality of second area designating objects based on the positions of the first area designating object and the second area designating object; (b) selects, from the plurality of combination patterns, a combination pattern in which the sum of certainty factors of the first area designating object and the plurality of second area designating objects that make up that combination pattern is the greatest; and (c) identifies the plurality of pairs using the selected combination pattern.

5. the learning device is trained by machine learning using a plurality of original images including a pair of area specifying objects for specifying one rectangular area as training data; the area-specifying objects in the plurality of original images in the training data have a plurality of colors, a plurality of line widths, and a plurality of shapes; 5. The image processing system according to claim 1, wherein:

Images