Image processing system

Abstract

To obtain an image processing system in which a region designation object that specifies a processing target in a target image is appropriately detected.SOLUTION: An object detection unit 22 detects region designation objects that are additionally handwritten onto a document, within a target image using object detection with a machine-learned model. A region extraction unit 23 extracts free-form regions specified by the detected region designation objects. A processing execution unit 24 performs predetermined processing on the extracted free-form regions within the target image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image processing system. [Background technology]

[0002] An image processing device identifies the processing items specified by a specified color in the processing list for a document image obtained by scanning a document containing the processing items "delete," "acquire," and "restrict," and then executes the identified processing items on the parts of the document image that are surrounded or filled in with the specified color (see, for example, Patent Document 1).

[0003] Furthermore, an image processing device (a) separates handwritten portions from an original image of a document, (b) determines whether the handwritten portions are an enclosing line surrounding a target area or a classification symbol for classifying the target area, and (c) classifies the image of the target area by the classification symbol (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-27243 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-151722 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, it is conceivable to identify a target area of ​​any shape designated by a handwritten area designation object such as an encircling line, and then perform predetermined image processing on that target area. In this case, if the area designation object is handwritten, there is a possibility that the area designation object will not be properly detected in the target image due to fluctuations in the shape of the area designation object.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an image processing system that can appropriately detect an area designating object that designates a processing target in a target image. [Means for solving the problem]

[0007] The image processing system of the present invention includes a target image acquisition unit that acquires a document image of a document as a target image, an object detection unit that detects an area-specifying object that has been additionally handwritten on the document in the target image by object detection using a machine-learned learner, an area extraction unit that extracts a free-form area specified by the detected area-specifying object, and a processing execution unit that performs a predetermined process on the free-form area extracted in the target image. [Effects of the Invention]

[0008] According to the present invention, an image processing system is provided that can appropriately detect an area designating object that designates a processing target in a target image.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a target image including an area designation object. [Figure 3] FIG. 3 is a diagram illustrating the area designation object identified in the target image shown in FIG. [Figure 4] FIG. 4 is a diagram showing an image area within the area specification object in FIG. [Figure 5] FIG. 5 is a diagram showing an example of a free-form region specified from a region specifying object. [Figure 6]FIG. 6 is a diagram showing an example of an extracted free-form region. [Figure 7] FIG. 7 is a diagram showing an example of an original image newly generated by rearranging the image of the free-form area 131 shown in FIG. [Figure 8] FIG. 8 is a flowchart illustrating the operation of the image processing system shown in FIG. [Figure 9] FIG. 9 is a flowchart illustrating the specification of a free-form region according to the second embodiment. [Figure 10] FIG. 10 is a diagram showing an example of an area-specifying object whose inner contour can be detected. [Figure 11] FIG. 11 is a diagram showing an example of an area-specified object whose inner contour cannot be detected. [Figure 12] FIG. 12 is a diagram showing an example of an XOR image used to estimate a line width when an inner contour cannot be detected. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] Embodiment 1

[0013] 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention, which includes a processing server 1 and an image forming device 2 (such as a multifunction peripheral) that can communicate with each other via a network.

[0014] The processing server 1 includes a processing unit 11 , a storage unit 12 , and a communication unit 13 .

[0015] The arithmetic processing device 11 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 12 into the RAM and executing it on the CPU. The arithmetic processing device 11 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0016] The storage device 12 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processing described below. The storage device 12 also stores system setting data. The system setting data includes parameters of the learning device described below.

[0017] The communication device 13 is a device that performs data communication with an external device such as the image forming device 2, and is, for example, a network interface or a peripheral device interface.

[0018] On the other hand, the image forming device 2 is equipped with an image reading device, a communication device, an operation panel, etc., and generates an original image from an original document as a target image using the image reading device, and transmits the target image to the processing server 1 using the communication device and receives the processed target image.

[0019] In the processing server 1, the arithmetic processing device 11 operates as the above-mentioned processing units, that is, a target image acquisition unit 21, an object detection unit 22, a region extraction unit 23, a processing execution unit 24, a user edit processing unit 25, and an output processing unit 26.

[0020] The target image acquisition unit 21 acquires (image data of) a document image of a document as a target image from the storage device 12, communication device 13, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a slip, a flyer, etc.

[0021] The object detection unit 22 detects area-specified objects that have been handwritten on the original document in the target image by object detection using a machine-learned learner (such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Object Detector), or Faster R-CNN (Regions with Convolutional Neural Networks)).

[0022] Note that the handwritten area-specified object is an independent foreground image, not a background. Furthermore, if the size of the original image differs from the size of the input data of this learning device (object detection model), the original image is converted to the size of the input data of the learning device (object detection model). For example, if the size of the original image is 2480 pixels x 3507 pixels, which is A4 size at 300 dpi, and the size of the input data is 640 pixels x 640 pixels, the resolution of the original image is converted to the size of the original image as 640 pixels x 640 pixels. Furthermore, the output data of this learning device includes, for each detected object, the class classification (whether it is an area-specified object or another object), the bounding box (position and size), and the confidence level of the class classification (a value between 0 and 1).

[0023] 2 is a diagram showing an example of a target image including an area designation object. For example, as shown in FIG. 2, a target image 101 includes an area designation object 111, which in this embodiment is an encircling line (a closed curve of any shape).

[0024] When the area designation object 111 is an encircling line, the image area inside the encircling line in the target image 101 is designated as the target of image processing, which will be described later.

[0025] Fig. 3 is a diagram illustrating an area designation object identified in the target image shown in Fig. 2. Fig. 4 is a diagram showing an image area within the area designation object in Fig. 3.

[0026] Specifically, for example, as shown in FIG. 3, the object detection unit 22 identifies a bounding box 111a of the area specification object 111 (that is, the position and size of the object 111) by the above-mentioned object detection.

[0027] Furthermore, the above-mentioned learning device has been machine-learned using area-specifying objects (encircling lines in this case) of multiple colors, multiple line widths, and multiple shapes as training data, which allows the area-specifying object to be properly detected regardless of the writing implement used to write the area-specifying object on the manuscript and the shape of the object being handwritten.

[0028] Fig. 5 is a diagram showing an example of a free-form region specified from a region specifying object. Fig. 6 is a diagram showing an example of an extracted free-form region. Region extraction unit 23 specifies a free-form region 131 specified by a detected region specifying object 111 from target image 101 or image region 121, for example, as shown in Fig. 5, and extracts free-form region 131, for example, as shown in Fig. 6.

[0029] Furthermore, the processing execution unit 24 executes a predetermined process on the free-form region 131 extracted from the target image 101 .

[0030] Fig. 7 is a diagram showing an example of an original image newly generated by rearranging the image of the free-form shape area 131 shown in Fig. 6. For example, as shown in Fig. 7, the processing execution unit 24 rearranges the image of the free-form shape area 131 to generate a new original image 141. At this time, the image of the free-form shape area 131 is placed within the original image 141 at a position and size specified in advance (for example, left edge aligned in the main scanning direction, center aligned in the sub-scanning direction, and enlarged by 170%).

[0031] The above-mentioned position and size may be determined by the user (user operation or user setting). For example, the position in the main scanning direction may be selected from a plurality of options such as center alignment or left edge alignment, the position in the sub-scanning direction may be selected from a plurality of options such as center alignment or top edge alignment, and the size may be selected from a magnification rate of 100% to 200%, each of which may be selected by user operation or user setting. The magnification rate may be set to the maximum value that fits within the size of the print sheet at the time of printing. Also, although there is one free-form shape region here, multiple free-form shape regions may be extracted from one target image and rearranged to generate a new original image.

[0032] Furthermore, user edit processing unit 25 performs editing based on a user operation on the processing results (such as the above-mentioned new document image 141) by processing execution unit 24. Specifically, output processing unit 26 displays the processing results to the user, and user edit processing unit 25 edits the processing results in accordance with the user operation.

[0033] Furthermore, the output processing unit 26 outputs the above-mentioned processing results. For example, the output processing unit 26 transmits the processing results to the image forming device 2, causes the image forming device 2 to print the processing results, or stores the processing results in a predetermined storage device (such as the image forming device 2 or another server).

[0034] Next, the operation of the image processing system will be described with reference to Fig. 8, which is a flowchart illustrating the operation of the image processing system shown in Fig. 1.

[0035] The target image acquisition unit 21 of the processing server 1 acquires a target image (step S1). Here, the above-mentioned area designation object 111 is additionally written by hand on a document printed by the image forming device 2, and then the image of the document is read to obtain a document image, and the document image (image data) is sent from the image forming device 2 to the processing server 1 as the target image.

[0036] When the target image is acquired in the processing server 1, the object detection unit 22 detects the area-specifying object 111 in the target image 101 by object detection (step S2).

[0037] Based on the detected area-specifying object 111, the area extracting unit 23 identifies the free-form area 131 handwritten by the user in the target image 101, and extracts the free-form area 131 from the target image 101 (step S3).

[0038] Then, the processing execution unit 24 performs a predetermined process (such as generating a new document image as described above) based on the extracted free-form region 131 (step S4).

[0039] Thereafter, the output processing unit 26 displays the processing result to the user, for example, on the image forming device 2 (step S5).

[0040] The user visually checks the displayed processing results, determines whether the image processing has been performed appropriately by the processing execution unit 24, and performs a user operation indicating the determination result on the image forming device 2 (the input device of the operation panel). If the user determines that the image processing has not been performed appropriately, the user performs a user operation for editing on the image forming device 2 (the input device of the operation panel). The image forming device 2 notifies the user editing processing unit 25 of the processing server 1 of the user operation.

[0041] Based on the notification, user edit processing unit 25 determines whether the image processing has been performed appropriately (step S6). If it is determined that the image processing has been performed appropriately, output processing unit 26 outputs the processing results as necessary.

[0042] On the other hand, if it is determined that the image processing has not been performed appropriately, the user edit processing unit 25 edits the processing result (such as the new document image described above) in accordance with the user's operation (step S7), after which the output processing unit 26 outputs the edited processing result.

[0043] As described above, according to the first embodiment, the object detection unit 22 detects an area-specifying object that has been added to a document by handwriting in a target image by object detection using a machine-learned learner. The area extraction unit 23 extracts a free-form area specified by the detected area-specifying object. The processing execution unit 24 executes a predetermined process on the free-form area extracted from the target image.

[0044] As a result, the object detection does not particularly limit the writing implement used to write the area designation object 111 on the document, and the area designation object 111 that designates the image processing target in the target image 101 can be appropriately detected.

[0045] Embodiment 2

[0046] Fig. 9 is a flowchart illustrating how to specify a free-form region in embodiment 2. Fig. 10 is a diagram showing an example of a region-specified object whose inner contour can be detected. Fig. 11 is a diagram showing an example of a region-specified object whose inner contour cannot be detected.

[0047] In the second embodiment, in specifying a free-form region, first, the region extraction unit 23 binarizes the image within the bounding box 111a of the region specification object 111 to generate a binary image (step S11), and then contracts white regions (background regions) in the binary image (expands black regions) (step S12). Note that the binarization at this time is performed using an existing method such as Otsu's binarization method or k-means (n=2). The width of the contraction may be a preset fixed value or a value specified by the user.

[0048] Next, the area extraction unit 23 extracts a plurality of contours 211 to 216 as closed curves in the contracted binarized image 201 (i.e., within the bounding box 111a of the area specification object 111) using existing technology, as shown in, for example, FIG. 10 (step S13).

[0049] Then, the area extraction unit 23 determines whether or not to select the contour having the largest area of ​​the bounding box of the extracted contours 211 to 216 as the outer contour 211 (step S14), and whether or not to select the contour having the second largest area of ​​the bounding box of the extracted contours as the inner contour 212 (step S15).

[0050] Here, as shown in FIG. 10, for example, if the area ratio (S2 / S1, S1: area of ​​bounding box 211a, S2: area of ​​bounding box 212a) of the second largest area bounding box 212a to the largest area bounding box 211a among the bounding boxes of the multiple contours 211 to 216 is equal to or greater than a predetermined threshold (a value less than 1, for example, 0.95), the area extraction unit 23 determines the contour of the second largest area bounding box 212a as the inner contour 212 and extracts the area inside the inner contour 212 as the free-form area 221 (step S16).

[0051] On the other hand, for example, as shown in FIG. 11, if the area-specified object is connected to an internal object and the above-mentioned area ratio (in FIG. 11, the area ratio of the bounding box 312a of the contour 312 to the bounding box 311a of the outer contour 311 among the contours 311 to 316 detected in the binarized image 301 after contraction) is less than the above-mentioned predetermined threshold, the area extraction unit 23 does not regard the contour 312 of the bounding box with the second largest area as the inner contour, but estimates the line width of the enclosing line as the area-specified object (step S17), and specifies and extracts the enclosing line as a closed curve of the estimated line width, and the area inside the closed curve as a free-form area (step S18).

[0052] Here, the above-mentioned estimation of line width will be explained. Fig. 12 is a diagram showing an example of an XOR image used for estimating line width when an inner contour cannot be detected.

[0053] The area extraction unit 23 generates a first binary image 321 obtained by filling in the inside of the outer contour 311, as shown in FIG. 12, and a second binary image 322 obtained by expanding the white parts of the first binary image 321 by a predetermined expansion width, and derives an exclusive OR image (XOR image) 323 between the binary image 321 and the binary image 322.

[0054] The region extraction unit 23 also derives a logical product image (AND image) of the XOR image and a binary image obtained from the image inside the outer contour of the target image. The AND image is an image in which pixels that are black (foreground) in both images are treated as black pixels (foreground pixels), and pixels that are white (background) in either image are treated as white pixels (background pixels).

[0055] Then, the area extraction unit 23 varies the expansion width and estimates the line width based on the area of ​​the XOR image 323 and the area of ​​the AND image.

[0056] For example, the expansion width is gradually increased, and when the area ratio of the AND image to the XOR image 323 (Sa / Sx, Sa: area of ​​the AND image, Sx: area of ​​the XOR image 323) becomes less than a predetermined threshold (for example, any value between 0.85 and 0.95), the expansion width is taken as the estimated value of the line width. In other words, when the expansion width of the XOR image 323 is narrow, the AND image matches the XOR image 323, so the area ratio becomes 1. When the expansion width of the XOR image 323 becomes wide, part of the AND image (a part other than the part where the area specification object is connected to the internal object) becomes narrower than the XOR image 323, so the area ratio becomes less than 1. Therefore, the expansion width when the area ratio is no longer 1 is taken as the estimated value of the line width.

[0057] The other configurations and operations of the image processing system according to the second embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0058] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0059] For example, in the first and second embodiments, the user edit processing unit 25 may be provided in the image forming apparatus 2 instead of the processing server 1. The processing server 1 may also be built into the image forming apparatus 2.

[0060] Furthermore, in the second embodiment, if the inner contour cannot be detected, the extraction of the free-form region based on the line width estimation described above may not be performed, and the processing execution unit 24 may not perform image processing. [Industrial Applicability]

[0061] The present invention is applicable to, for example, an image processing system that executes image processing on a user-specified portion of an original image. [Explanation of symbols]

[0062] 21 Target image acquisition unit 22 Object detection unit 23 Region extraction part 24 Processing execution unit

Claims

1. a target image acquisition unit that acquires a document image of a document as a target image; an object detection unit that detects an area-specifying object that is additionally handwritten on the document in the target image by object detection using a machine-learned learner; an area extraction unit that extracts a free-form area designated by the detected area designating object; a processing execution unit that executes a predetermined process on the free-form region extracted from the target image; An image processing system comprising:

2. the area designation object is an encircling line, the region extraction unit (a) extracts a plurality of contours as closed curves within a bounding box of the region specification object; (b) of the extracted plurality of contours, the contour with the largest area of ​​the bounding box of the contour is defined as an outer contour, and the contour with the second largest area of ​​the bounding box of the contour is defined as an inner contour, and extracts the region inside the inner contour as the free-form region; 2. The image processing system according to claim 1, wherein:

3. 3. The image processing system according to claim 2, wherein the region extraction unit: (a) if an area ratio of the second-largest bounding box to the largest bounding box among the bounding boxes of the plurality of contours is equal to or greater than a predetermined threshold, selects the contour of the bounding box with the second largest area as the inner contour and extracts the area inside the inner contour as the free-form region; and (b) if the area ratio is less than the predetermined threshold, does not select the contour of the second-largest area as the inner contour, estimates a line width of the encircling line, treats the encircling line as a closed curve of the estimated line width, and extracts the area inside the closed curve as the free-form region.

4. The image processing system of claim 3, wherein the region extraction unit (a) derives an exclusive OR image between a first binary image obtained by filling in the inside of the outer contour and a second binary image obtained by expanding the white portions of the first binary image by a predetermined expansion width, (b) derives a logical product image between the exclusive OR image and a binary image obtained from an image inside the outer contour in the target image, and (c) changes the expansion width and estimates the line width based on the area of ​​the exclusive OR image and the area of ​​the logical product image.

5. 5. The image processing system according to claim 1, wherein the learning device performs machine learning on the area-specifying object using as training data a plurality of original images including the area-specifying object of a plurality of colors, a plurality of line widths, and a plurality of shapes.

Images