Image processing system

Abstract

To appropriately detect an instruction object that specifies a replacement processing target in a target image.SOLUTION: An instruction object detection unit detects, through object detection using a machine-learned model, a first instruction object 111 that specifies a processing target object among handwritten character objects additionally written on a document in the target image 101, and a second instruction object 112 that specifies a replacement object to be used for the replacement processing on the processing target object. An instruction identification unit identifies a processing target object 113 specified by the detected first instruction object 111 and identifies a replacement object 114 specified by the detected second instruction object 112. The replacement processing unit performs replacement processing on the basis of the identified replacement object 114 on the identified processing target object 113 in the target image 101.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing system. [Background technology]

[0002] An image processing device identifies the processing items specified by a specified color in the processing list for a document image obtained by scanning a document containing the processing items "delete," "acquire," and "restrict," and then executes the identified processing items on the parts of the document image that are surrounded or filled in with the specified color (see, for example, Patent Document 1).

[0003] Furthermore, an image processing device (a) separates handwritten portions from an original image of a document, (b) determines whether the handwritten portions are an enclosing line surrounding a target area or a classification symbol for classifying the target area, and (c) classifies the image of the target area by the classification symbol (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-27243 [Patent Document 2] Japanese Patent Application Laid-Open No. 2012-151722 Summary of the Invention [Problem to be solved by the invention]

[0005] As described above, it is conceivable to identify a target area designated by a handwritten pointing object such as a surrounding line and replace the target area with other content. In this case, if the pointing object is handwritten, there is a possibility that the pointing object will not be properly detected in the target image due to fluctuations in the shape of the pointing object.

[0006] The present invention has been made in view of the above-mentioned problems, and has as its object to provide an image processing system that can appropriately detect a pointing object that designates a target for replacement processing in a target image. [Means for solving the problem]

[0007] The image processing system of the present invention includes a target image acquisition unit that acquires a document image of a document as a target image; an instruction object detection unit that detects, by object detection using a machine-learned learner, a first instruction object that designates a target object and a second instruction object that designates a replacement object to be used in a replacement process for the target object, among handwritten character objects that have been additionally written on the document in the target image; an instruction identification unit that identifies the target object designated by the detected first instruction object and identifies the replacement object designated by the detected second instruction object; and a replacement processing unit that performs a replacement process on the identified target object in the target image based on the identified replacement object. [Effects of the Invention]

[0008] According to the present invention, an image processing system is provided that can appropriately detect a pointing object that designates a replacement processing target in a target image.

[0009] The above and other objects, features and advantages of the present invention will become more apparent from the following detailed description taken in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention. [Figure 2] FIG. 2 is a diagram showing an example of a target image including a first pointing object and a second pointing object. [Figure 3]FIG. 3 is a diagram illustrating a pointing object identified in the target image shown in FIG. [Figure 4] FIG. 4 is a diagram illustrating a character object (character area) identified in the target image shown in FIG. [Figure 5] FIG. 5 is a diagram showing an example of a target image after the replacement process for the target image shown in FIG. [Figure 6] FIG. 6 is a diagram showing another example of the target image including the first pointing object and the second pointing object in FIG. [Figure 7] FIG. 7 is a flowchart illustrating the operation of the image processing system shown in FIG. [Figure 8] FIG. 8 is a diagram showing another example of a target image including a first pointing object and a second pointing object. [Figure 9] FIG. 9 is a diagram illustrating a pointing object identified in the target image shown in FIG. [Figure 10] FIG. 10 is a diagram showing an example of a target image after the replacement process for the target image shown in FIG. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] Embodiment 1

[0013] 1 is a block diagram showing the configuration of an image processing system according to an embodiment of the present invention, which includes a processing server 1 and an image forming device 2 (such as a multifunction peripheral) that can communicate with each other via a network.

[0014] The processing server 1 includes a processing unit 11 , a storage unit 12 , and a communication unit 13 .

[0015] The arithmetic processing device 11 includes a computer, which executes programs to function as various processing units. Specifically, the computer includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), etc., and functions as a predetermined processing unit by loading a program stored in the ROM or storage device 12 into the RAM and executing it on the CPU. The arithmetic processing device 11 may also include an ASIC (Application Specific Integrated Circuit) that functions as a specific processing unit.

[0016] The storage device 12 is a non-volatile storage device such as a flash memory, and stores programs and data necessary for the processing described below. The storage device 12 also stores system setting data. The system setting data includes parameters of the learning device described below.

[0017] The communication device 13 is a device that performs data communication with an external device such as the image forming device 2, and is, for example, a network interface or a peripheral device interface.

[0018] On the other hand, the image forming device 2 is equipped with an image reading device, a communication device, an operation panel, etc., and generates a document image from a document (menu, flyer, etc.) as a target image using the image reading device, and transmits the target image to the processing server 1 using the communication device and receives the processed target image.

[0019] In the processing server 1, the arithmetic processing device 11 operates as the above-mentioned processing units, that is, a target image acquisition unit 21, a pointed object detection unit 22, a character area detection unit 23, a pointed specification unit 24, a replacement processing unit 25, and an output processing unit 26.

[0020] The target image acquisition unit 21 acquires (image data of) a document image of a document as a target image from the storage device 12, communication device 13, etc., and stores it in RAM, etc. For example, this document is a printout output from a printing device, and this document image is read from the document by an image reading device. This document is, for example, a slip, a flyer, etc.

[0021] The instruction object detection unit 22 detects a first instruction object that specifies the object to be processed and a second instruction object that specifies a replacement object to be used in the replacement process for the object to be processed, among handwritten objects that are additionally written on the original in the target image, by object detection using a machine-learned learner (such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Object Detector), or Faster R-CNN (Regions with Convolutional Neural Networks)).

[0022] Note that handwritten objects are separate, individual foreground images that do not include the background. Furthermore, by performing machine learning using training data including manuscript images in which various handwritten pointing objects are superimposed on printed objects, pointing objects can be properly detected even when they are superimposed on printed objects in the target image. Furthermore, if the size of the manuscript image differs from the size of the input data to this learning device (object detection model), the manuscript image is converted to match the size of the input data to the learning device (object detection model). For example, if the manuscript image size is 2480 pixels x 3507 pixels, which is A4 size at 300 dpi, and the input data size is 640 pixels x 640 pixels, the resolution of the manuscript image is converted to 640 pixels x 640 pixels. Furthermore, the output data from this learning device includes, for each detected object, the class classification (whether it is a first pointing object, a second pointing object, or another object), a bounding box (position and size), and a confidence level of the class classification (a value between 0 and 1).

[0023] Fig. 2 is a diagram illustrating an example of a target image including a first instruction object and a second instruction object, Fig. 3 is a diagram illustrating instruction objects identified in the target image shown in Fig. 2.

[0024] The first and second pointing objects have different shapes. In this embodiment, the first pointing object 111 is a strikeout line (two straight lines, for example), and the second pointing object 112 is an encircling line (any closed curve, such as a circle), as shown in FIG. 2.

[0025] 2, for example, a first pointing object 111 is depicted superimposed on an object 113 (indicated as "14" in FIG. 2) to be replaced by the replacement process in the target image 101. Also, as shown in FIG. 2, for example, a replacement object 114 (indicated as "13" in FIG. 2) is depicted in the area indicated by the second pointing object 112 (here, the area inside the encircling line) in the target image 101.

[0026] The learning device performs machine learning using training data of first and second pointing objects of multiple colors, multiple line widths, and multiple shapes, respectively, for the first and second pointing objects. This allows the first and second pointing objects to be properly detected regardless of the writing implement used to write them and the variations in shape due to handwriting.

[0027] Then, as shown in FIG. 3, for example, the instruction object detection unit 22 identifies a bounding box 111a of the first instruction object 111 (i.e., the position and size of the object 111) and a bounding box 112a of the second instruction object 112 (i.e., the position and size of the object 112) through the above-described object detection.

[0028] FIG. 4 is a diagram illustrating a character object (character area) identified in the target image shown in FIG.

[0029] Furthermore, the character area detection unit 23 detects character objects 121 (character areas) of printed characters and handwritten characters in the target image 101 by using an existing character recognition process, for example, as shown in Fig. 4. Specifically, the character area detection unit 23 identifies a bounding box of each character object 121.

[0030] The instruction specifying unit 24 also specifies the processing target object 113 specified by the detected first instruction object 111 , and specifies the substitute object 114 specified by the detected second instruction object 112 .

[0031] Specifically, the character object 121 closest to the position of the first pointing object 111 identified as described above (i.e., the character object 121 having a bounding box overlapping the bounding box of the first pointing object 111) is identified as the target object 113, and the character object 121 closest to the position of the second pointing object 112 identified as described above is identified as the replacement object 114. Based on the relative positional relationship between these objects, the pointing identification unit 24 also associates the first pointing object 111 and the second pointing object 112, which are adjacent to each other in the target image 101, with each other.

[0032] FIG. 5 is a diagram showing an example of a target image after the replacement process for the target image shown in FIG.

[0033] The replacement processing unit 25 performs replacement processing on the identified processing target object 113 in the target image 101 based on the identified replacement object 114 described above, as shown in FIG. 5, for example.

[0034] In embodiment 1, the replacement processing unit 25 erases the processing target object 113 in the target image 101 and adds a replacement object 114 (raster image) specified by the detected second instruction object 112 as an image object at the position of the processing target object 113.

[0035] At this time, the replacement processing unit 25 generates the replacement object 114 to be added so that it has the same font type, color, and size as the processing target object 113. At this time, for example, the size of the replacement object 114 to be added is converted so that the bounding box of the replacement object 114 to be added (here, the banding box as the character object 121) matches the size (width or height) of the bounding box of the processing target object 113 (here, the banding box as the character object 121).

[0036] 6 is a diagram illustrating another example of a target image including a first pointing object and a second pointing object. For example, as shown in FIG. 6, if a first pointing object 131 is superimposed on the second pointing object 112 or the replacement object 114, and the target image 101 includes the second pointing object 132 and the replacement object 133, the replacement process described above is performed based on the second pointing object 132 and the replacement object 133, without using the second pointing object 112 and the replacement object 114 superimposed on the first pointing object 131.

[0037] Furthermore, the output processing unit 26 outputs the target image after the above-described processing. For example, the output processing unit 26 transmits image data of the target image to the image forming device 2, causes the image forming device 2 to print the target image, or stores the target image in a predetermined storage device (such as the image forming device 2 or another server).

[0038] Next, a description will be given of the operation of the image processing system according to Embodiment 1. Fig. 7 is a flowchart illustrating the operation of the image processing system shown in Fig. 1.

[0039] The target image acquisition unit 21 of the processing server 1 acquires a target image (step S1). Here, the above-mentioned first and second instruction objects and replacement object 114 are additionally written by hand on a document printed by the image forming device 2, and then the image of the document is read to obtain a document image, and the document image (image data) is sent from the image forming device 2 to the processing server 1 as the target image.

[0040] When the target image is acquired in the processing server 1, the pointing object detection unit 22 detects the first pointing object 111 and the second pointing object 112 in the target image 101 by object detection (step S2). In addition, the text area detection unit 23 detects the text object 121 in the target image 101.

[0041] Based on the object detection results and the character object detection results, the instruction identification unit 24 identifies the processing target object 113 designated by the detected first instruction object 111 and the substitute object 114 designated by the detected second instruction object 112 (step S3). Here, the correspondence between the first instruction object 111 and the processing target object 113, the correspondence between the second instruction object 112 and the substitute object 114, and the correspondence between the first instruction object 111 and the second instruction object 112 are identified.

[0042] Then, based on the correspondence identified by the instruction identification unit 24, the replacement processing unit 25 performs a replacement process on the identified processing target object 113 in the target image 101 with the identified replacement object 114 described above (step S4).

[0043] When the replacement process is completed, the output processing unit 26 displays the target image after the replacement process (step S5).

[0044] As described above, according to the first embodiment, the pointing object detection unit 22 detects, by object detection using a machine-learned learner, a first pointing object 111 that specifies a target object and a second pointing object 112 that specifies a replacement object to be used in a replacement process for the target object, among handwritten character objects that have been additionally written on a document in the target image 101. The pointing identification unit 24 identifies the target object 113 specified by the detected first pointing object 111, and also identifies the replacement object 114 specified by the detected second pointing object 112. The replacement processing unit 25 performs a replacement process on the specified target object 113 in the target image 101 based on the specified replacement object 114.

[0045] In particular, in the first embodiment, the processing target object 113 is replaced with an image object (raster image) of the replacement object 114.

[0046] This allows object detection to properly detect the instruction objects 111 and 112 that specify the replacement processing targets in the target image 101 without any particular limitation on the writing implements used to write the first instruction object 111 and the second object 112 on the manuscript.

[0047] Embodiment 2

[0048] Fig. 8 is a diagram showing another example of a target image including a first instruction object and a second instruction object. Fig. 9 is a diagram illustrating instruction objects identified in the target image shown in Fig. 8. Fig. 10 is a diagram showing an example of a target image after a replacement process is performed on the target image shown in Fig. 8.

[0049] 8, the pointing object detection unit 22 detects a first pointing object 211 (bounding box 211a) and a second pointing object 212 (bounding box 212a), and the text area detection unit 23 detects a text object 221. The pointing identification unit 24 identifies a processing target object 213 designated by the first pointing object 211 and a replacement object 214 designated by the second pointing object 212.

[0050] In the second embodiment, the replacement processor 25 performs character recognition on the replacement object 214 and converts the replacement object 214 (here, a raster image of "20%") into text data. The replacement processor 25 then erases the target object 213 (as well as the first pointing object 211, the second pointing object 212, and the replacement object 214) from the target image 201, and adds an image object (a raster image obtained by rasterizing the text data) based on the identified text data to the position of the target object 213 (i.e., places the object as the foreground image). In this case, the replacement processor 25 generates the image object to be added (specifically, the characters in the image object) so that it has the same font type, color, and size as the target object 213.

[0051] Furthermore, in the second embodiment, if the font type of the processing target object 213 cannot be identified or if the processing target object 213 is a handwritten character, the replacement processing unit 25 may identify a font type that is closest to the processing target object 213 based on the character shape, and generate an image object corresponding to the replacement object 214 in the identified font type. Alternatively, if the font type of the processing target object 213 cannot be identified or if the processing target object 213 is a handwritten character, the replacement processing unit 25 may generate an image object corresponding to the replacement object 214 in a font type designated by the user (user setting or user operation).

[0052] The other configurations and operations of the image processing system according to the second embodiment are the same as those of the first embodiment, and therefore a description thereof will be omitted.

[0053] Embodiment 3

[0054] In the third embodiment, the user can select the replacement object 114 used in the replacement process from text data and image data.

[0055] In the third embodiment, the replacement processing unit 25 identifies the operation mode designated by the user (pre-stored user settings or user operation).

[0056] If the identified operating mode is the first mode, as in embodiment 1, the replacement processing unit 25 erases the object to be processed in the target image and adds a replacement object (raster image) specified by the detected second instruction object as an image object at the position of the object to be processed.

[0057] On the other hand, if the identified operating mode is the second mode, as in embodiment 2, the replacement processing unit 25 (a) performs character recognition processing on the replacement object specified by the second instruction object and converts the replacement object into text data, and (b) erases the object to be processed in the target image and adds an image object based on the identified text data to the position of the object to be processed.

[0058] The other configurations and operations of the image processing system according to the third embodiment are the same as those of the first or second embodiment, and therefore a description thereof will be omitted.

[0059] It should be noted that various changes and modifications to the above-described embodiments will be apparent to those skilled in the art. Such changes and modifications may be made without departing from the spirit and scope of the subject matter and without diminishing its intended advantages. In other words, it is intended that such changes and modifications be included within the scope of the claims.

[0060] For example, in the first to third embodiments, the processing server 1 may be built into the image forming device 2.

[0061] Furthermore, in the above first to third embodiments, the first instruction object is a strikethrough line, but it may also be an underline, a band-shaped line drawn with a light-transmitting marker such as a highlighter, etc. The writing implement used to add the above instruction object to the manuscript may be a light-transmitting marker, a ballpoint pen, a pencil, etc., and may be a writing implement with an achromatic writing color or a writing implement with a chromatic writing color.

[0062] In addition, in the above embodiments 1 to 3, the character area detection unit 23 may convert character objects in the target image into a series of text data using character recognition processing, perform morphological analysis on the series of text data, and subject the above-mentioned replacement processing to the morpheme portion (all or part) of the text data for the above-mentioned target object that corresponds to the morpheme of the text data for the above-mentioned replacement object.

[0063] Furthermore, in the above-mentioned embodiments 1 to 3, when erasing the first and second instruction objects, the processing target object, and the replacement object, the objects are replaced with a background image based on the background image around the object, for example, using existing inpainting technology, so that the area of ​​the object matches the surrounding background image after erasure.

[0064] In addition, in the above-mentioned embodiments 1 to 3, the replacement object may be extracted as a character object detected by the character area detection unit 23, or may be extracted as an object remaining after removing the second pointing object (such as an encircling line) from the foreground image within the bounding box of the second pointing object.

[0065] Furthermore, in the first to third embodiments, the text area detection unit 23 detects text objects. Alternatively, the pointing object detection unit 22 may detect text objects together with pointing objects in the target image. [Industrial Applicability]

[0066] The present invention is applicable to, for example, an image processing system that executes a replacement process desired by a user on an original image. [Explanation of symbols]

[0067] 21 Target image acquisition unit 22 Pointing object detection unit 24 Instruction identification part 25 Replacement processing section

Claims

1. a target image acquisition unit that acquires a document image of a document as a target image; a pointing object detection unit that detects, by object detection using a machine-learned learner, a first pointing object that designates a processing target object and a second pointing object that designates a replacement object to be used in a replacement process for the processing target object, among handwritten character objects that are additionally written on the document in the target image; an instruction specifying unit that specifies the target object specified by the detected first instruction object and specifies the substitute object specified by the detected second instruction object; a replacement processing unit that performs a replacement process on the identified processing target object in the target image based on the identified replacement object; An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein the first pointing object and the second pointing object have different shapes.

3. the first instruction object is a strikethrough; the second pointing object is an encircling line; 3. The image processing system according to claim 2, wherein:

4. 2. The image processing system according to claim 1, wherein the learning device is machine-learned using a plurality of original images including the first pointing object and the second pointing object of a plurality of colors, a plurality of line widths, and a plurality of shapes as training data for the first pointing object and the second pointing object, respectively.

5. The image processing system described in claim 1, characterized in that the replacement processing unit (a) performs character recognition processing on the replacement object specified by the second instruction object to convert the replacement object into text data, and (b) erases the processing target object in the target image and adds an image object based on the identified text data to the position of the processing target object.

6. 2. The image processing system according to claim 1, wherein the replacement processing unit erases the object to be processed in the target image and adds the replacement object specified by the detected second pointing object as an image object at the position of the object to be processed.

7. The replacement processing unit identifies an operation mode designated by a user, When the operation mode is the first mode, the replacement processing unit erases the target object in the target image, and adds, as an image object, the replacement object designated by the detected second pointing object at a position of the target object; When the operation mode is the second mode, the replacement processing unit (a) performs character recognition processing on the replacement object designated by the second instruction object to convert the replacement object into text data, and (b) erases the processing target object in the target image and adds an image object based on the identified text data to the position of the processing target object.

2. The image processing system according to claim 1, wherein:

8. 8. The image processing system according to claim 5, wherein the replacement processing unit generates the image object so that the size of the image object matches the size of the object to be processed.

Images