Image recognition device and image recognition method

Abstract

To provide an image recognition device which is excellent in recognition accuracy of a selection accepting figure included in an input image.SOLUTION: An image recognition device according to one embodiment of the present invention is capable of recognizing a selection accepting figure for accepting item selection. which is included in an input image having at least a part of a document inputted thereto. The image recognition device includes: a vertex candidate output unit which outputs a detection result of vertex candidates of the selection accepting figure on the basis of feature points of an outer shape of the figure included in the input image; and a recognition result output unit which outputs a recognition result of the selection accepting figure on the basis of the detection result of vertex candidates outputted from the vertex candidate output unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image recognition device and an image recognition method. [Background technology]

[0002] 2. Description of the Related Art Image recognition devices that use OCR (Optical Character Reader) technology to recognize images of characters and figures formed on a recording medium are known.

[0003] As the above-mentioned image recognition device, one has been disclosed that, when defining a form image to be recognized, accepts the specification of an area containing items included in the form image and options for the items, and displays a definition that associates the items with the options extracted from the area (see, for example, Patent Document 1). Summary of the Invention [Problem to be solved by the invention]

[0004] An image recognition device is required to have excellent accuracy in recognizing a selection acceptance figure that is included in an input image such as a form image and that accepts a selection for an item.

[0005] An object of the present invention is to provide an image recognition device that has excellent accuracy in recognizing a selection acceptance figure included in an input image. [Means for solving the problem]

[0006] An image recognition device according to one aspect of the present invention is an image recognition device that is included in an input image in which at least a portion of a document is input, and that is capable of recognizing a selection acceptance figure that accepts a selection for an item, the image recognition device comprising: a vertex candidate output unit that outputs a detection result of vertex candidates of the selection acceptance figure based on feature points of the outer shape of the figure included in the input image; a vertex candidate evaluation unit that determines whether or not the vertex candidate is a vertex candidate of the selection receiving figure based on the vertex candidate detection result received from the vertex candidate output unit; The aforementioned Vertex candidate evaluation result a recognition result output unit that outputs a recognition result of the selection receiving figure based on the The vertex candidate evaluation unit determines whether or not the vertex candidates are vertex candidates of the selection receiving figure by determining whether or not lines connecting the plurality of vertex candidates are colored, or by comparing the distance between the vertex candidates with the size of characters included in the input image. do. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image recognition device that has excellent accuracy in recognizing a selection receiving figure included in an input image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of an input image of an image recognition device according to an embodiment; [Figure 2] 1 is a diagram illustrating an example of an output image of an image recognition device according to an embodiment; [Figure 3] 1 is a diagram illustrating an example of the configuration of an image forming apparatus including an image recognition device according to an embodiment. [Figure 4] 1 is a block diagram of an example of the configuration of an image forming apparatus according to an embodiment; [Figure 5] FIG. 2 is a block diagram of an example of the hardware configuration of a controller according to the embodiment. [Figure 6] FIG. 2 is a block diagram illustrating an example of the hardware configuration of an IPU according to the embodiment. [Figure 7] FIG. 2 is a block diagram illustrating an example of the functional configuration of an IPU according to the embodiment. [Figure 8] FIG. 4 is a flowchart of an example of an operation of reading a document by the image forming apparatus according to the embodiment. [Figure 9] FIG. 10 is a flowchart of an example of recognition processing by an IPU according to an embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of an input image according to the embodiment. [Figure 11] FIG. 10 is a diagram illustrating a histogram of an input image according to the embodiment. [Figure 12] FIG. 10 is a diagram illustrating an example of a binarized image according to the embodiment. [Figure 13] FIG. 10 is a diagram showing an example of an outer shape image according to the embodiment; [Figure 14] FIG. 1 is a first diagram illustrating an example of vertex candidate detection processing according to the embodiment. [Figure 15] FIG. 2 is a second diagram illustrating an example of vertex candidate detection processing according to the embodiment. [Figure 16] FIG. 3 is a third diagram illustrating an example of vertex candidate detection processing according to the embodiment. [Figure 17] 10A to 10C are diagrams illustrating an example of a vertex candidate evaluation process according to the embodiment. [Figure 18] FIG. 2 is a diagram showing a first example of an output image according to the embodiment. [Figure 19] FIG. 10 is a diagram showing a second example of an output image according to the embodiment. [Figure 20] 10A to 10C are diagrams illustrating an example of tilt detection processing according to an embodiment. [Figure 21] 10A to 10C are diagrams illustrating an example of color inversion processing according to an embodiment. [Figure 22] 10A to 10C are diagrams illustrating an example of a process for enlarging a black region according to an embodiment. [Figure 23] FIG. 10 is a diagram showing a first example of an output image according to another embodiment. [Figure 24] FIG. 10 is a block diagram illustrating an example of the functional configuration of an IPU according to another embodiment. [Figure 25] FIG. 10 is a diagram showing a second example of an output image according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings. In the drawings, the same components are designated by the same reference numerals, and redundant explanations will be omitted where appropriate.

[0010] The image recognition device according to the embodiment is capable of recognizing a selection acceptance figure that accepts a selection for an item and that is included in an input image that includes at least a part of a document.

[0011] A selection acceptance figure is a figure formed around a target item for selection, into which a mark indicating that the item has been selected is input. For example, a selection acceptance figure is a rectangular figure formed around a target item for selection included in a document, and when selected, a check mark indicating that the item has been selected is added, such as a check box. A target item means an item selected using the selection acceptance figure.

[0012] In the embodiment, a case where the selection receiving figure is a check box will be described as an example.

[0013] 1 and 2 are diagrams illustrating input and output images for an image recognition device according to an embodiment, with FIG. 1 illustrating an input image Im0 and FIG. 2 illustrating an output image Q. In FIG.

[0014] As shown in FIG. 1, the input image Im0 includes an image of a target item 510 and a check box 511 formed adjacent to the target item 510.

[0015] The image recognition device according to the embodiment recognizes check boxes 511 from within an input image Im0, and outputs an output image Q to which a recognition mark 521 indicating the recognized check box 511 has been added, as shown in FIG.

[0016] By recognizing the check box 511 using the image recognition device according to the embodiment, a downstream processing device can accurately recognize whether or not a check mark 522 is added to the check box 511 in the document Pi.

[0017] The image recognition device according to the embodiment includes a vertex candidate output unit that outputs a detection result of vertex candidates for a check box 511 based on feature points of the external shape of a figure included in an input image Im0, and a recognition result output unit that outputs a recognition result of the check box 511 based on the detection result of the vertex candidates output from the vertex candidate output unit. This configuration provides an image recognition device with excellent recognition accuracy for the check box 511 included in the input image Im0. Here, a vertex refers to a corner of the figure in the check box 511.

[0018] An image forming apparatus having an image recognition apparatus according to an embodiment will be described below as an example. This image forming apparatus is a multi-function peripheral (MFP) having a function of forming an image on a recording medium such as paper, and a function of reading an original document such as paper on which an image has been formed. The image recognition apparatus according to the embodiment recognizes a check box 511 from an input image Im0, which is input with at least a portion of an image of an original document read by the image forming apparatus.

[0019] <Configuration example of image forming apparatus 1> (Overall configuration example) 3 is a diagram showing an example of the overall configuration of the image forming apparatus 1. As shown in FIG.

[0020] The paper feed section 400 has paper feed cassettes 421 and 422 that store paper of different sizes, and a paper feed means 423 consisting of various rollers that transport the paper stored in the paper feed cassettes 421 and 422 to an image formation position by the plotter 300.

[0021] Plotter 300 includes an exposure device 331, a photosensitive drum 332, a developing device 333, a transfer belt 334, and a fixing device 335. Plotter 300 exposes photosensitive drum 332 with exposure device 331 to form a latent image on photosensitive drum 332 based on image data of an original scanned by scanner 100 or image data input from an external device such as a PC via a network I / F. Plotter 300 develops the image by supplying different color toners to photosensitive drum 332 with developing device 333. Plotter 300 transfers the toner image developed on photosensitive drum 332 with transfer belt 334 to paper supplied from paper feed unit 400, and then fixes the color image to the paper by melting the toner constituting the toner image transferred to the paper with fixing device 335.

[0022] Scanner 100 has an ADF (Auto Document Feeder) 41, a scanner unit 42, and a paper output tray 43. Scanner 100 drives ADF 41 to transport a document placed on ADF 41 to scanner unit 42. Scanner 100 drives scanner unit 42 to capture an image of the document transported from ADF 41. If a document is not placed on ADF 41 but is placed directly on scanner unit 42, scanner unit 42 captures an image of the placed document. In other words, scanner unit 42 operates as a document imaging unit.

[0023] 3 is, for example, a differential mirror drive type scanner unit 42 with an integrated optical sensor drive system. The scanner unit 42 includes a first mirror unit 204, a second mirror unit 210, a lens 216, and a first sensor board 215.

[0024] The first mirror unit 204 is an optical unit having a mirror and a light source including a light emitting unit such as an LED (Light Emitting Diode). The first mirror unit 204 illuminates the placed document with light from the light source and reflects the light reflected by the document toward the second mirror unit 210.

[0025] The second mirror unit 210 reflects the light from the first mirror unit 204 toward the lens 216. The lens 216 causes the light from the second mirror unit 210 to form an approximate image on a photoelectric conversion element 214 provided on the first sensor board 215. The photoelectric conversion element 214 photoelectrically converts the approximate image of the original document into an analog image signal, thereby reading the original document.

[0026] 4 is a block diagram illustrating an example of the configuration of the image forming apparatus 1. The image forming apparatus 1 includes a scanner 100, a controller 200, a plotter 300, an image memory 401, and an IPU (Image Processing Unit) 500.

[0027] The image forming apparatus 1 performs image recognition processing on an input image Im0, which is at least a portion of an original document Pi read by the scanner 100, using the IPU 500. The IPU 500 may perform image processing other than image recognition processing. The image forming apparatus 1 forms an image on paper using the plotter 300 based on the image data after image processing by the IPU 500 and image data of a test chart (test pattern), and outputs the paper on which the image has been formed as a printout Po.

[0028] The controller 200 controls the entire image forming apparatus 1.

[0029] The image memory 401 includes a volatile memory, a hard disk, etc. The image memory 401 performs temporary storage for transmitting and receiving image data between the scanner 100, the controller 200, the plotter 300, and the IPU 500, and permanent storage for later use.

[0030] The IPU 500 is an example of an image recognition device that can recognize a check box 511 that is included in the input image Im0 and that accepts a selection for an item.

[0031] The image forming apparatus 1 can be connected to a PC 2, a server 3, etc. via a network NW and can send and receive image data and the like.

[0032] <Example of hardware configuration of controller 200> Fig. 5 is a block diagram showing an example of the hardware configuration of the controller 200. As shown in Fig. 5, the controller 200 has a CPU (Central Processing Unit) 10, a RAM (Random Access Memory) 11, a ROM (Read Only Memory) 12, a HDD (Hard Disk Drive) 13, and a communication I / F (Interface) 14. These are electrically connected to each other via a bus 15. The communication I / F 14 is also connected to a display device 16 and an input operation unit 17.

[0033] The CPU 10 is an arithmetic unit that controls the operation of the entire image forming apparatus 1. The RAM 11 is a volatile storage medium that enables high-speed reading and writing of information. The CPU 10 uses the RAM 11 as a work area when processing information. The ROM 12 is a non-volatile read-only storage medium that stores programs such as firmware.

[0034] The HDD 13 is a non-volatile storage medium that enables reading and writing of information, and stores an OS (Operating System), various control programs, application programs, and the like. The communication I / F 14 is an interface that connects and controls the bus 15 to various hardware and networks.

[0035] The display device 16 is a visual user interface for the user to check the state of the image forming apparatus 1. The input operation unit 17 is a user interface for the user to input information to the image forming apparatus 1, such as a keyboard or a mouse.

[0036] In the above hardware configuration, the image forming apparatus 1 reads a program stored in a recording medium such as the ROM 12, the HDD 13, or an optical disk into the RAM 11, and operates according to the control of the CPU 10, thereby constituting a software control unit. The image forming apparatus 1 realizes the functions of the image forming apparatus 1 by combining the software control unit configured in this way with the hardware.

[0037] <Hardware Configuration Example of IPU500> FIG. 6 is a block diagram showing an example of the hardware configuration of the IPU 500. As shown in FIG. 6, the IPU 500 includes a CPU 20, a RAM 21, a ROM 22, an HDD 23, and a communication I / F 24. These are electrically connected to each other via a bus 25.

[0038] The CPU 20 is an arithmetic unit that controls the operation of the entire IPU 500. The RAM 21 is a volatile storage medium that enables high-speed reading and writing of information. The CPU 20 uses the RAM 21 as a work area when processing information. The ROM 22 is a non-volatile read-only storage medium that stores programs such as firmware.

[0039] The HDD 23 is a non-volatile storage medium that enables reading and writing of information, and stores an OS, various control programs, application programs, etc. The communication I / F 24 is an interface that connects and controls the bus 25 to various hardware, networks, etc.

[0040] In the above hardware configuration, the IPU 500 reads a program stored in a recording medium such as the ROM 22, HDD 23, or optical disk into the RAM 21 and operates according to the control of the CPU 20, thereby constituting a software control unit. The IPU 500 constitutes functional blocks that realize the functions of the IPU 50 using the software control unit configured in this way.

[0041] <Functional Configuration Example of IPU500> FIG. 7 is a block diagram showing an example of the functional configuration of the IPU 500. As shown in FIG. 7, the IPU 500 includes an input unit 31, a binarization processing unit 32, an outer shape extraction unit 33, a feature point extraction unit 34, a vertex candidate output unit 35, a vertex candidate evaluation unit 36, a registration unit 37, and a recognition result output unit 38. The IPU 500 realizes these functions by the CPU 20 reading a program stored in the ROM 22 and executing it using the RAM 21 as a work area.

[0042] The input unit 31acquires an input image Im0 in which at least a part of the read image of the document Pi by the scanner 100 is input, from the scanner 100. The input unit 31 also acquires information such as the size, resolution, color, or background of the input image Im0. The input unit 31 outputs the acquired input image Im0 and information such as the size, resolution, color, or background of the input image Im0 to the binarization processing unit 32.

[0043] The binarization processing unit 32 performs binarization processing on the input image Im0 using a binarization threshold based on a histogram generated from the pixel values ​​of each of the multiple pixels contained in the input image Im0, and outputs the processed result, the binarized image Im1, to the outline shape extraction unit 33.

[0044] The binarization threshold is not limited to being based on a histogram, and may be determined based on color information of the input image Im0, etc. However, determining the binarization threshold based on a histogram is more preferable because it reduces the influence of images other than figures in the input image Im0. Furthermore, in order to perform the binarization process at high speed, as preprocessing for the binarization process, if the input image Im0 is a color image, monochrome image conversion or conversion to reduce the image resolution may be performed.

[0045] The outer shape extraction unit 33 extracts the outer shape of a figure included in a binarized image Im1 based on the input image Im0, and outputs the extracted outer shape image Im2 to the feature point extraction unit 34. For example, the outer shape extraction unit 33 detects colored pixels, such as pixels colored black, from among the multiple pixels included in the input image Im0, and extracts the outer shape of a figure formed by the colored pixels.

[0046] The feature point extraction unit 34 extracts feature points in the outer shape of the figure extracted by the outer shape extraction unit 33, and outputs the extraction results to the vertex candidate output unit 35. Feature points refer to characteristic parts of the figure extracted by the outer shape extraction unit 33. Characteristic parts can be defined in advance depending on the figure to be recognized. In this embodiment, for example, corners in the outer shape are extracted as feature points. A corner refers to a part where two straight lines intersect. In this embodiment, the extraction result is position information of the feature points within the input image Im0. For example, it is preferable to perform feature point extraction processing by linear approximation of the outer shape, as this simplifies the processing and shortens the processing time.

[0047] The vertex candidate output unit 35 outputs the detection result of the vertex candidates of the check box 511 based on the feature points extracted by the feature point extraction unit 34. A vertex candidate means the position of a part that may be the vertex of the check box 511. The detection result of the vertex candidate is, for example, the position of the vertex candidate in the input image Im0. In this embodiment, the vertex candidate output unit 35 outputs a vertex candidate image Im3 including vertex candidate information to the vertex candidate evaluation unit 36 ​​as the detection result of the vertex candidate.

[0048] In this embodiment, the vertex candidate output unit 35 outputs vertex candidates for the check box 511 based on feature points extracted from the binarized image Im1 obtained by binarizing the input image Im0 using a binarization threshold based on a histogram generated from the pixel values ​​of each of the multiple pixels included in the input image Im0.

[0049] Furthermore, in this embodiment, the vertex candidate output unit 35 outputs the detection result of the vertex candidates of the check box 511 based on the number of pixels, among the pixels included in the image area connecting the multiple feature points extracted by the feature point extraction unit 34, whose pixel values ​​match the pixel values ​​of the pixels included in the feature points.

[0050] The vertex candidate output unit 35 may be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0 that are distinguished into handwritten areas and non-handwritten areas in the input image Im0 based on at least one of color component information and histogram information of the input image Im0. Here, the handwritten area refers to an area that corresponds to the handwritten area included in the input image Im0 when an original Pi including an area handwritten by a user of the image forming apparatus 1 is read by the scanner 100.

[0051] The handwritten region is based on a pattern or mark handwritten by a user on the document Pi, and therefore the image density is not constant compared to an image formed on the document Pi by an image forming device, etc. Therefore, the vertex candidate output unit 35 can distinguish, for example, a region in which the fluctuation range of the density of the image included in the input image Im0 is greater than a predetermined fluctuation range threshold as a handwritten region, and a region equal to or less than the fluctuation range threshold as a non-handwritten region.

[0052] Furthermore, the vertex candidate output unit 35 may be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0, which are detected from a ruled line area included in the input image Im0 and in which the tilt of the input image Im0 has been corrected. Here, the ruled line area refers to an area that corresponds to the ruled line included in the input image Im0 when the original Pi including the ruled line is read by the scanner 100.

[0053] The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0 obtained by color-inverting the pixel values ​​of each of the multiple pixels included in the input image Im0. Color inversion refers to converting a predetermined color into a color that is complementary to the color. For example, color inversion is black-and-white inversion, which converts white image areas in a black-and-white image into black and vice versa.

[0054] Furthermore, the vertex candidate output unit 35 may be configured to output vertex candidates for the check box 511 based on feature points in a color component image obtained by decomposing the pixel values ​​of each of the multiple pixels included in the input image Im0 into color components. Here, a color component image refers to an image obtained by color-separating the color input image Im0, which uses multiple primary colors, into each primary color. For example, if the input image Im0 is a color image with red (R), green (G), and blue (B) as the three primary colors, the color component images correspond to an image consisting of only red, an image consisting of only green, and an image consisting of only blue.

[0055] Furthermore, the vertex candidate output unit 35 may be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0 that are separated into a form region and a non-form region in the input image Im0 based on at least one of the color component information and histogram information of the input image Im0. Here, the form region refers to the region corresponding to the form included in the input image Im0 when an original Pi including the form is read by the scanner 100. A form refers to a document with blank spaces for filling in information, and is a general term for ledgers, slips, etc.

[0056] The vertex candidate output unit 35 may also be configured to output vertex candidates for the check box 511 based on feature points of the input image Im0 obtained by enlarging a black region included in the input image Im0. The black region refers to an image region whose pixel values ​​indicate black.

[0057] The vertex candidate evaluation unit 36 ​​evaluates the vertex candidate information included in the vertex candidate image Im3 received from the vertex candidate output unit 35 to determine whether the vertex candidate is the vertex of the check box 511.

[0058] Because the check box 511 is a rectangular shape, the vertex candidate evaluation unit 36 ​​can determine whether or not a vertex candidate is a vertex of the check box 511 by evaluating whether or not the vertex candidate corresponds to one of the four corner vertices of the rectangular shape. In other words, the vertex candidate evaluation unit 36 ​​can determine whether or not a vertex candidate corresponds to one of the four corner vertices of the rectangular shape based on at least one of the size information and position information of the check box 511.

[0059] The evaluation criteria used by the vertex candidate evaluation unit 36 ​​include, for example, the following: (1) The lines connecting multiple vertex candidates are colored. (2) The vertex candidate forms part of a rectangle or square. (3) The distance between the vertex candidates is an appropriate size compared to the size of the characters contained in the input image Im0.

[0060] When two or three vertex candidates are detected from the input image Im0, the vertex candidate evaluation unit 36 ​​estimates the other vertex candidates out of the four vertex candidates by calculation based on the detected vertex candidates, and evaluates the vertex candidates using this estimation result as the vertex candidate.

[0061] The registration unit 37 receives the evaluation result from the vertex candidate evaluation unit 36, and if the vertex candidate is the vertex of the check box 511, it registers information corresponding to the presence of a vertex and the position information of this vertex. If there is no vertex of the check box 511, the registration unit 37 registers information corresponding to the absence of a vertex. Thereafter, the registration unit 37 outputs the position information of the vertex to the recognition result output unit 38.

[0062] The recognition result output unit 38 outputs the recognition result of the check box 511 received from the registration unit 37. In other words, the recognition result output unit 38 outputs the recognition result of the check box 511 based on the vertex candidate detection result output from the vertex candidate output unit 35. In this embodiment, the recognition result output unit 38 outputs information on the presence or absence of a vertex, and, if a vertex is present, position information of the check box 511, as the recognition result of the check box 511. Because the check box 511 has a rectangular outer shape, the recognition result of the check box 511 includes the position of at least one vertex of the rectangle in the check box 511.

[0063] The recognition result output unit 38 outputs, for example, the vertex position information of the check box 511 to the controller 200, which is an external device of the IPU 500. However, the external device is not limited to the controller 200, and may be another external device such as the plotter 300 or the display device 16.

[0064] <Example of document reading operation by image forming apparatus 1> 8 is a flowchart showing an example of an original reading operation by the image forming apparatus 1. The image forming apparatus 1 starts the operation of FIG.

[0065] First, in step S81, the image forming apparatus 1 causes the scanner 100 to read the document Pi placed on the ADF 41 or the scanner unit .

[0066] Next, in step S82, the image forming apparatus 1 causes the controller 200 to count the number of originals Pi that have been read by the scanner 100.

[0067] Next, in step S83, the image forming apparatus 1 determines whether or not the scanner 100 has read all of the documents Pi using the controller 200. For example, the controller 200 can determine whether or not all of the documents Pi have been read by determining whether or not documents Pi are placed on the ADF 41 or the scanner unit 42, or whether or not the number of read documents Pi has reached the specified number of documents Pi to be read, which was input using the input operation unit 17.

[0068] If it is determined in step S83 that not all of the documents Pi have been read (step S83, No), the image forming apparatus 1 repeats the operations from step S81 onwards. On the other hand, if it is determined that all of the documents Pi have been read (step S83, Yes), in step S84, the image forming apparatus 1 displays information about the documents Pi on the display device 16. This information about the documents Pi is, for example, information such as the number of pages of the documents Pi that have been read or the size of the documents Pi.

[0069] Subsequently, in step S85, the image forming apparatus 1 determines whether or not to recognize the check box 511 on the read document Pi using the controller 200. This determination can be made based on, for example, a user operation using the input operation unit 17.

[0070] In step S85, if it is determined that the checkbox 511 is not recognized (step S85, No), the image forming apparatus 1 ends the operation. On the other hand, if it is determined that the checkbox 511 is recognized (step S85, Yes), in step S86, the image forming apparatus 1 executes the recognition process of the checkbox 511. The details of this recognition process will be described below with reference to FIG. 9. After the image forming apparatus 1 finishes the recognition process of the checkbox 511, it ends the operation.

[0071] As described above, the image forming apparatus 1 can read the document Pi.

[0072] <Example of Recognition Process by IPU500> FIG. 9 is a flowchart showing an example of the recognition process of the checkbox 511 by the IPU500. The IPU500 starts the process of FIG. 9 at the timing of step 86 in the operation of FIG. 8.

[0073] First, in step S91, the IPU500 acquires an input image Im0 from the scanner 100 by inputting it through the input unit 31. The input unit 31 also acquires information such as the size, resolution, color, or background of the input image Im0. The input unit 31 outputs the acquired input image Im0 and the information such as the size, resolution, color, or background of the input image Im0 to the binarization processing unit 32.

[0074] Subsequently, in step S92, the IPU500 executes binarization processing of the input image Im0 using a binarization threshold value based on a histogram generated from the pixel values of each of the plurality of pixels included in the input image Im0 by the binarization processing unit 32, and outputs a binarized image Im1, which is the processing result, to the outer shape extraction unit 33.

[0075] Next, in step S93, the IPU 500 causes the outer shape extraction unit 33 to extract the outer shapes of figures included in the binarized image Im1 based on the input image Im0, and outputs the extracted outer shape image Im2 to the feature point extraction unit 34. If multiple figures are included in the binarized image Im1, the outer shape extraction unit 33 outputs the outer shape image Im2 including the extracted outer shapes of all of the figures to the feature point extraction unit 34.

[0076] Next, in step S94, IPU 500 causes feature point extraction unit 34 to extract feature points in the outer shape of the figure extracted by outer shape extraction unit 33, and outputs the extraction results to vertex candidate output unit 35. If the outer shape of the figure includes multiple feature points, feature point extraction unit 34 outputs the extraction results of all feature points to vertex candidate output unit 35. Furthermore, if the outer shapes of multiple figures are included in outer shape image Im2, feature point extraction unit 34 outputs the extraction results of all feature points in the outer shapes of all figures to vertex candidate output unit 35.

[0077] Next, in step S95, the IPU 500 causes the vertex candidate output unit 35 to output the detection result of the vertex candidates for the check box 511 based on the feature points extracted by the feature point extraction unit 34. For example, the vertex candidate output unit 35 outputs a vertex candidate image Im3 including vertex candidate information as the vertex candidate detection result to the vertex candidate evaluation unit 36. If multiple vertex candidates are included, the vertex candidate output unit 35 outputs a vertex candidate image Im3 including the detection results of all the vertex candidates to the vertex candidate evaluation unit 36.

[0078] Next, in step S96, the IPU 500 determines whether the vertex candidate is a vertex of the check box 511 by using the vertex candidate evaluation unit 36 ​​to evaluate the vertex candidate information included in the vertex candidate image Im3 received from the vertex candidate output unit 35. If there are multiple vertex candidates, the vertex candidate evaluation unit 36 ​​determines whether all of the vertex candidates are vertices of the check box 511.

[0079] Subsequently, in step S97, the IPU 500 receives the evaluation result from the vertex candidate evaluation unit 36 by the registration unit 37, and if the vertex candidate is the vertex of the checkbox 511, the position information of this vertex is registered. The registration unit 37 outputs the position information of all vertices to the recognition result output unit 38.

[0080] Subsequently, in step S98, the IPU 500 outputs the recognition result of the checkbox 511 received from the registration unit 37 by the recognition result output unit 38.

[0081] As described above, the IPU 500 can execute the recognition process of the checkbox 511 included in the input image Im0.

[0082] <Example of recognition processing result by IPU500> Next, an example of the recognition processing result by the IPU 500 will be described.

[0083] (An example of the input image Im0) First, FIG. 10 is a diagram showing an example of the input image Im0. The input image Im0 shown in FIG. 10 shows the peripheral regions of the checkboxes 511a and 511b, which are smaller regions in the input image Im0 shown in FIG. 1, extracted and displayed.

[0084] (An example of the binary image Im1) FIG. 11 is a diagram illustrating the histogram of the input image Im0. In FIG. 11, the horizontal axis represents the pixel values of the pixels included in the input image Im0, and the vertical axis represents the frequency. The binarization processing unit 32 can determine an appropriate binarization threshold for extracting the figures included in the input image Im0 based on a histogram like FIG. 11. The binarization processing unit 32 can obtain a binary image Im1 as shown in FIG. 12 by the binarization process.

[0085] (An example of the outer shape image Im2) FIG. 13 is a diagram showing an example of the outer shape image Im2. As shown in FIG. 13, the outer shape extraction unit 33 extracts the outer shape 512 of the figure included in the input image Im0.

[0086] (An example of vertex candidate 514) 14 to 16 are diagrams for explaining an example of the process of detecting the vertex candidates 514, with FIG. 14 being FIG. 1, FIG. 15 being FIG. 2, and FIG. 16 being FIG. 3.

[0087] As shown in FIG. 14, the feature point extractor 34 extracts feature points 513 based on the outer shape 512 included in the outer shape image Im2.

[0088] 15 shows an area around the check box 511a extracted from the outer shape image Im2. The vertex candidate output unit 35 detects vertex candidates 514a and 514b, which are two of the four corners of the check box 511a.

[0089] FIG. 16 is a diagram showing the check box 511a in FIG. 15 at a larger scale than FIG. 15. As shown in FIG. 16, the positions of vertex candidates 514a and 514b correspond to the outer frame portion of the black image area in the check box 511a. Therefore, the vertex candidate output unit 35 outputs position information of the vertex candidate 514a corrected by referring to an outer position 514a' and an inner position 514a'' of the black image area near the vertex candidate 514a. Furthermore, the vertex candidate output unit 35 outputs position information of the vertex candidate 514b corrected by referring to an outer position 514b' and an inner position 514b'' of the black image area near the vertex candidate 514b.

[0090] FIG. 17 is a diagram for explaining an example of the evaluation process of the vertex candidates 514. In FIG.

[0091] The vertex candidate evaluation unit 36 ​​evaluates the vertex candidates 514a and 514b based on the number of pixels included in the image area connecting the detected vertex candidates 514a and 514b, whose pixel values ​​match pixel values ​​corresponding to black included in the feature point 513. If the number of pixels included in the image area connecting the vertex candidates 514a and 514b whose pixel values ​​correspond to black is greater than a predetermined pixel number threshold, the vertex candidate evaluation unit 36 ​​designates the vertex candidates 514a and 514b as vertex candidates, respectively.

[0092] 17, for example, when a falsely detected vertex candidate 515 is detected, most of the pixels included in the image region connecting the falsely detected vertex candidate 515 and the vertex candidate 514a have pixel values ​​corresponding to white, and the number of pixels corresponding to black is equal to or less than a predetermined pixel number threshold. Therefore, the vertex candidate evaluation unit 36 ​​evaluates the falsely detected vertex candidate 515 as being inappropriate as a vertex candidate, and can remove the falsely detected vertex candidate 515 from the vertex candidates.

[0093] (Example of output image) Figures 18 and 19 are diagrams showing output images, where Figure 18 shows an output image Q according to a first example, and Figure 19 shows an output image Q' according to a second example.

[0094] 18, the output image Q includes a recognition mark 521 indicating the check box 511. On the other hand, in FIG. 19, the output image Q′ includes, in addition to the recognition mark 521, an item mark 523 indicating the target item 510.

[0095] (Tilt detection processing example) Fig. 20 is a diagram illustrating an example of the inclination detection process performed by the vertex candidate output unit 35. Fig. 20 shows a binarized image Im1 obtained by binarizing the input image Im0.

[0096] Here, if the scanner 100 reads the document Pi in a tilted state, the input image Im0 and the binarized image Im1 will be tilted, and the IPU 500 may not be able to detect the vertex candidates 514 of the check box 511.

[0097] 20, ruled line region 516 corresponding to the ruled lines included in document Pi is provided along the outline of document Pi. Therefore, vertex candidate output unit 35 corrects the tilt of input image Im0 and binarized image Im1 based on the tilt of ruled line region 516, and detects vertex candidates 514 from the corrected binarized image Im1. This enables IPU 500 to ensure high accuracy in detecting vertex candidates 514 of check box 511.

[0098] (Example of color inversion processing) Fig. 21 is a diagram illustrating an example of color inversion processing, showing a color-inverted image Im1' obtained by inverting black and white from a binarized image Im1 obtained by binarizing an input image Im0.

[0099] The vertex candidate output unit 35 can detect the vertex candidates 514 using the color-inverted image Im1' even when the document Pi has a color-inverted image formed thereon or when the background of the image is colored. This allows the IPU 500 to ensure high accuracy in detecting the vertex candidates 514 of the check box 511.

[0100] (Example of enlarging the black area) FIG. 22 is a diagram illustrating an example of the enlargement process of a black region. The binarized image Im1 is an image obtained by binarizing the input image Im0, and shows the state before the enlargement process of the black region is performed. The enlarged black region image Im1'' is an image in which the area of ​​the black region is increased by thickening the frame of the frame figure of the check box 511 in the binarized image Im1.

[0101] For example, if the document Pi has a check box 511 that is partially missing due to degradation of the image formed on the document Pi, the accuracy of recognizing the check box 511 may be reduced. By performing a process to enlarge the black area, the missing part of the check box 511 can be filled in, allowing the IPU 500 to ensure high accuracy in detecting the vertex candidates 514 of the check box 511.

[0102] <Effects of IPU500> Next, the effects of IPU500 will be described.

[0103] In conventional software using OCR technology, if check boxes are not defined individually, the OCR technology may not be able to correctly recognize the check boxes. Also, when recognizing a form by OCR technology, simply defining the frame containing the check box may not correctly identify the check box or may reduce the recognition accuracy. Therefore, in order to improve the recognition accuracy of check boxes, it was necessary for the user to individually define multiple check boxes formed within the ruled lines included in the form manually, which was time-consuming.

[0104] The IPU500 according to the embodiment is an image recognition device that can recognize a check box 511 (selection reception graphic) that receives a selection for an item and is included in an input image Im0 in which at least a part of the manuscript Pi is input. The IPU500 also includes a vertex candidate output unit 35 that outputs a detection result of vertex candidates 514 of the check box based on feature points 513 of the outer shape of the graphic included in the input image Im0, and a recognition result output unit 38 that outputs a recognition result of the check box 511 based on the detection result of the vertex candidates 514 output from the vertex candidate output unit 35. For example, the check box 511 has a rectangular outer shape, and the recognition result of the check box 511 includes at least one vertex position of the rectangle in the check box 511.

[0105] In the embodiment, since the check box 511 is recognized based on the vertex candidates 514 based on the feature points 513 of the outer shape, the recognition accuracy of the check box 511 can be improved. Thereby, in the embodiment, an IPU500 with excellent recognition accuracy of the check box 511 included in the input image Im0 can be provided.

[0106] In this embodiment, the rectangular frame-shaped checkbox 511 is used as an example of the selection receiving figure, but the figure is not limited to this. As long as the figure has vertices, it may be a polygon other than a rectangle, such as a triangle or a hexagon, and the figure does not have to be a frame-shaped figure.

[0107] Furthermore, in this embodiment, the vertex candidate output unit 35 outputs the detection result of the vertex candidates 514 of the check box 511 based on the number of pixels, of which pixel values ​​match the pixel values ​​of the pixels included in the feature points 513, among the multiple pixels included in the image region connecting the multiple feature points 513. As a result, in this embodiment, it is possible to remove erroneously detected vertex candidates 514 and detect the vertex candidates 514 of the check box 511, and therefore it is possible to provide an IPU 500 with excellent recognition accuracy for the check box 511.

[0108] Furthermore, in this embodiment, the vertex candidate output unit 35 outputs the vertex candidates 514 of the check box 511 based on feature points 513 extracted from the binarized image Im1 obtained by binarizing the input image Im0 using a binarization threshold based on a histogram generated from the pixel values ​​of each of the multiple pixels included in the input image Im0. This makes it possible to detect the vertex candidates 514 of the check box 511 even in a document Pi on which a handwritten pattern has been written or a document Pi on which a noise pattern has been overwritten on the check box 511, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0109] The vertex candidate output unit 35 may be configured to output the vertex candidates 514 of the check box 511 based on feature points 513 of the input image Im0 that are differentiated into handwritten areas and non-handwritten areas in the input image Im0, based on at least one of the color component information and histogram information of the input image Im0. This makes it possible to detect the vertex candidates 514 of the check box 511 even in a document Pi on which a handwritten pattern has been entered, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0110] Furthermore, the recognition result output unit 38 may be configured to output the recognition result of the check box 511 based on at least one of size information and position information of the figure of the check box 511. This makes it possible to provide an output image Q including target items 510 that accept selections using the check box 511 to a device that performs processing in a process downstream of the IPU 500.

[0111] Furthermore, the vertex candidate output unit 35 may be configured to output the vertex candidates 514 of the check box 511 based on the feature points 513 of the input image Im0, which are detected from the ruled line region 516 included in the input image Im0 and in which the tilt of the input image Im0 has been corrected. This makes it possible to detect the vertex candidates 514 of the check box 511 even in the input image Im0 obtained by reading a tilted document Pi, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0112] Furthermore, the vertex candidate output unit 35 may be configured to output the vertex candidates 514 of the check box 511 based on the feature points 513 of the input image Im0 in which the pixel values ​​of each of the multiple pixels included in the input image Im0 are inverted. This makes it possible to detect the vertex candidates 514 of the check box 511 even in a document Pi on which a color-inverted image is formed or a document Pi with a colored background, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0113] Furthermore, the vertex candidate output unit 35 may be configured to output the vertex candidates 514 of the check box 511 based on feature points 513 in a color component image obtained by decomposing the pixel values ​​of each of the multiple pixels included in the input image Im0 into color components. This makes it possible to detect the vertex candidates 514 of the check box 511 even in a color document Pi or a document Pi on which a handwritten pattern has been written, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0114] Furthermore, the vertex candidate output unit 35 may be configured to output vertex candidates 514 of the check box 511 based on feature points 513 of the input image Im0 that distinguish between form regions and non-form regions in the input image Im0 based on at least one of color component information and histogram information of the input image Im0. This makes it possible to detect the vertex candidates 514 of the check box 511 even in a document Pi on which a handwritten pattern has been written or a document Pi on which a noise pattern has been overwritten on the check box 511, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0115] Furthermore, the vertex candidate output unit 35 may be configured to output the vertex candidates 514 of the check box 511 based on the feature points 513 of the input image Im0 in which a black region included in the input image Im0 is enlarged. This makes it possible to detect the vertex candidates 514 of the check box 511 even in an original document Pi in which part of the check box 511 is missing due to deterioration of the image formed on the original document Pi, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0116] [Other Preferred Embodiments] Other embodiments will be described below. Note that the same components as those in the above-described embodiment are denoted by the same reference numerals, and redundant description will be omitted as appropriate.

[0117] 23 is a diagram showing an output image Qa according to another embodiment. The output image Qa includes a peripheral area 524 of a recognition mark 521 indicating a check box 511, which is determined based on the detection result of a vertex candidate 514 output from the vertex candidate output unit 35. By outputting such an output image Qa, the recognition result output unit 38 can recognize the check box 511 even when a handwritten mark does not fit within the check box 511, thereby providing an IPU 500 with excellent recognition accuracy for the check box 511.

[0118] FIG. 24 is a block diagram showing an example of the functional configuration of the IPU500b according to other embodiments. As shown in FIG. 24, the IPU500b has a misrecognition evaluation unit 39.

[0119] Based on the recognition result of the check box 511 output from the recognition result output unit 38, the misrecognition evaluation unit 39 outputs a detection result of at least one of characters and figures other than the check box 511.

[0120] FIG. 25 is a diagram showing an output image Qb according to other embodiments. The output image Qb' in FIG. 25 includes a misrecognition check box 525 in which the character "Dan" other than the check box 511 is misrecognized as a check box.

[0121] On the other hand, the output image Qb in FIG. 25 includes a misrecognition mark 526 indicating that the misrecognition check box 525 is a misrecognition. This misrecognition mark 526 is detected based on the recognition result of the check box 511 output from the recognition result output unit 38 by the misrecognition evaluation unit 39, and is a mark given by the misrecognition evaluation unit 39. In this way, the IPU500b can prevent misrecognition by characters and figures other than the check box 511 and improve the recognition accuracy of the check box 511.

[0122] Although the preferred embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and various modifications and substitutions can be made to the above-described embodiments without departing from the scope described in the claims.

[0123] In the above-described embodiments, an electrophotographic image forming apparatus using toner has been exemplified, but the present invention is not limited thereto, and the embodiments are also applicable to a liquid ejection type image forming apparatus using a liquid such as ink. Since uneven adhesion, scattering, bleeding, etc. to the recording medium also occur in ink as in the case of toner, the same effects as those in the case of using toner can be obtained.

[0124] The embodiments also include an image recognition method. For example, the image recognition method is an image recognition method using an image recognition device that recognizes a selection acceptance figure included in an input image containing at least a portion of a document and that accepts selections for items, wherein the image recognition device outputs, via a vertex candidate output unit, a detection result of vertex candidates of the selection acceptance figure based on feature points of the outer shape of the figure included in the input image, and outputs, via a recognition result output unit, a recognition result of the selection acceptance figure based on the detection result of the vertex candidates output from the vertex candidate output unit. Such an image recognition method can achieve the same effects as the IPU 500 described above.

[0125] The ordinal numbers, quantities, and other figures used in the description of the embodiments are all provided as examples to specifically explain the technology of the present invention, and the present invention is not limited to the illustrated figures. Furthermore, the connection relationships between the components are provided as examples to specifically explain the technology of the present invention, and do not limit the connection relationships that realize the functions of the present invention.

[0126] Each function of the embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to perform each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to perform each function described above. [Explanation of symbols]

[0127] 1. Image forming device 2 PC 3 Server 10, 20 CPUs 11, 21 RAM 12, 22 ROM 13, 23 HDD 14, 24 communication I / F 15, 25 bus 16 Display device 17 Input operation section 41 ADF 42 Scanner unit 43 Paper output tray 31 Input section 32 Binarization processing section 33 External shape extraction part 34 Feature point extraction unit 35 Vertex candidate output unit 36 Vertex candidate evaluation unit 37 Registration Department 38 Recognition result output unit 39 Misperception Evaluation Department 100 scanners 200 Controller 204 1st mirror unit 210 Second mirror unit 214 Photoelectric conversion element 215 First Sensor Board 216 Lens 300 plotter 331 Exposure equipment 332 Photosensitive drum 333 Developing device 334 Transfer belt 335 Fixing device 401 Image Memory 421, 422 paper cassettes 423 Paper feeding means 500 IPU (an example of an image recognition device) 510 Target Items 511, 511a, 511b Check boxes (examples of selection acceptance shapes) 512 External shape 513 minutiae 514, 514a, 514b Vertex candidates 515 False positive vertex candidates 516 Border Area 521 Recognition Mark 522 check mark 523 Item Mark 524 Surrounding Area 525 False Recognition Checkbox 526 Misidentified mark Im0 Input image Im1 binarized image Im2 external shape image NW Network Pi manuscript Po printed matter Q Output Image [Prior art documents] [Patent documents]

[0128] [Patent Document 1] Japanese Patent Publication No. 2021-039429

Claims

1. An image recognition device capable of recognizing a selection acceptance figure that accepts a selection for an item, the selection acceptance figure being included in an input image that includes at least a part of a document, a vertex candidate output unit that outputs a detection result of vertex candidates of the selection receiving figure based on feature points of the outer shape of the figure included in the input image; a vertex candidate evaluation unit that determines whether or not the vertex candidate is a vertex candidate of the selection receiving figure based on the vertex candidate detection result received from the vertex candidate output unit; a recognition result output unit that outputs a recognition result of the selection receiving figure based on a determination result by the vertex candidate evaluation unit, The vertex candidate evaluation unit determines whether the vertex candidate is a vertex candidate of the selection receiving figure by determining whether the straight lines connecting the multiple vertex candidates are colored, or by comparing the distance between the vertex candidates with the size of the characters contained in the input image.

2. the selection receiving figure has a rectangular outer shape, The image recognition device according to claim 1 , wherein the recognition result of the selection receiving figure includes the position of at least one vertex of the rectangle in the selection receiving figure.

3. 3. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the detection result of the vertex candidate based on the number of pixels, among a plurality of pixels included in an image region connecting a plurality of the feature points, whose pixel values ​​match pixel values ​​of pixels included in the feature points.

4. 4. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points extracted from a binarized image obtained by binarizing the input image using a binarization threshold based on a histogram generated from pixel values ​​of each of a plurality of pixels included in the input image.

5. 5. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image that are distinguished into handwritten regions and regions other than the handwritten regions based on at least one of color component information and histogram information of the input image.

6. 6. The image recognition device according to claim 1, wherein the recognition result output unit outputs the recognition result of the selection acceptance figure based on at least one of size information and position information of the selection acceptance figure.

7. 7. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image after correcting the tilt of the input image, the feature points being detected from a ruled line region included in the input image.

8. 8. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image obtained by color-inverting pixel values ​​of each of a plurality of pixels included in the input image.

9. 9. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points in a color component image obtained by decomposing pixel values ​​of each of a plurality of pixels included in the input image into color components.

10. 10. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points of the input image that are distinguished into a form region and a region other than the form region based on at least one of color component information and histogram information of the input image.

11. 11. The image recognition device according to claim 1, wherein the vertex candidate output unit outputs the vertex candidates based on the feature points in the input image obtained by enlarging a black region included in the input image.

12. 12. The image recognition device according to claim 1, wherein the recognition result output unit outputs a recognition result of the selection reception figure, the recognition result including a peripheral area of ​​a recognition mark indicating the selection reception figure, determined based on the detection result of the vertex candidate output from the vertex candidate output unit.

13. 13. The image recognition device according to claim 1, further comprising an error recognition evaluation unit that outputs a detection result of at least one of characters and figures other than the selection acceptance figure based on the recognition result of the selection acceptance figure output from the recognition result output unit.

14. An image recognition method using an image recognition device capable of recognizing a selection acceptance figure that accepts a selection for an item, the selection acceptance figure being included in an input image that includes at least a part of a document, the image recognition device comprising: a vertex candidate output unit outputs a detection result of vertex candidates of the selection receiving figure based on feature points of the outer shape of the figure included in the input image; a vertex candidate evaluation unit determines whether or not the vertex candidate is a vertex candidate of the selection receiving figure based on the detection result of the vertex candidate received from the vertex candidate output unit; a recognition result output unit outputs a recognition result of the selection receiving figure based on the vertex candidate detection result output from the vertex candidate output unit; The vertex candidate evaluation unit determines whether the vertex candidate is a vertex candidate of the selection receiving figure by determining whether the straight lines connecting the multiple vertex candidates are colored, or by comparing the distance between the vertex candidates with the size of the characters contained in the input image.

Images