Document verification device and program

The document verification device and program improve accuracy and reduce labor costs by employing pattern and color matching techniques, including grayscale conversion and brightness adjustment, to automate the verification of documents.

JP7947834B2Active Publication Date: 2026-11-11DAI NIPPON PRINTING CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2022138988
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2022-09-01
Publication Date
2026-11-11
Estimated Expiration
2042-09-01

AI Technical Summary

Technical Problem

Existing document verification methods, such as those based on the proportion of colored areas, struggle to accurately distinguish between documents with similar coloring, leading to human error and high labor costs.

Method used

A document verification device and program that utilizes pattern matching and color matching techniques, including grayscale conversion, brightness adjustment, and feature point extraction to compare target and reference document images, ensuring accurate verification through multiple grid-like regions and outputting determination results.

Benefits of technology

Enhances document verification accuracy by comparing images based on feature points, color tones, and shape, reducing human error and labor costs by automating the verification process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007947834000001
    Figure 0007947834000001
  • Figure 0007947834000002
    Figure 0007947834000002
  • Figure 0007947834000003
    Figure 0007947834000003
Patent Text Reader

Abstract

To provide a document checking device and a program that allow an operator to more accurately check a document by using image processing.SOLUTION: A document checking device 1 comprises: a pattern collation unit 14 that acquires a target document image, and collates a pattern of the target document image with a pattern of a master image stored in a master image storage unit 32 based on the feature points of both images; a hue collation unit 16 that divides each of the target document image and the master image into a plurality of grid squares, and compares the average colors of corresponding squares of both images to collate the hues of both images; and a checking result output unit 20 that outputs a determination result related to the degree of sameness between both images based on at least one collation result of the collation of patterns and the collation of hues.SELECTED DRAWING: Figure 2
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a document verification device and a program.

Background Art

[0002] Conventionally, for example, as in the case of DM (direct mail), there has been performed an operation of enclosing documents in envelopes in order to uniformly send the same document to a plurality of customers for the purpose of product awareness and solicitation. Such a series of operations for sending to customers and the like may be performed by a service providing company that provides document enclosing and sealing service. For example, when receiving a commission from a requesting company that has customers, the service providing company creates documents by printing, encloses and seals the created documents, and then performs a series of operations up to the shipping operation. In that case, it is necessary to reliably enclose and seal the commissioned documents. At present, this is performed by visual checks by multiple people, but there is a risk of human error by workers, and it also imposes a work burden on the workers. Furthermore, since the work is performed by multiple people, labor costs have been high. Under such circumstances, for example, a measure to reduce the burden on workers and save labor can be conceived by automatically performing determination using images of documents.

[0003] Here, as a method of discrimination using images of documents, for example, a document discrimination device is disclosed, which comprises: a table relating the proportion occupied by a colored area within a predetermined section to characters; a storage unit that stores a format string defined for each form; an image extraction unit that extracts an image of a predetermined area from an image of a form; a text conversion processing unit that divides the extracted image into predetermined sections, calculates the proportion for each section, extracts characters corresponding to the calculated proportion from the table stored in the storage unit, and generates a string by replacing the images in the sections with the characters; a keyword generation unit that extracts a predetermined number of strings as keywords from the string generated by the text conversion processing unit; and a format discrimination unit that discriminates the form of a document by extracting a format string containing the keyword or a string similar to the keyword from the format strings stored in the storage unit. (See, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2011-243053 [Overview of the Initiative] [Problems that the invention aims to solve]

[0005] The method described in Patent Document 1 is based on the proportion of the colored area used for identification. However, there are different documents with similar coloring, and it was difficult to distinguish between such documents, as well as various other documents.

[0006] Therefore, the present invention aims to provide a document verification device and program that enables more accurate document verification using image processing. [Means for solving the problem]

[0007] The present invention solves the above problem by the following means. The first invention is a document verification device for verifying the degree of similarity between a reference document image, which is an image of a reference document, and comprising: a reference image storage unit for storing the reference document image; a target document image acquisition means for acquiring a target document image, which is a photographed image of the document to be verified; a pattern matching means for comparing the patterns of the target document image acquired by the target document image acquisition means and the reference document image stored in the reference image storage unit based on the feature points of the two images; a color matching means for dividing the target document image acquired by the target document image acquisition means and the reference document image stored in the reference image storage unit into a plurality of grid-like regions and comparing the average color of the corresponding regions of both images to compare the color tones of the two images; and a determination result output means for outputting a determination result regarding the degree of similarity between the two images based on the matching result of at least one of the pattern matching means and the color matching means. The second invention is a document verification device of the first invention, further comprising a color conversion means for converting the target document image to grayscale, and a pattern matching means for extracting feature points from the grayscaled reference document image and the target document image after conversion to grayscale by the color conversion means, and matching the patterns of both images. The third invention is a document verification device of the first or second invention, wherein the pattern matching means extracts a rectangle from the target document image that estimates the shape of the reference document image using the feature points of both images, and matches the patterns of both images based on the shape of the extracted rectangle. The fourth invention is a document verification device according to any of the first to third inventions, wherein the device includes a brightness adjustment means that adjusts the brightness by comparing the pixel values ​​of the target document image acquired by the target document image acquisition means with the reference document image stored in the reference image storage unit, and the color matching means compares the colors of both images after adjustment by the brightness adjustment means. The fifth invention is a document verification device of the fourth invention, wherein the brightness adjustment means divides the target document image acquired by the target document image acquisition means and the reference document image stored in the reference image storage unit into a plurality of grid-like regions, compares the color difference of the corresponding regions of both images, and adjusts the brightness so that the color difference is smallest. The sixth invention is a document verification device of the fifth invention, wherein the size of the area divided by the brightness adjustment means is larger than the area divided by the color matching means. The seventh invention is a document verification device according to any of the first to sixth inventions, wherein the target document image acquisition means acquires the target document image via a shooting unit that photographs the document to be verified, and the judgment result output means outputs the judgment result relating to the equality of both images to a display unit. The eighth invention is a program for causing a computer to function as a document verification device based on any of the first through seventh inventions. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a document verification device and program that enable more accurate document verification using image processing. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an overview of the document verification device according to this embodiment. [Figure 2] This figure shows the functional blocks of the document verification device in the document verification system according to this embodiment. [Figure 3] This figure shows an example of items in the master image storage unit of the document verification device according to this embodiment. [Figure 4] This flowchart shows the master registration process in the document verification device according to this embodiment. [Figure 5] This flowchart shows the document verification process in the document verification device according to this embodiment. [Figure 6] This is a flowchart illustrating pattern matching processing in the document verification apparatus according to the present embodiment. [Figure 7] This is a diagram for explaining the pattern matching processing in the document verification apparatus according to the present embodiment. [Figure 8] This is a flowchart illustrating brightness adjustment processing in the document verification apparatus according to the present embodiment. [Figure 9] This is a diagram for explaining the brightness adjustment processing in the document verification apparatus according to the present embodiment. [Figure 10] This is a diagram for explaining the brightness adjustment processing in the document verification apparatus according to the present embodiment. [Figure 11] This is a flowchart illustrating color tone matching processing in the document verification apparatus according to the present embodiment. [Figure 12] This is a diagram for explaining the color tone matching processing in the document verification apparatus according to the present embodiment. [Figure 13] This is a flowchart illustrating management code verification processing in the document verification apparatus according to the present embodiment. [Figure 14] This is a diagram for explaining the management code verification processing in the document verification apparatus according to the present embodiment. [Mode for Carrying Out the Invention]

[0010] Hereinafter, modes for carrying out the present invention will be described with reference to the drawings. Note that this is merely an example, and the technical scope of the present invention is not limited thereto. (Embodiment) <Document Verification Apparatus 1> Figure 1 is a diagram showing an overview of the document verification apparatus 1 according to the present embodiment. The document verification apparatus 1 is used, for example, in the process of enclosing documents to be sent to customers of a business partner in envelopes, which is performed by a contracted service company that has received a request from the business partner enterprise.

[0011] In the document verification device 1 shown in FIG. 1, a worker P takes one document 51a (document to be verified) from the top of a stack of documents 51 placed in the document placement area on the right side of the worker P, and places the document on a placement table 40 in front of the worker P. Here, the document 51a is, for example, a letter, catalog, pamphlet or the like sent to a customer by a business partner such as a financial institution like a bank or a credit card company, or an insurance company like a life insurance or non-life insurance company. The document 51a may be assigned a management code, which is a unique code for each document for identifying the document.

[0012] The placement table 40 is provided with a placement area 41 where the document 51a is placed, such that the placement area 41 is visible to the worker P. Accordingly, the worker P places the document 51a within the range of the placement area 41. Then, when the worker P operates a switch 8 provided on the placement table 40, a camera 5 (imaging unit) captures an image of the placement table 40. Then, the document is automatically identified from the captured image, and a verification result indicating whether the document is a correct document is displayed on a monitor 4 (display unit). The switch 8 is an operation member for acquiring an image via the camera 5. The switch 8 is provided, for example, on the right front side of the placement table 40 as viewed from the worker P, and is operable with the right hand of the worker P. Note that the switch 8 may be provided at a position where it can be operated with the left hand of the worker P. The camera 5 is pre-adjusted and provided at a position where it can capture an image of the placement table 40.

[0013] When the worker P can confirm from the verification result displayed on the monitor 4 that the document is a correct document, that is, that the document is of the same type as the document to be verified this time, the worker P picks up the document 51a placed on the placement table 40 again, and encloses the document into an empty envelope 55 located in the envelope placement area on the left side of the worker P. Through this series of operations by the worker P, the document 51a is enclosed and sealed in the envelope 55.

[0014] Next, functional blocks of the document verification device 1 in the document verification system 100 will be described. FIG. 2 is a diagram showing the functional blocks of the document verification device 1 in the document verification system 100 according to the present embodiment. Figure 3 shows an example of items in the master image storage unit 32 of the document verification device 1 according to this embodiment. As shown in Figure 2, the document verification system 100 comprises a document verification device 1 and a document management server 190. The document verification device 1 and the document management server 190 are connected to each other via a communication network, such as an internet connection.

[0015] The document verification device 1 comprises a control unit 10, a storage unit 30, a monitor 4, a camera 5, a speaker 6, a switch 8, and a communication interface unit 39. The control unit 10 is a central processing unit (CPU) that controls the entire document verification device 1. The control unit 10 works in cooperation with the aforementioned hardware to perform various functions by appropriately reading and executing the operating system (OS) and application programs stored in the memory unit 30.

[0016] Before explaining the control unit 10, let's explain the storage unit 30. The memory unit 30 is a storage area such as a hard disk or semiconductor memory element for storing programs, data, etc., necessary for the control unit 10 to perform various processes. The storage unit 30 comprises a program storage unit 31 and a master image storage unit 32 (reference image storage unit).

[0017] The program storage unit 31 is a storage area that stores various programs used by the document verification device 1. The program storage unit 31 stores the document verification program 31a. The document verification program 31a is a program that allows the control unit 10 of the document verification device 1 to execute various functions described later. The master image storage unit 32 is a storage area that stores the master image (reference document image), which is the image used for comparison to confirm whether the documents are of the same type.

[0018] Figure 3 shows an example of items in the master image storage unit 32. The master image storage unit 32 shown in Figure 3 stores color images, pixel values, grayscale images, image feature points, location information, management code images (reference string images), code image feature points, etc., using the document ID (IDentification) as the key. The document ID is identification information that identifies the type of document, and is different for each document, such as Document A, Document B, Document C, etc. The color image is a colored version of the document. Pixel values ​​are the pixel values ​​of a color image, and can be represented, for example, by RGB. A grayscale image is an image that has been converted from a color image to grayscale. Image feature points are the feature points of a grayscale image. Location information is a coordinate value indicating the position of a management code (a predetermined string of characters) within a specific area of ​​a document. The coordinate value is the value of the rectangle surrounding the management code. A management code image is an image containing a management code, and it is a grayscale image. The code image feature points are the feature points of the management code image.

[0019] Next, the control unit 10 will be described. As shown in Figure 2, the control unit 10 includes a start reception unit 11, an image acquisition unit 12 (target document image acquisition means), a color conversion unit 13 (color conversion means), a pattern matching unit 14 (pattern matching means, first pattern matching means), a brightness adjustment unit 15 (brightness adjustment means), a color matching unit 16 (color matching means), a region extraction unit 17 (region extraction means), a preprocessing unit 18 (preprocessing means), a region pattern matching unit 19 (second pattern matching means, cutting means), and a confirmation result output unit 20 (judgment result output means).

[0020] The start reception unit 11 accepts the start of processing by receiving an operation on the switch 8 by the worker P. The image acquisition unit 12, in response to the reception by the start reception unit 11, acquires a target document image, which is a captured image including the document 51a on the mounting table 40, via the camera 5. The color conversion unit 13 converts the target document image acquired by the image acquisition unit 12 into grayscale. Alternatively, the color conversion unit 13 may convert a color image stored in the master image storage unit 32 into grayscale and store it in the master image storage unit 32.

[0021] The image matching unit 14 extracts feature points from the target document image converted to grayscale by the color conversion unit 13. Next, the image matching unit 14 uses the image feature points stored in the master image storage unit 32 and the feature points of the target document image to extract a rectangle that estimates the shape of the master image from the target document image. The image matching unit 14 can perform the extraction of feature points and rectangles using, for example, known open-source libraries for image processing and image analysis. If the feature points of both images are similar, the image matching unit 14 extracts a rectangular rectangle that represents the shape of the entire target document contained in the target document image. On the other hand, if the feature points of both images are not similar, the image matching unit 14 extracts a distorted rectangle that differs from the shape of the entire target document.

[0022] The pattern matching unit 14 then matches the pattern of the target document image with that of the master image based on the extracted rectangular shape. For example, the pattern matching unit 14 determines that the pattern has been matched if the corners of the rectangle are close to 90 degrees or if the sides of the rectangle are longer than a certain length. The pattern matching unit 14 also matches the pattern of the target document image with that of the master image based on the number of feature points of the extracted target document image and the number of image feature points of the master image. For example, based on the number of matching feature points, the pattern matching unit 14 determines that the pattern has been matched for the entire document if the ratio to the total number of feature points is above a threshold.

[0023] The brightness adjustment unit 15 compares the pixel values ​​of the target document image with the color image stored in the master image storage unit 32, and adjusts the brightness of the darker image to be closer to the brightness of the brighter image. The pixel values ​​may be represented by, for example, RGB. More specifically, the brightness adjustment unit 15 divides the image into multiple grid-like squares (regions), compares the color difference between squares at the same position, and adjusts the brightness to minimize the color difference. Here, the number of grid squares M may be fine, but it may also be a somewhat larger grid depending on the processing power of the computer.

[0024] The color matching unit 16 divides the image, after the brightness has been adjusted by the brightness adjustment unit 15, into multiple grid-like cells and compares the average color of cells at the same position to match the color tones of both images. More specifically, the color matching unit 16 calculates N average colors for each cell and calculates the color difference between cells at the same position. The color matching unit 16 then calculates the percentage of cells whose calculated color difference exceeds a certain threshold, and determines that color matching has been successful if the calculated percentage is below the threshold. The number of grid-like cells N processed by the color matching unit 16 is greater than the number of grid-like cells M processed by the brightness adjustment unit 15. In other words, the division into multiple cells by the color matching unit 16 is finer than the division into multiple cells by the brightness adjustment unit 15.

[0025] The region extraction unit 17 extracts a matching region image from the target document image. More specifically, the region extraction unit 17 extracts a matching region image from the target document image based on the position information of the master image storage unit 32. The matching region image is an image containing a management code located in a portion of the document. The preprocessing unit 18 performs preprocessing on the matching region image extracted by the region extraction unit 17. The preprocessing unit 18 can perform processing using various preprocessing methods. Examples of preprocessing methods include contrast adjustment, sharpening, brightness conversion, and thinning. Contrast adjustment adjusts the contrast to be as high as possible so that the standard deviation when the pixel values ​​are distributed is as large as possible. Sharpening means emphasizing the edges of the image, which can produce a sharp image. Brightness conversion means performing gamma correction, which can emphasize areas with low brightness. Thinning is a process in which, in a binarized image, lines are thinned so that only the center pixel of the line remains, and the center line is calculated.

[0026] The region pattern matching unit 19 extracts feature points from the matching region image after preprocessing by the preprocessing unit 18. Next, the region pattern matching unit 19 uses the feature points of the code image stored in the master image storage unit 32 and the feature points of the matching region image to extract a rectangle that estimates the shape of the management code image from the matching region image. If the feature points of both images are similar, the region pattern matching unit 19 extracts a rectangular rectangle that encloses the code image included in the matching region image. On the other hand, if the feature points of both images are not similar, the region pattern matching unit 19 extracts a distorted rectangle. The region pattern matching unit 19 then matches the patterns of the matching region image and the management code image based on the extracted rectangular shape. The pattern matching determination by the region pattern matching unit 19 is the same as the processing by the pattern matching unit 14. Furthermore, if a match is found, the region pattern matching unit 19 extracts a rectangular region from the matching region image.

[0027] The confirmation result output unit 20 outputs to the monitor 4 the determination result regarding the degree of similarity between the two images to be compared, that is, the confirmation result of whether the target document image is of the same type as the master image, based on the determination result from the pattern matching unit 14, the determination result from the color matching unit 16, and the determination result from the area pattern matching unit 19. Alternatively, the control unit 10 may perform OCR (Optical Character Recognition) processing on the rectangle extracted by the area pattern matching unit 19, and the confirmation result output unit 20 may determine whether the string obtained by the OCR processing matches a code registered in the document management server 190. Details of each process will be described later.

[0028] As shown in Figure 1, the monitor 4 is positioned, for example, on the far side of the platform 40 from the perspective of the worker P, and is easily visible to the worker P who is placing documents on the platform 40. The monitor 4 is a display device that functions as a display unit, for example, composed of an LCD panel or the like. Camera 5 is an imaging device. As shown in Figure 1, Camera 5 is mounted on the vertically upper side of the mounting table 40, with a lens (not shown) facing the mounting table 40, and photographs the mounting area 41 of the mounting table 40. Speaker 6 is an audio output device and is located, for example, near (not shown) the monitor 4. Switch 8 is a button operated by worker P.

[0029] The communication interface unit 39 is, for example, an interface for communicating with the document management server 190. Here, "computer" refers to an information processing device equipped with a control unit, memory device, etc., and the document verification device 1 is an information processing device equipped with a control unit 10, a memory unit 30, etc., and is included in the concept of a computer. Image processing is computationally intensive. Therefore, it is desirable that the document verification device 1 be composed of multiple CPUs, and that the control unit 10 perform multiple processes in parallel.

[0030] <Document Management Server 190> The document management server 190 is a server that stores information about various documents that are checked using the document verification device 1. The document management server 190 includes a storage unit (not shown) that stores, for example, the document name, the name of the requesting company, a management code, etc., using the document ID as a key. In addition to the storage unit described above, the document management server 190 also includes a control unit, a communication interface unit, and the like, although these are not shown in the diagram.

[0031] <Explanation of the process> Next, we will explain the processing performed by the document verification device 1. First, we will explain the process of registering information about the source document in the master image storage unit 32, which is performed in advance for document verification. Figure 4 is a flowchart showing the master registration process in the document verification device 1 according to this embodiment.

[0032] First, worker P places the reference document, that is, the document to be used for comparison, on the mounting platform 40. Then, for example, when worker P operates the switch 8, in step S (hereinafter referred to simply as "S") 71 in Figure 4, the control unit 10 receives a signal from the switch 8, takes a picture of the mounting platform 40, and acquires the captured image as the master image. In step S72, the control unit 10 calculates the overall average pixel value of the master image and stores it in the master image storage unit 32 along with the acquired master image.

[0033] In step S73, the control unit 10 (color conversion unit 13) converts the master image to grayscale. In S74, the control unit 10 extracts feature points from the grayscale image, which is the master image converted to grayscale, and stores the extracted feature points together with the grayscale image in the master image storage unit 32. In S75, the control unit 10 receives a specification for the position of the management code area in the master image. The control unit 10 may also receive the position of the area containing the management code as coordinate values ​​via an input device such as a mouse (not shown). In S76, the control unit 10 extracts a management code image containing the management code from the grayscale image based on the received location information. In S77, the control unit 10 extracts feature points using the management code image and stores the extracted feature points along with the management code image and location information in the master image storage unit 32. After that, the control unit 10 terminates this process.

[0034] In the above description, information regarding the master image is registered in the master image storage unit 32 using the document verification device 1. However, only the processing in S71 may be performed using the document verification device 1, and subsequent processing may be performed using, for example, a general personal computer (PC). In that case, the information obtained through processing should be stored in the master image storage unit 32. Also, although the master image was described as being obtained by the camera 5 photographing the mounting table 40, for example, the original document used to create the document may also be used. However, it is preferable to obtain the master image using the device of the document verification device 1 used when verifying the document, as it is better to have the same shooting conditions, such as the amount of light. Furthermore, if the document does not contain a management code, processing from S75 onwards will not be performed.

[0035] Next, we will describe the process of placing the target document on the mounting platform 40 and checking whether or not it is the same type as the master image registered in the master image storage unit 32. Figure 5 is a flowchart showing the document verification process in the document verification device 1 according to this embodiment. First, worker P places the document 51a to be enclosed in the envelope onto the support table 40, as explained in Figure 1. In S11 of Figure 5, the control unit 10 (start reception unit 11) determines whether or not it has received an operation of switch 8. When worker P places a document on the placement area 41 of the placement table 40 and, for example, presses switch 8, the control unit 10 determines that it has received an operation of switch 8. If the operation of switch 8 has been received (S11: YES), the control unit 10 moves the process to S12. On the other hand, if the operation of switch 8 has not been received (S11: NO), the control unit 10 remains in this process until it receives an operation of switch 8.

[0036] In S12, the control unit 10 performs a pattern matching process. In S13, the control unit 10 performs brightness adjustment processing. In S14, the control unit 10 performs a color matching process. In S15, the control unit 10 determines whether or not to check the management code. For example, if the master image storage unit 32 has information on items after the location information, the control unit 10 decides to check the management code. If the management code is checked (S15: YES), the control unit 10 moves the process to S16. On the other hand, if the management code is not checked (S15: NO), the control unit 10 moves the process to S17. In S16, the control unit 10 performs a management code verification process.

[0037] In S17, the control unit 10 outputs the confirmation result to the monitor 4. Figure 1 shows that the monitor 4 displays a "○" mark indicating a match as a confirmation result, overlaid on the image captured by camera 5. If the pattern matching process in S12 or the color matching process in S14 is determined to be unsuccessful and indicates a mismatch between identical items, the process may proceed to S17 without performing the subsequent matching processes. If the verification results indicate a mismatch, for example, an "X" mark will be displayed on the monitor above the image. In this case, the control unit 10 may also output a buzzer sound from the speaker 6.

[0038] In S18, the control unit 10 determines whether or not to terminate the process. For example, if worker P performs an operation to terminate the process (not shown), the control unit 10 determines that the process should be terminated. If the process is to be terminated (S18: YES), the control unit 10 terminates the process. On the other hand, if the process is not to be terminated (S18: NO), the control unit 10 moves the process to S11 and remains in the process until worker P operates switch 8.

[0039] In this way, by repeatedly taking a document 51a from the stack of documents 51 and pressing the switch 8, worker P can confirm whether or not all documents 51a in the stack of documents 51 are the same type as the master image stored in the master image storage unit 32. As a result, the document verification device 1 can take over some of the visual checks performed by multiple workers, thereby reducing the burden on workers.

[0040] Next, we will explain the details of each of the processes described above. First, let's explain the pattern matching process in S12 of Figure 5. Figure 6 is a flowchart showing the image matching process in the document verification device 1 according to this embodiment. Figure 7 is a diagram illustrating the pattern matching process in the document verification device 1 according to this embodiment.

[0041] In S21 of Figure 6, the control unit 10 (image acquisition unit 12) photographs the mounting table 40 and acquires the target document image as the captured image. In S22, the control unit 10 temporarily stores the acquired target document image in the storage unit 30. In S23, the control unit 10 (color conversion unit 13) converts the target document image to grayscale and temporarily stores the converted target document image in the storage unit 30. In S24, the control unit 10 (pattern matching unit 14) extracts feature points from the grayscaled target document image. In S25, the control unit 10 (pattern matching unit 14) uses the grayscale image of the master image and its feature points to estimate the rectangle of the master image from the target document image, and extracts the estimated rectangle.

[0042] Here, we will explain the process using a specific example. Figure 7(A) shows explanatory diagram 60 when similar documents are compared. Diagram 60 visualizes the matching feature points between the grayscale master image 61 and the target document image 62, represented by lines. The bundle of lines 63 connecting the feature points contains numerous lines due to the large number of matching points. Additionally, a rectangle 64 is extracted to enclose the target document image 62.

[0043] On the other hand, Figure 7(B) shows explanatory diagram 65 for the case where different types of documents are compared. Figure 65 visualizes the matching feature points between the grayscale master image 61 and another target document image 66, represented by lines. The number of lines in the bundle of lines 67 connecting the feature points is overwhelmingly smaller than in Figure 7(A) due to the small number of matching points. In addition, an irregular rectangle 68 is extracted at a different location from the target document image 66.

[0044] In step S26 of Figure 6, the control unit 10 (pattern matching unit 14) checks whether the pattern is the same as the master image based on the number of matching feature points and the shape of the rectangle. In the case of explanatory diagram 60 shown in Figure 7(A), the bundle of lines 63 contains many lines, and the rectangle 64 is rectangular in shape. Therefore, the control unit 10 confirms that the pattern is the same as the master image. On the other hand, in the case of explanatory diagram 65 shown in Figure 7(B), the bundle of lines 67 contains only a few lines, and the rectangle 68 is irregularly shaped. Therefore, the control unit 10 confirms that the pattern is different from the master image. Subsequently, the control unit 10 moves the process to S13 in Figure 5.

[0045] Next, we will explain the brightness adjustment process in S13 of Figure 5. Figure 8 is a flowchart showing the brightness adjustment process in the document verification device 1 according to this embodiment. Figures 9 and 10 are diagrams illustrating the brightness adjustment process in the document verification device 1 according to this embodiment.

[0046] In step S31 of Figure 8, the control unit 10 (brightness adjustment unit 15) calculates the overall average pixel value of the target document image. The average pixel value can be expressed, for example, in RGB. Other indices (e.g., HSV) may also be used. In S32, the control unit 10 (brightness adjustment unit 15) compares the calculated average pixel value with the pixel value of the master image storage unit 32. Through this process, the control unit 10 can determine which image, the master image or the target document image, is brighter and which is darker. Figure 9(A) shows an example of calculating the average pixel value for the entire image. Image 71 is a color image of the master image, and Image 72 is the image of the target document. Images 71 and 72 are similar except for the difference in the color of the top part. Here, the overall average pixel value 71a of image 71 is greater than the overall average pixel value 72a of image 72. Therefore, image 71, represented by the average pixel value 71a, is brighter than image 72, represented by the average pixel value 72a.

[0047] In step S33 of Figure 8, the control unit 10 (brightness adjustment unit 15) divides the master image and the target document image into a grid and generates an average color array for each cell (from 1 to M). Figure 9(B) shows image 72 divided into multiple cells 73. Each cell 73 consists of M cells, numbered 1, 2, 3, ... from the top left to the right, with M in the bottom right. Although not shown, image 71 is similarly divided into M cells. Figure 10(A) shows the arrangement of average colors for images 71 and 72, with the average colors of the cells from 1 to M arranged horizontally.

[0048] In S34 of Figure 8, the control unit 10 (brightness adjustment unit 15) gradually brightens the darker average color array and adopts the stage where the color difference between the two cells exceeds the threshold and is the lowest. As shown in Figure 9(A), image 72 is darker than image 71. Therefore, the control unit 10 generates an array of average colors in which the array of average colors of image 72 is gradually brightened, as shown in the arrays 74a, 74b, 74c, 74d, ... in Figure 10(B). Then, as shown in Figure 10(C), the control unit 10 compares each of the average color arrays 74a, 74b, ... obtained by gradually brightening the average color array of image 72 with the average color array of image 71, comparing the cells at the same position. The control unit 10 then adopts the array 74x that has the fewest number of NG cells where the color difference exceeds the threshold. Subsequently, the control unit 10 moves the process to S14 in Figure 5.

[0049] Next, we will explain the color matching process in S14 of Figure 5. Figure 11 is a flowchart showing the color matching process in the document verification device 1 according to this embodiment. Figure 12 is a diagram illustrating the color matching process in the document verification device 1 according to this embodiment.

[0050] In step S41 of Figure 11, the control unit 10 (color matching unit 16) adjusts the target image to the brightness adopted by the brightness adjustment process described above. In S42, the control unit 10 (color matching unit 16) divides the master image and the target document image (one of the images being the image adjusted in S41) into a grid, and calculates the average color for each cell (from 1 to N). Here, the number of cells N is greater than the number of cells M divided in the brightness adjustment process. In Figure 12, the adjusted target document image 82, obtained by adjusting the brightness of image 72, is divided into multiple cells 83. Similarly, image 71 is divided into multiple cells 81. Each of the cells 81 and 83 consists of N cells, numbered 1, 2, 3, ... from the top left to the right, with N in the bottom right.

[0051] In S43 of Figure 11, the control unit 10 (color matching unit 16) calculates the color difference of the average color calculated for each square in the same position. In S44, the control unit 10 (color matching unit 16) calculates the percentage of masses whose color difference exceeds a certain threshold, and confirms whether the colors are the same based on the calculated percentage. The upper color tones of image 71 in Figure 12 and the adjusted target document image 82 are different. Therefore, the control unit 10 confirms that the color tones are not the same. Subsequently, the control unit 10 moves the process to S15 in Figure 5.

[0052] Next, we will explain the management code verification process in S16 of Figure 5. Figure 13 is a flowchart showing the management code verification process in the document verification device 1 according to this embodiment. Figure 14 is a diagram illustrating the management code verification process in the document verification device 1 according to this embodiment.

[0053] In S51 of Figure 13, the control unit 10 (region extraction unit 17) acquires the position information of the corresponding master image stored in the master image storage unit 32 that corresponds to the target document image converted to grayscale. Then, based on the acquired position information, the control unit 10 (region extraction unit 17) extracts a matching region image from the grayscale target document image. Figure 14(A) shows an example of obtaining a matching region image 91 from a target document image 90 based on location information. Here, it is preferable that the location information is specified in a way that allows for the extraction of a wider area so as to ensure that the management code is included.

[0054] In step S52 of Figure 13, the control unit 10 (preprocessing unit 18) performs preprocessing on the extracted matching region image. As preprocessing, the control unit 10 performs, for example, contrast adjustment and sharpening. In S53, the control unit 10 (region pattern matching unit 19) extracts feature points from the pre-processed matching region image. In S54, the control unit 10 (region pattern matching unit 19) uses the code image feature points stored in the master image storage unit 32 and the extracted feature points to extract a rectangle that estimates the shape of the management code image from the matching region image. In S55, the control unit 10 (region pattern matching unit 19) cuts out a rectangle extracted from the matching region image.

[0055] Figure 14(B) visualizes the feature points of the management code image 95 stored in the master image storage unit 32 and the matching area image 91, with matching points represented by lines. The control unit 10 then extracts the rectangle 92 containing only the management code, and cuts it out to obtain the code area image 93.

[0056] In S56 of Figure 13, the control unit 10 obtains a string from the extracted code region image. The control unit 10 can obtain a string from the extracted code region image by, for example, performing OCR processing. In S57, the control unit 10 sends the acquired string to the document management server 190 (see Figure 2) and receives the matching confirmation result from the document management server 190. Subsequently, the control unit 10 moves the process to S17 in Figure 5.

[0057] Thus, the document verification device 1 of this embodiment has the following advantages. (1) The system acquires the target document image, compares the patterns of the target document image and the master image stored in the master image storage unit 32 based on their characteristic points, divides the target document image and the master image into multiple grid-like cells, compares the average color of the corresponding cells in both images to compare the colors of both images, and outputs a determination result regarding the degree of similarity between the two images based on the matching result of at least one of the pattern matching and color matching. Therefore, it is possible to verify whether the target document image is the same as the master image by comparing it from two different perspectives: the image itself and its color. This allows for accurate document verification using image processing. As a result, the burden on worker P in workplaces handling various types of documents can be reduced.

[0058] (2) Feature points are extracted from the target document image and the master image after conversion to grayscale, and the patterns of both images are compared, so that logos and text with color components excluded can be extracted as features. (3) Using the feature points of both images, a rectangle is extracted from the target document image to estimate the shape of the master image, and the patterns of both images are compared based on the shape of the extracted rectangle. Therefore, based on the shape of the rectangle extracted using feature points, it is possible to confirm whether or not the patterns are of the same type.

[0059] (4) Compare the pixel values ​​of the target document image and the master image, adjust the brightness of the darker image to match the brightness of the brighter image, and then compare the color tones of both images using the adjusted images. Therefore, color matching can be performed while mitigating the influence of environmental factors such as shooting conditions. In addition, since the brightness is adjusted to the brighter side of the image, color matching becomes easier. (5) Divide the target document image and the master image into multiple grid cells, compare the color differences of the corresponding cells in both images, and adjust the brightness so that the color difference is minimized. Therefore, by bringing the brightness of the images closer to the same conditions, it becomes possible to compare the colors. (6) Processing efficiency can be improved by making the size of the grid for brightness adjustment larger than the size of the grid for color matching.

[0060] (7) The camera 5 acquires images of the target documents, and the judgment result regarding the degree of similarity between the two images is output to the monitor 4. This makes it convenient for worker P to compare the images while working and to check the results by looking at the monitor 4.

[0061] (8) The target document image is acquired, the patterns of both images are compared based on the feature points of the target document image and the master image, a matching region image is extracted from the target document image, and the patterns of both images are compared based on the feature points of the extracted matching region image and the management code image stored in the master image storage unit 32. Therefore, by performing a two-stage image processing comparison to verify the management code contained in the target document image, the management code can be verified more accurately.

[0062] (9) Using the feature points of both images, a rectangle is extracted that estimates the shape of the management code image, and the patterns of the management code image and the matching region image are compared based on the shape of the extracted rectangle. Then, the rectangle extracted from the matching region image is cut out. Therefore, based on the shape of the rectangle extracted using feature points, it is possible to confirm whether the patterns are of the same type, that is, their identity with the management code image. Furthermore, by performing character recognition on the extracted rectangle, the management code can be matched as text.

[0063] (10) Based on the position information of the management code image in the master image stored in the master image storage unit 32, a matching area image is extracted from the target document image. Therefore, it is possible to easily extract matching region images from the target document images.

[0064] (11) After preprocessing the matching region image, the preprocessed matching region image is used to compare it with the management code image. Therefore, the accuracy of the matching can be improved.

[0065] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above. Furthermore, the effects described in the embodiments are merely a list of the most preferred effects arising from the present invention, and the effects of the present invention are not limited to those described in the embodiments. The embodiments described above and the modified forms described later can be used in combination as appropriate, but a detailed explanation is omitted.

[0066] (Transformed form) (1) In this embodiment, the document verification device has been described as a device that verifies the management code after verifying the overall image and color, but is not limited to this. For example, after verifying the overall image, the device may verify the management code without verifying the color. Alternatively, the device may verify only the management code without verifying the overall image or color, and then extract the management code as a string using OCR to verify that it matches the string. Even if that is done, if each document is assigned a management code, it is possible to determine whether they are of the same type or not by checking the management code.

[0067] (2) In this embodiment, a method has been described in which the master image is processed in advance and stored in the master image storage unit 32, but the system is not limited to this. For example, the master image may be acquired each time a document verification process is performed. However, processing and storing the image in advance is thought to contribute to improving the efficiency of the document verification process.

[0068] (3) In this embodiment, the size of the cells divided in the brightness adjustment process and the color matching process was explained using the example where the cells divided in the brightness adjustment process are larger than the cells divided in the color matching process. This is because the accuracy of the color matching process can be expected to improve if the matching is performed in units of finer size. However, the size of the cells can be set to an appropriate size depending on the size and content of the image.

[0069] (4) In this embodiment, an example was described in which an operator takes one document from a stack of documents, places it on a display stand, and then photographs it with a camera, but the invention is not limited to this. For example, the documents may be placed on the display stand in a stack, photographed with a camera for confirmation, and then the photographed document at the top may be sealed.

[0070] (5) In this embodiment, several different processes have been described as preprocessing, but the embodiment is not limited to these. It is not necessary to use all of the processes; one or more processes may be used. (6) In this embodiment, the system configuration of the document verification device has not been specifically described, but it may be a standalone device configured to perform verification processing independently within the terminal, or it may have a control unit and storage unit provided on a server or PC, and the processing may be done on the server or PC. [Explanation of symbols]

[0071] 1. Document verification device 4 monitors 5 Cameras 6 speakers 8 switches 10 Control Unit 11. Start Reception Section 12 Image acquisition unit 13 Color conversion section 14. Pattern matching section 15 Brightness adjustment section 16. Color matching section 17 Region extraction part 18 Pre-processing section 19 Area Pattern Matching Unit 20. Output section for confirmation results 30 Storage section 31a Document Verification Program 39 Communication Interface Section 40 Mounting platform 41 Mounting area 51 bundles 51a Document 60, 65 Explanatory Diagrams 61 Grayscale Images 62, 66, 82, 90 Target document images 64,68,92 rectangle 71,72 images 73, 81, 83 squares 74a,74b,74c,74d,···,74x array 91 Matching area image 93 Code region image 95 Management Code Image 100 Document Verification System 190 Document Management Server P worker

Claims

1. A document verification device that verifies the degree of similarity with a reference document image, which is an image of a reference document, A reference image storage unit that stores the aforementioned reference document images, A means for acquiring a target document image, which is a photograph of the document to be verified, A pattern matching means compares the patterns of the target document image acquired by the target document image acquisition means and the reference document image stored in the reference image storage unit based on their characteristic points. A brightness adjustment means that adjusts the brightness by comparing the pixel values ​​of the target document image acquired by the target document image acquisition means with the reference document image stored in the reference image storage unit, A color matching means divides the target document image and the reference document image, after adjustment by the brightness adjustment means, into a plurality of grid-like regions, and compares the average color of the corresponding regions of both images to match the color tones of both images. A determination result output means outputs a determination result relating to the degree of similarity between the two images based on the matching results of the pattern matching means and the color matching means, Equipped with, The brightness adjustment means divides the target document image acquired by the target document image acquisition means and the reference document image stored in the reference image storage unit into a plurality of grid-like regions, compares the color difference of the corresponding regions of both images, and adjusts the brightness so that the color difference is minimized, in a document verification device.

2. In the document verification device described in claim 1, The system includes a color conversion means for converting the aforementioned target document image to grayscale, The pattern matching means is a document verification device that extracts the feature points from the grayscaled reference document image and the grayscaled target document image by the color conversion means, and matches the patterns of both images.

3. In the document verification device described in claim 1, The pattern matching means is a document verification device that uses the feature points of both images to extract a rectangle from the target document image that estimates the shape of the reference document image, and matches the patterns of both images based on the shape of the extracted rectangle.

4. In the document verification device described in Claim 1, A document verification device wherein the size of the region divided by the brightness adjustment means is larger than the region divided by the color matching means.

5. In the document verification device according to any one of claims 1 to 4, The aforementioned document image acquisition means acquires the document image via the shooting unit that photographs the document to be confirmed, The judgment result output means outputs the judgment result relating to the same degree of matching between the two images to a display unit, and is a document verification device.

6. Computers, A means for acquiring a target document image, which is a photograph of the document to be verified, A pattern matching means compares the patterns of the target document image acquired by the target document image acquisition means and the reference document image, which is an image of a reference document stored in the reference image storage unit, based on the feature points of the two images. A brightness adjustment means that adjusts the brightness by comparing the pixel values ​​of the target document image acquired by the target document image acquisition means with the reference document image stored in the reference image storage unit, A color matching means divides the target document image and the reference document image, after adjustment by the brightness adjustment means, into a plurality of grid-like regions, and compares the average color of the corresponding regions of both images to match the color tones of both images. A determination result output means outputs a determination result relating to the degree of similarity between the two images based on the matching results of the pattern matching means and the color matching means, and make it work A program that causes the brightness adjustment means to function by dividing the target document image acquired by the target document image acquisition means and the reference document image stored in the reference image storage unit into a plurality of grid-like regions, comparing the color difference of the corresponding regions of both images, and adjusting the brightness so that the color difference is minimized.

Citation Information

Patent Citations

  • Job procedure extrapolating system and program

    JP2009224958A

  • Business form discrimination device, business form classification system, business form discrimination method and business form discrimination program

    JP2011243053A

  • Information processor

    JP2018156517A