An anchor-free template matching-based fixed format card identification method and device, computer equipment and storage medium

Abstract

The application discloses a fixed format card identification method and device based on anchor-free template matching, computer equipment and a storage medium. The application selects a standard template, aligns a to-be-identified sample with the standard template based on an anchor-free alignment algorithm, extracts identification content from the to-be-identified sample according to the coordinates of a standard frame of to-be-extracted content marked in advance on the standard template, and finally stores the identification content in a target file. The method has excellent robustness and accuracy for to-be-identified samples affected by inclination, air permeability, illumination and the like.

Description

Technical Field

[0001] This invention relates to the field of identification and extraction technology, and in particular to a method, apparatus, computer equipment, and storage medium for identifying fixed-format cards based on anchor-free template matching. Background Technology

[0002] With the continuous development of the social economy, financial management has increasingly become one of the core aspects of enterprise management. In financial management, invoice processing is a tedious yet crucial task. As business volume increases, traditional manual invoice processing can no longer meet actual needs; therefore, automated invoice recognition technology has emerged.

[0003] Existing invoice recognition technologies are mainly based on deep learning methods. However, these methods have some problems when processing value-added tax (VAT) invoices, such as high training costs, long development cycles, high commercialization costs, low recognition efficiency, slow recognition speed, and weak adaptability. Summary of the Invention

[0004] In view of this, the present invention provides a fixed-format card recognition method, device, computer equipment and storage medium based on anchor-free template matching. It does not require the use of deep learning methods. It only needs to select a high-quality standard template to quickly and accurately identify key information in invoices, improve work efficiency and is applicable to large-scale invoice processing needs.

[0005] A fixed-format card recognition method based on anchor-free template matching specifically includes the following steps:

[0006] S1, Select the standard template;

[0007] S2, mark the content to be extracted on the standard template according to user requirements and record the position coordinates of the marking box;

[0008] S3 uses OCR text recognition technology to identify the position and coordinates of all text blocks in the standard template and the sample to be recognized respectively;

[0009] S4. Based on the position and coordinates of the text blocks in the standard template and the sample to be identified, the transformation matrix from the sample to be identified to the standard template is calculated using the anchorless alignment algorithm, thus completing the alignment of the sample to be identified with the standard template.

[0010] S5, extract the recognition content from the position of the sample to be recognized, which is the same as the position coordinate of the annotation box;

[0011] S6. Save the extracted recognition content to the target file.

[0012] Preferably, in step S4, the transformation matrix from the sample to be identified to the standard template is calculated using an anchorless alignment algorithm. The specific steps for aligning the sample to be identified with the standard template are as follows:

[0013] Select n text blocks from the standard template, denoted as the first text block; select m text blocks from the sample to be identified, denoted as the second text block;

[0014] By performing distance matching between n first text blocks and m second text blocks, n pairs of matching text blocks are obtained;

[0015] Four pairs of text blocks are randomly selected from n pairs of matching text blocks, and the homography matrix is ​​calculated based on the vertex coordinates of the four selected pairs of text blocks.

[0016] The homography matrix is ​​used to transform the coordinates of the remaining text blocks in the sample to be identified and the difference between the transformed coordinates and the coordinates of the corresponding text blocks in the standard template is calculated. When the calculated difference is less than a set threshold, it means that the homography matrix calculation is successful and the current homography matrix is ​​used as the transformation matrix from the sample to be identified to the standard template; otherwise, it means that the homography matrix calculation has failed and the above steps are repeated to randomly select 4 pairs of text blocks from n pairs of matching text blocks until the requirements are met.

[0017] The homography matrix obtained in the previous step is used to transform the coordinates of the text blocks in the sample to be recognized, thereby aligning the sample to be recognized with the standard template.

[0018] Preferably, the specific steps for performing distance matching between n first text blocks and m second text blocks are as follows:

[0019] Feature vectors are extracted for each first text block and each second text block respectively;

[0020] Calculate the distance between the feature vector of a certain first text block and the feature vectors of all second text blocks. If the distance between the feature vector of only one second text block and the feature vector of the first text block is less than a set threshold, it means that the current first text block and the second text block are successfully matched; otherwise, the current first text block fails to match.

[0021] Repeat the above steps to traverse each first text block and perform distance matching between each first text block and each second text block in turn to obtain n pairs of matching text blocks.

[0022] Preferably, the feature vector includes nine values, namely the aspect ratio of the text block currently being extracted, and the aspect ratios of the eight adjacent text blocks located above, below, left, right, upper left, lower left, upper right, and lower right.

[0023] Preferably, when extracting feature vectors from text blocks, if an adjacent text block in a certain direction does not exist, it is replaced with 0.

[0024] Preferably, each text block has 4 vertex coordinates.

[0025] The remaining text block refers to the text block that belongs to the sample to be extracted after randomly selecting 4 pairs of text blocks from n pairs of matching text blocks.

[0026] Preferably, the fixed-format cards include invoices, property ownership certificates, marriage certificates, and bank cards.

[0027] A fixed-format card recognition device based on anchor-free template matching, comprising:

[0028] The template selection module is used to manually select a standard template from a large number of samples that is free from tilt, rotation, perspective, and lighting effects.

[0029] The template annotation module is used to annotate the content to be extracted on a standard template according to user requirements and record the position coordinates of the annotation box;

[0030] The text recognition module is used to identify the position and coordinates of all text blocks in the standard template and the sample to be recognized using OCR text recognition technology.

[0031] The sample and template alignment module is used to calculate the transformation matrix from the sample to the standard template based on the position and coordinates of the text blocks in the standard template and the sample to be identified, using an anchorless alignment algorithm, and complete the alignment of the sample to be identified with the standard template.

[0032] The extraction and recognition module is used to extract the recognition content from the position of the sample to be recognized at the same position coordinate as the marked box, and save the extracted recognition content to the target file.

[0033] A computer device, the computer device comprising:

[0034] One or more processors;

[0035] Memory, used to store one or more programs.

[0036] When the one or more programs are executed by the one or more processors, the one or more processors implement the method.

[0037] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method.

[0038] The beneficial effects of this invention are:

[0039] 1. This invention selects a standard template, aligns the sample to be identified with the standard template based on an anchorless alignment algorithm, then extracts the identification content from the sample according to the coordinates of the standard box of the content to be extracted marked on the standard template in advance, and finally stores the identification content in the target file. This method has excellent robustness and accuracy for samples to be identified that are affected by tilt, ventilation, lighting and other factors.

[0040] 2. This invention utilizes an anchor-free alignment algorithm to align the sample to be identified with a standard template. Without the need for deep learning methods, it can quickly and accurately identify key information in the sample, greatly improving work efficiency and making it suitable for large-scale invoice processing needs. Furthermore, the anchor-free alignment algorithm does not require pre-marking anchor points or searching for anchor point positions, and can simply, conveniently, efficiently, and stably identify content of interest to users.

[0041] 3. This invention utilizes an anchorless alignment algorithm to accurately identify key information in VAT invoices, avoiding the reliance on text recognition in traditional methods and improving recognition accuracy.

[0042] 4. This invention allows users to select and extract different areas of interest from invoices according to their needs, demonstrating strong flexibility and adaptability. It is also applicable to other fixed-format card and document extraction tasks, such as property ownership certificates, marriage certificates, and bank cards.

[0043] 5. This invention is less affected by factors such as printing quality and shooting angle, and has good stability and reliability. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0047] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0048] It should be understood that although the terms first, second, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms and should not be construed as indicating or implying relative importance. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0049] In the following description, suffixes such as "module," "part," or "unit" used to denote elements are used only for the convenience of the description of the invention and have no specific meaning in themselves. Therefore, "module" and "part" can be used interchangeably.

[0050] This invention provides a fixed-format card recognition method based on anchor-free template matching. The method selects a standard template, aligns the sample to be recognized with the standard template using an anchor-free alignment algorithm, extracts the recognition content from the sample to be recognized based on the coordinates of the standard box marked on the standard template, and finally stores the recognition content in a target file. This method has excellent robustness and accuracy for samples to be recognized that are affected by tilt, ventilation, lighting, etc.

[0051] This invention requires no parameterized network training, is simple and efficient, and can be applied to fixed-format cards and certificates, such as invoices, property ownership certificates, marriage certificates, bank cards, etc.

[0052] To better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings.

[0053] This invention provides a fixed-format card recognition method based on anchor-free template matching, specifically including the following steps:

[0054] S1, Select a standard template.

[0055] The standard template refers to a sample selected from a large number of existing samples that is free from tilt, rotation, perspective, and lighting effects.

[0056] S2, mark the content to be extracted on the standard template according to user requirements and record the position coordinates of the marking box.

[0057] S3 uses OCR text recognition technology to identify the position and coordinates of all text blocks in the standard template and the sample to be recognized.

[0058] Taking VAT invoices as an example, OCR text recognition technology is used to identify the position and coordinates of text blocks including "invoice code", "invoice number", "buyer", "total price and tax (uppercase)" and "lowercase" in standard invoices and invoices to be identified.

[0059] S4. Based on the position and coordinates of the text blocks in the standard template and the sample to be identified, the transformation matrix from the sample to the standard template is calculated using the anchorless alignment algorithm, thus completing the alignment between the sample to be identified and the standard template.

[0060] Specifically, the transformation matrix from the sample to be identified to the standard template is calculated using an anchorless alignment algorithm. The specific steps to align the sample to be identified with the standard template are as follows:

[0061] S41, select n text blocks from the standard template, denoted as the first text block; select m text blocks from the sample to be identified, denoted as the second text block;

[0062] S42, perform distance matching between n first text blocks and m second text blocks:

[0063] First, feature vectors are extracted for each first text block and each second text block. The feature vector includes nine values: the aspect ratio of the text block being extracted, and the aspect ratios of the eight adjacent text blocks located above, below, left, right, upper left, lower left, upper right, and lower right. When extracting feature vectors for a text block, if an adjacent text block in a certain position does not exist, it is replaced with 0. Regardless of whether the sample to be identified has been scaled, translated, or perspectived relative to the standard template, the aspect ratio of the text block itself and the aspect ratios of the surrounding text blocks remain unchanged. This is the principle of feature vector design.

[0064] Then, calculate the distance between the feature vector of the first text block and the feature vectors of all the second text blocks. If the distance between the feature vector of only one second text block and the feature vector of the first text block is less than the set threshold, it means that the current first text block is successfully matched and the first text block is paired with the second text block whose calculated distance is less than the set threshold. Otherwise, the current first text block fails to match.

[0065] Repeat the above steps to iterate through each first text block and perform distance matching between each first text block and each second text block in turn to obtain n pairs of matching text blocks.

[0066] For example, suppose 10 text blocks A1...A are selected from the standard module.10 Twenty text blocks B1...B1 were selected from the samples to be identified. 20 Feature vectors are extracted from each of the 30 text blocks; then, the A1...A1 modules of the standard module are iterated sequentially. 10 For each of these 10 text blocks, calculate its feature vector and the corresponding text block B1...B1. 20 The distance between the feature vectors is calculated. If, among the 20 calculated vector distances, only the distance between the feature vector of text block B1 and the feature vector of text block A1 is less than a set threshold, then text block A1 and text block B1 are successfully matched. This step is then repeated sequentially for text blocks A2...A... 10 Distance matching is performed with each second text block to obtain 9 pairs of matching text blocks.

[0067] S43, randomly select 4 pairs of text blocks from n pairs of matching text blocks, and calculate the homography matrix based on the vertex coordinates of the 4 pairs of selected text blocks. Each text block has 4 vertex coordinates; that is, the 4 vertex coordinates of each of the n pairs of matching text blocks are recorded respectively, so that 4*n pairs of vertex coordinates can be obtained. Using the random sampling method, 4 pairs are randomly selected from the 4*n pairs of vertex coordinates, and the homography matrix is ​​calculated using these 4 randomly selected pairs of coordinate values.

[0068] S44. Use the homography matrix to transform the coordinates of the remaining text blocks in the sample to be identified and calculate the difference between the transformed coordinates and the coordinates of the corresponding text blocks in the standard template. When the calculated difference is less than a set threshold, it means that the homography matrix calculation is successful and the current homography matrix is ​​used as the transformation matrix from the sample to be identified to the standard template; otherwise, it means that the homography matrix calculation has failed and the above steps are repeated to randomly select 4 pairs of text blocks from n pairs of matching text blocks until the requirements are met.

[0069] The failure to calculate the homography matrix indicates that the coordinates of the four pairs of vertices randomly selected in step S43 above are incorrect and need to be reselected.

[0070] The remaining text block refers to the text block that belongs to the sample to be extracted after randomly selecting 4 pairs of text blocks from n pairs of matching text blocks.

[0071] S45, use the homography matrix obtained in the previous step to transform the coordinates of the text blocks in the sample to be recognized, so as to align the sample to be recognized with the standard template.

[0072] S5, map the position coordinates of the content to be extracted that are pre-marked on the standard template to the sample to be identified, and extract the content to be identified from the position of the sample to be identified corresponding to the position coordinates of the marked box.

[0073] S6. Save the extracted recognition content to the target file.

[0074] The present invention also provides a fixed-format card recognition device based on anchor-free template matching, including a template selection module, a template annotation module, a text recognition module, a sample and template alignment module, and an extraction and recognition module.

[0075] The template selection module is used to manually select a standard template from a large number of samples that is free from tilt, rotation, perspective, and lighting effects.

[0076] The template annotation module is used to annotate the content to be extracted on a standard template according to user requirements and record the position coordinates of the annotation box;

[0077] The text recognition module is used to identify the position and coordinates of all text blocks in the standard template and the sample to be recognized using OCR text recognition technology.

[0078] The sample and template alignment module is used to calculate the transformation matrix from the sample to the standard template based on the position and coordinates of the text blocks in the standard template and the sample to be identified, using an anchorless alignment algorithm, and complete the alignment of the sample to be identified with the standard template.

[0079] The extraction and recognition module is used to extract the recognition content from the position of the sample to be recognized at the same position coordinate as the marked box, and save the extracted recognition content to the target file.

[0080] The present invention also provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method.

[0081] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the method described in the embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0082] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0083] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

Claims

1. A fixed-format card recognition method based on anchor-free template matching, characterized in that, Specifically, the following steps are included: S1, Select the standard template; S2, mark the content to be extracted on the standard template according to user requirements and record the position coordinates of the marking box; S3 uses OCR text recognition technology to identify the position and coordinates of all text blocks in the standard template and the sample to be recognized respectively; S4. Based on the position and coordinates of the text blocks in the standard template and the sample to be identified, the transformation matrix from the sample to be identified to the standard template is calculated using the anchorless alignment algorithm, thus completing the alignment of the sample to be identified with the standard template. Specifically, firstly, n text blocks are selected from the standard template, denoted as the first text block; and m text blocks are selected from the samples to be identified, denoted as the second text block. By performing distance matching between n first text blocks and m second text blocks, n pairs of matching text blocks are obtained; Four pairs of text blocks are randomly selected from n pairs of matching text blocks, and the homography matrix is ​​calculated based on the vertex coordinates of the four selected pairs of text blocks. The homography matrix is ​​used to transform the coordinates of the remaining text blocks in the sample to be identified and the difference between the transformed coordinates and the coordinates of the corresponding text blocks in the standard template is calculated. When the calculated difference is less than a set threshold, it means that the homography matrix calculation is successful and the current homography matrix is ​​used as the transformation matrix from the sample to be identified to the standard template; otherwise, it means that the homography matrix calculation has failed and the above steps are repeated to randomly select 4 pairs of text blocks from n pairs of matching text blocks until the requirements are met. The homography matrix obtained in the previous step is used to transform the coordinates of the text blocks in the sample to be recognized, thereby aligning the sample to be recognized with the standard template. S5, extract the recognition content from the position of the sample to be recognized, which is the same as the position coordinate of the annotation box; S6. Save the extracted recognition content to the target file.

2. The fixed-format card recognition method based on anchor-free template matching according to claim 1, characterized in that, The specific steps for performing distance matching between n first text blocks and m second text blocks are as follows: Feature vectors are extracted for each first text block and each second text block respectively; Calculate the distance between the feature vector of a certain first text block and the feature vectors of all second text blocks. If the distance between the feature vector of only one second text block and the feature vector of the first text block is less than a set threshold, it means that the current first text block and the second text block are successfully matched; otherwise, the current first text block fails to match. Repeat the above steps to traverse each first text block and perform distance matching between each first text block and each second text block in turn, to obtain n pairs of matching text blocks.

3. The fixed-format card recognition method based on anchor-free template matching according to claim 2, characterized in that, The feature vector includes nine values: the aspect ratio of the text block currently being extracted, and the aspect ratios of the eight adjacent text blocks located above, below, left, right, upper left, lower left, upper right, and lower right.

4. The fixed-format card recognition method based on anchorless template matching according to claim 3, characterized in that, When extracting feature vectors from text blocks, if an adjacent text block in a certain position does not exist, it is replaced with 0.

5. The fixed-format card recognition method based on anchorless template matching according to claim 1, characterized in that, Each text block has 4 vertex coordinates. The remaining text block refers to the text block that belongs to the sample to be extracted after randomly selecting 4 pairs of text blocks from n pairs of matching text blocks.

6. The fixed-format card recognition method based on anchor-free template matching according to claim 1, characterized in that, The fixed-format cards and certificates include invoices, property ownership certificates, marriage certificates, and bank cards.

7. A fixed-format card recognition device based on anchor-free template matching, characterized in that, include: The template selection module is used to manually select a standard template from a large number of samples that is free from tilt, rotation, perspective, and lighting effects. The template annotation module is used to annotate the content to be extracted on a standard template according to user requirements and record the position coordinates of the annotation box; The text recognition module is used to identify the position and coordinates of all text blocks in the standard template and the sample to be recognized using OCR text recognition technology. The sample and template alignment module is used to select n text blocks from the standard template, denoted as the first text block; select m text blocks from the sample to be identified, denoted as the second text block; and perform distance matching between the n first text blocks and the m second text blocks to obtain n pairs of matching text blocks. Four pairs of text blocks are randomly selected from n pairs of matching text blocks. The homography matrix is ​​calculated based on the vertex coordinates of the four selected pairs of text blocks. The homography matrix is ​​used to transform the coordinates of the remaining text blocks in the sample to be identified, and the difference between the transformed coordinates and the coordinates of the corresponding text blocks in the standard template is calculated. When the calculated difference is less than a set threshold, it means that the homography matrix calculation is successful, and the current homography matrix is ​​used as the transformation matrix from the sample to be identified to the standard template. Otherwise, it means that the homography matrix calculation has failed, and the above steps are repeated to randomly select four pairs of text blocks from n pairs of matching text blocks again until the requirements are met. The homography matrix obtained in the previous step is used to transform the coordinates of the text blocks in the sample to be recognized, thereby aligning the sample to be recognized with the standard template. The extraction and recognition module is used to extract the recognition content from the position of the sample to be recognized at the same position coordinate as the marked box, and save the extracted recognition content to the target file.

8. A computer device, characterized in that, The computer device includes: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method as described in any one of claims 1-6.

Images