Invoice identification method based on bounding box corner detection and correction
Through bounding box corner detection and perspective transformation correction technology, the low accuracy caused by layout differences and rotation angles in invoice recognition is solved, and high accuracy and automated invoice recognition are achieved.
Patent Information
- Application Number
- CN202410640710.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-22
- Publication Date
- 2024-12-06
AI Technical Summary
The existing invoice recognition technology has complex field filtering logic and low accuracy due to layout differences and rotation angles, and requires manual proofreading, which affects the recognition effect.
The correction method based on bounding box corner point detection is adopted, and the four corner points of the invoice are found through yolov5 object detection and Harris corner point detection, the perspective transformation matrix is calculated for image correction, and the invoice type is identified using the SVTR text recognition algorithm, combined with the LabelImg tool annotation Field location, automatic identification.
It improves the accuracy and robustness of invoice recognition, reduces the need for manual proofreading, and can maintain image alignment under large rotation angles, achieving automated recognition.
Smart Images

Figure FT_1 
Figure FT_2 
Figure FT_3
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of computer vision recognition, and in particular is an invoice recognition method based on bounding box corner point detection and correction. Background Art
[0002] As a kind of audit and reimbursement voucher, invoices are used more and more widely with the development of society. However, the audit and reimbursement of a large number of invoices is a heavy burden for the staff of each unit. With the development of artificial intelligence technology, text OCR recognition has replaced manual recognition of text information as a result of social development. Similarly, text OCR recognition technology needs to be widely used in the invoice recognition process.
[0003] At present, invoice recognition has become mature and applied with the development of deep learning technology. A method based on OCR recognition and field matching filtering has emerged. However, due to the differences in invoice fields and spacing in various formats, the field filtering logic is complex, fields conflict, and the accuracy is low. Often, invoice recognition requires repeated manual proofreading. At the same time, the invoice stamp covers the font, the invoice angle is not correct, and the clarity is not enough, which will affect the recognition effect.
[0004] Therefore, invoice recognition is still a challenging task; this paper proposes an invoice recognition method based on bounding box corner detection and correction. Summary of the invention
[0005] The present invention proposes a new idea for invoice recognition, which can solve the problems of complex logic, field conflicts and low accuracy in processing invoices by text recognition and filtering. The bounding box corner point detection and correction method proposed in the present invention can be transferred to text recognition scenarios with four bounding boxes, such as business licenses and table recognition.
[0006] An invoice recognition method based on bounding box corner point detection and correction includes the following steps: S1. Object Detection The yolov5 target detection algorithm is used to detect the corner areas and invoice category areas on the invoice bounding box; the corner areas are four corner areas, and the invoice category areas include VAT general invoices, VAT electronic invoices, VAT special invoices, and VAT electronic special invoices; the R channel of the RGB color image is taken, the R channel is copied to replace the G and B channels, and the images composed of three channels are replaced and fused to perform detection training and reasoning on the four corner areas of the bounding box and the invoice category areas; S2. Corner Detection For the four corner regions detected in step S1, the Harris corner detection algorithm is used to find the coordinate positions of the four corner points. Before using the Harris corner detection algorithm, the detected corner regions are first segmented using the OTSU binary segmentation method in the OpenCV computer vision library to obtain a binary image. Then, image morphology expansion and corrosion processing is performed. A 2*2 structural element is used to corrode the diagonal region to prevent the boundary line of the corner region from being broken after binarization. Then, a 3*3 structural element is used to expand the diagonal region to refine the boundary line. Finally, the Harris corner detection algorithm is used to obtain the coordinate positions of the corner points using a 3*3 size template. S3. Image Correction For the invoice category area detected in step S1, the invoice category is obtained by OCR recognition. The coordinate order of the perspective transformation of the four corner points is inferred based on the position of the invoice category box and the quadrilateral box formed by the four corner points. The perspective transformation matrix is calculated based on the coordinates of the four corner points detected and the four boundary points of the standard invoice type under 1500*950 pixels. According to the projection matrix, the current invoice is projected on a 1500*950 pixel image to obtain a corrected invoice image. The four corner points obtained in S1 form a quadrilateral. The invoice category area box is analyzed to be located at the top, bottom, left and right of the large quadrilateral. The invoice category box is below the large quadrilateral, indicating that the invoice is rotated 180 degrees. The point in the lower right corner actually corresponds to the first coordinate of the perspective transformation projection, and the point in the lower left corner actually corresponds to the The second coordinate, the upper left corner point corresponds to the third coordinate of the perspective transformation projection, and the upper right corner point corresponds to the fourth coordinate of the perspective transformation projection; according to the relative position of the invoice boundary box relative to the invoice category box, the appropriate coordinate order of the four corner points in other directions is inferred by this method, and the upright invoice category area cropped image can be obtained; similarly, there may be red seal interference in this area, and the red seal is removed by the method described in S1 before OCR text recognition, and then the SVTR text recognition algorithm is used to recognize the text content to obtain the invoice type; the perspective transformation projection matrix is calculated based on the coordinates of the four corner points detected and sorted and the four boundary points of the standard invoice type at a size of 1500*950, and the current invoice is projected on a 1500*950 size image according to the projection matrix to obtain the corrected invoice image and invoice category; S4. The SVTR text recognition OCR algorithm described in step S3 identifies the invoice category. Specifically, the invoice category text area has been detected in S1. The SVTR text recognition algorithm integrated in the Baidu paddleOCR project is used to perform text recognition on the intercepted invoice category area image, so that the type of the invoice can be accurately identified; S5. Key field marking For invoices, use the LabelImg rectangular box annotation tool to mark the area position of each text content of this type of standard invoice at 1500*950 pixels and save it as a Json file. Import the standard invoice template, scale it to 1500*950 pixels, and then mark the invoice field area to be identified. Save the area coordinates in the Json file according to the invoice type and different fields. Subsequent identification will directly read and match without repeated marking; S6. The recognition module obtains the coordinates of the area to be recognized in the invoice category in step S4 according to the invoice category obtained in step S3, and completes the cropping of the recognition area image of the corrected invoice image obtained in step S3 according to the recognition area identifier and coordinates. Finally, the SVTR text recognition algorithm is used to recognize the text content of the cropped image and match it one by one with the field type identified by the LabelImg tool to obtain the invoice recognition result.
[0007] Furthermore, the training data for target detection in S1 is ≥ 50,000 invoice annotation data.
[0008] Furthermore, the S5 standard invoice template is a standard invoice template downloaded from the official website of the national or local tax bureau.
[0009] Beneficial effects of the invention: The invention creatively uses the target detection algorithm to detect the four corner areas of the largest outer frame of the invoice, and first processes the corner areas through image morphological corrosion and dilation before using the Harris corner detection algorithm to prevent corner detection failure, making the detection algorithm more reliable and accurate; According to the positional relationship between the invoice category box and the rectangle formed by the four corner points, the coordinate correspondence when calculating the perspective transformation matrix is determined to prevent the coordinate correspondence from being misaligned due to the large rotation angle of the invoice. This method can ensure that an aligned image can be obtained after the invoice is perspectively distorted, and the algorithm is very robust. The invoice recognition area only needs to use the rectangular marking tool to mark the recognition area coordinates of a type of invoice in advance. The new invoice recognition can be aligned to the marked invoice through correction, and the invoice recognition result can be obtained conveniently and accurately.
[0010] The beneficial effect of the present invention is that the captured non-standard invoice image is projected onto a standard invoice format of fixed size and fixed format through perspective transformation, and then the invoice recognition area is cropped by pre-marked field positions to obtain accurate recognition results. This method has great guiding and practical value for the recognition of invoices, business licenses, cards and certificates containing obvious boundary information, and provides a very innovative method for card and certificate recognition, which can bring positive social value. BRIEF DESCRIPTION OF THE DRAWINGS
[0011] Figure 1This is a schematic diagram of the four corner regions and invoice category region detection boxes used for training annotation and reasoning; Figure 2 It is a schematic diagram illustrating the steps involved in extracting corner points from a corner point area; Figure 3 It is a schematic diagram of the relative position relationship between the rectangular frame formed by the corner points and the invoice category frame; Figure 4 It is a schematic diagram of the original invoice after correction and projection at 1500*900 pixels; Figure 5 This is a schematic diagram of the area to be identified for a standard invoice of 1500*900 pixels using the rectangular frame labeling tool LabelImg. DETAILED DESCRIPTION
[0012] The purpose of the present invention is to provide an invoice recognition method based on bounding box corner point detection and correction, which includes a target detection module, a corner point detection module, an image correction module, a key field marking module and a recognition module; it brings a new recognition idea and technical solution for the recognition of bills and cards such as invoices; it can formulate personalized recognition solutions without relying on filtering rules; in order to better illustrate the technical requirements and purposes of the present invention, the present invention is described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0013] S1. The target detection module uses the yolov5 target detection algorithm to simultaneously detect the four corner areas and the invoice category area on the invoice bounding box; the training data of this module includes 50,000 invoice annotation data, which is divided into two categories (corner areas and category areas). The training invoice categories include VAT general invoices, VAT electronic invoices, VAT special invoices and VAT electronic special invoices. In order to eliminate the interference of the red seal on the four corner areas and the invoice type area, we took the R channel of the RGB color image and copied two channels to form a three-channel image for bounding box four corner areas and category area detection training and reasoning. The model has a recognition accuracy of 99.8% on 2,000 test data; the four corner areas to be detected are as follows: Figure 1 As shown in the red box, the invoice type area is the field area on the invoice that contains "xx VAT xx invoice", such as Figure 1 The range shown in the blue box.
[0014] S2. The corner detection module uses the Harris corner detection algorithm to accurately find the coordinates of the four corners in the four corner regions detected in step S1. Specifically, before using the Harris corner detection algorithm, the detected corner regions are first segmented using the Otsu method (OTSU) in the OpenCV computer vision library to obtain a binary image, such as Figure 2As shown in (a), the image morphological dilation and corrosion processing is then performed, specifically using a 2*2 structural element to corrode the diagonal area, as shown in Figure 2 As shown in (b), to prevent the corner area boundary line from breaking after binarization, a 3*3 structure element is used to expand the diagonal area, as shown in Figure 2 As shown in (c), the boundary line is refined, and finally the Harris corner detection algorithm is used to obtain the accurate corner point coordinate position with a 3*3 size template, as shown in Figure 2 (d) as shown.
[0015] S3. The image correction module uses OCR to identify the invoice category area detected in step S1 to obtain the invoice category, and infers the coordinate order of the perspective transformation of the four corner points based on the position of the quadrilateral box formed by the invoice category box and the four corner points. The perspective transformation matrix is calculated based on the coordinates of the four corner points we detected and the four boundary points of the standard invoice type at 1500*950 pixels. For the perspective change matrix, see the glossary. According to the projection matrix, the current invoice is projected onto a 1500*950 pixel image to obtain a corrected invoice image. Specifically, for the four corner points obtained in S1 that form a quadrilateral, analyze the position of the invoice category area box in the large quadrilateral (upper, lower, left, right). Because the invoice category box is below the large quadrilateral, it means that the invoice is rotated about 180 degrees. The point in the lower right corner actually corresponds to the first coordinate of the perspective transformation projection, the point in the lower left corner actually corresponds to the second coordinate of the perspective transformation projection, the point in the upper left corner corresponds to the third coordinate of the perspective transformation projection, and the point in the upper right corner corresponds to the fourth coordinate of the perspective transformation projection. According to the relative position of the invoice boundary box relative to the invoice category box, the appropriate coordinate order of the four corner points in other positions can be inferred by this method. The relative external relationship diagram is as follows: Figure 3 As shown in (a) to (d), a cropped image of the upright invoice category area can be obtained; similarly, there may be red seal interference in this area. Before OCR text recognition, the red seal is removed using the method described in S1, and then the SVTR text recognition algorithm is used to recognize the text content to obtain the invoice type (VAT general, VAT special, electronic invoice, VAT electronic general, VAT electronic special); the projection matrix of the perspective transformation is calculated based on the coordinates of the four corner points we detected and sorted and the four boundary points of the standard invoice type when the size is 1500*950, and the current invoice is projected on a 1500*950 size image according to the projection matrix to obtain the corrected invoice image and invoice category. The correction effect is shown as follows: Figure 4 shown.
[0016] S4. The SVTR text recognition OCR algorithm described in step S3 identifies the invoice category. Specifically, the invoice category text area has been detected in S1. The SVTR text recognition algorithm integrated in the Baidu PaddleOCR project is used to perform text recognition on the intercepted invoice category area image, and the type of invoice (VAT general, VAT special, electronic invoice, VAT electronic general, VAT electronic special) can be accurately identified. S5. Key field marking module uses Labelimg rectangular box marking tool for various common invoices 4 Mark the area position of each text content of this type of standard invoice when it is 1500*950 and save it as a Json file. Specifically, download various standard invoice templates from the official website of the State Administration of Taxation, scale them to 1500*950, and then mark the invoice field area we want to identify. Save the area coordinates in the Json file according to the invoice type and different fields. Subsequent recognition will directly read and match without repeated marking. The invoice fields to be marked are as follows: Figure 5 shown.
[0017] S6. The recognition module obtains the coordinates of the area to be recognized in the invoice category in step S4 according to the invoice category obtained in step S3, and crops the corrected invoice image obtained in step S3 according to the recognition area identifier and coordinates. Finally, the SVTR text recognition algorithm is used to recognize the text content of the cropped image and match it one by one with the field type identified by the LabelImg tool to obtain the invoice recognition result.
[0018] Application scenarios of the present invention: 1. Tax reimbursement scenario: quickly identify and enter the information of each field of VAT general invoice or special invoice, and apply it to scenarios such as corporate tax accounting and internal reimbursement, effectively reduce the workload of manual accounting, and realize the automation of tax reimbursement. 2. Bill record: automatically identify and enter information such as invoice amount and invoice date, and apply it to financial accounting scenarios to help users quickly enter bill information, reduce user input costs, and improve user experience.
[0019] Glossary Corner point: A corner point is an extreme point, that is, a point with particularly prominent attributes in some aspect. It is an isolated point with the maximum or minimum intensity in some attributes, or the end point of a line segment. For an image, it represents the connection point of the object outline. In the present invention, the intersection point of the four sides of the outermost rectangular frame of the invoice is the corner point we want to detect.
[0020] Perspective transformation: It projects the image onto a new viewing plane, also known as projection mapping. The general transformation formula is: It is the left side of the original image, corresponding to the transformed image coordinates ,in . The 3*3 matrix formed represents the perspective transformation matrix.
Claims
1. An invoice recognition method based on bounding box corner point detection and correction, characterized in that The steps include: S1. Object Detection The yolov5 target detection algorithm is used to detect the corner areas and invoice category areas on the invoice bounding box; the corner areas are four corner areas, and the invoice category areas include VAT general invoices, VAT electronic invoices, VAT special invoices, and VAT electronic special invoices; the R channel of the RGB color image is taken, the R channel is copied to replace the G and B channels, and the images composed of three channels are replaced and fused to perform detection training and reasoning on the four corner areas of the bounding box and the invoice category areas; S2. Corner Detection For the four corner regions detected in step S1, the Harris corner detection algorithm is used to find the coordinate positions of the four corner points. Before using the Harris corner detection algorithm, the detected corner regions are first segmented using the OTSU binary segmentation method in the OpenCV computer vision library to obtain a binary image. Then, image morphology expansion and corrosion processing is performed. A 2*2 structural element is used to corrode the diagonal region to prevent the boundary line of the corner region from being broken after binarization. Then, a 3*3 structural element is used to expand the diagonal region to refine the boundary line. Finally, the Harris corner detection algorithm is used to obtain the coordinate positions of the corner points using a 3*3 size template. S3. Image Correction For the invoice category area detected in step S1, the invoice category is obtained by OCR recognition. The coordinate order of the perspective transformation of the four corner points is inferred based on the position of the invoice category box and the quadrilateral box formed by the four corner points. The perspective transformation matrix is calculated based on the coordinates of the four corner points detected and the four boundary points of the standard invoice type under 1500*950 pixels. According to the projection matrix, the current invoice is projected on a 1500*950 pixel image to obtain a corrected invoice image. The four corner points obtained in S1 form a quadrilateral. The invoice category area box is analyzed to be located at the top, bottom, left and right of the large quadrilateral. The invoice category box is below the large quadrilateral, indicating that the invoice is rotated 180 degrees. The point in the lower right corner actually corresponds to the first coordinate of the perspective transformation projection, and the point in the lower left corner actually corresponds to the The second coordinate, the upper left corner point corresponds to the third coordinate of the perspective transformation projection, and the upper right corner point corresponds to the fourth coordinate of the perspective transformation projection; according to the relative position of the invoice boundary box relative to the invoice category box, the appropriate coordinate order of the four corner points in other directions is inferred by this method, and the upright invoice category area cropped image can be obtained; similarly, there may be red seal interference in this area, and the red seal is removed by the method described in S1 before OCR text recognition, and then the SVTR text recognition algorithm is used to recognize the text content to obtain the invoice type; the perspective transformation projection matrix is calculated based on the coordinates of the four corner points detected and sorted and the four boundary points of the standard invoice type at a size of 1500*950, and the current invoice is projected on a 1500*950 size image according to the projection matrix to obtain the corrected invoice image and invoice category; S4. The SVTR text recognition OCR algorithm described in step S3 identifies the invoice category. Specifically, the invoice category text area has been detected in S1. The SVTR text recognition algorithm integrated in the Baidu paddleOCR project is used to perform text recognition on the intercepted invoice category area image, so that the type of the invoice can be accurately identified; S5. Key field marking For invoices, use the LabelImg rectangular box annotation tool to mark the area position of each text content of this type of standard invoice at 1500*950 pixels and save it as a Json file. Import the standard invoice template, scale it to 1500*950 pixels, and then mark the invoice field area to be identified. Save the area coordinates in the Json file according to the invoice type and different fields. Subsequent identification will directly read and match without repeated marking; S6. The recognition module obtains the coordinates of the area to be recognized in the invoice category in step S4 according to the invoice category obtained in step S3, and completes the cropping of the recognition area image of the corrected invoice image obtained in step S3 according to the recognition area identifier and coordinates. Finally, the SVTR text recognition algorithm is used to recognize the text content of the cropped image and match it one by one with the field type identified by the LabelImg tool to obtain the invoice recognition result.
2. According to the invoice recognition method based on bounding box corner point detection and correction, it is characterized in that The training data for target detection in S1 is ≥ 50,000 invoice annotation data.
3. According to the invoice recognition method based on bounding box corner point detection and correction, it is characterized in that The S5 standard invoice template is a standard invoice template downloaded from the official website of the national or local tax bureau.
