Document type financial report identification method and device in natural scene

Abstract

The invention provides a document financial report identification method and device in a natural scene, and the method comprises the steps: receiving a financial report file to be identified, converting the financial report file into an image format, and obtaining a financial report image; performing direction correction on the financial newspaper image, and erasing a covering covering the financial newspaper image; a first preset deep learning algorithm is adopted to detect the position of a table on the financial image, and a table area is cut out; in the table area, detecting unit lines of the table structure by adopting a second preset deep learning algorithm, and dividing cells according to the unit lines; performing centralization processing on each cell to extract information of each cell; standardizing the extracted information according to subject categories, and obtaining an identification result in a unified description form; and according to the business scene, carrying out structured output on the identification result. According to the method provided by the invention, the key information in the financial report document can be accurately detected and identified in a complex environment, and automatic document management and processing are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and information extraction, and in particular to a method and device for identifying document-type financial reports in natural scenes. Background Art

[0002] With the rapid development of information technology, enterprises have an increasing demand for document management and processing. Especially for important documents such as financial reports, how to quickly and accurately identify and extract key information has become a core issue in enterprise information management. However, in natural scenes, due to factors such as light, angle, occlusion, and dense tables, the quality of document images is often poor, which poses a great challenge to document recognition.

[0003] Traditional document recognition methods mainly rely on manual participation, which requires a lot of manpower and time. At the same time, due to the limitations of manual recognition, the accuracy and efficiency of recognition are difficult to meet the needs of enterprises. Therefore, it is of great practical significance to study a technology that can automatically recognize document-type financial reports in natural scenes.

[0004] In recent years, the rapid development of deep learning technology has provided new possibilities for document recognition. Deep learning models can automatically extract features and classify by learning a large amount of sample data, so they have achieved remarkable results in image recognition, natural language processing and other fields. However, applying deep learning to document recognition in natural scenes still faces some challenges, such as how to deal with the influence of light, angle, occlusion of dense table content, and how to improve the accuracy and efficiency of recognition.

[0005] In general, existing table recognition algorithms, whether based on deep learning, such as Microsoft's DETR, Baidu's ppstructure, masterTable and other deep learning algorithms, or based on traditional machine learning, such as camelot, pymupdf and other algorithms, all have poor recognition effects on natural scenes, dense tables, tables close to table lines, etc., and have poor generalization capabilities such as misplaced recognition content, incomplete and incorrect recognition of subject information. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a method and device for document-type financial report recognition in natural scenarios, so as to eliminate or improve one or more defects existing in the prior art, and solve the problems existing in the prior art solutions for financial report recognition, such as poor recognition effect on natural scenarios, dense tables, tables close to table lines, etc., misplaced recognition content, incomplete and erroneous recognition of subject information, and poor generalization ability.

[0007] In one aspect, the present invention provides a method for identifying financial reports of a document type in a natural scenario, the method comprising the following steps:

[0008] Receiving a financial report file to be identified, converting it into an image format, and obtaining a financial report image;

[0009] Directionally correcting the financial report image and erasing a covering object on the financial report image;

[0010] A first preset deep learning algorithm is used to detect the position of the table on the financial image, and a table area is cropped; in the table area, a second preset deep learning algorithm is used to detect the unit lines of the table structure, and cells are divided according to the unit lines; each cell is centrally processed to extract information of each cell;

[0011] The extracted information is standardized according to subject categories to obtain identification results in a unified description form;

[0012] According to the business scenario, the recognition results are output in a structured manner.

[0013] In some embodiments of the present invention, receiving a financial report file to be identified, converting the file into an image format, and obtaining a financial report image further includes:

[0014] The financial report image is preprocessed to enhance image quality; the preprocessing includes one or more operations of increasing brightness, increasing contrast, and improving resolution.

[0015] In some embodiments of the present invention, performing direction correction on the financial report image includes:

[0016] Inputting the financial report image into a pre-trained multi-angle classification model to obtain a tilt angle of the financial report image, and performing coarse-grained correction on the financial report image according to the obtained tilt angle;

[0017] The financial report image is corrected in fine grain by combining perspective transformation and geometric transformation methods.

[0018] In some embodiments of the present invention, erasing the covering on the financial report image includes:

[0019] Input the financial report image into a seal detection model trained based on deep learning to obtain the position of the covering object;

[0020] The mask is erased using a stamp erasure model trained based on a generative adversarial network.

[0021] In some embodiments of the present invention, each cell is centrally processed, including:

[0022] The cell is enlarged, and white space is filled in the upper, lower, left and right sides of the cell to improve the recognition accuracy.

[0023] In some embodiments of the present invention, in the table area, using a second preset deep learning algorithm to detect the unit lines of the table structure, and dividing the cells according to the unit lines, further includes:

[0024] Calculate the plane angle between the center points of each cell to determine the relative direction; calculate the distance between the center points of each cell to determine the relative position; use the angle between the center points to measure the relative direction of each cell on the plane to locate and divide the cells.

[0025] In some embodiments of the present invention, the extracted information is standardized according to subject categories to form a unified description, including:

[0026] A preset text classification algorithm is used to classify and map the extracted information, and standardize descriptions of different subjects into a unified description form.

[0027] In some embodiments of the present invention, the recognition result is output in a structured manner according to the business scenario, including:

[0028] Extract key information from the recognition result and determine the data type; wherein the key information at least includes the account name, amount and date; the data type at least includes text, number and date;

[0029] Mapping the key information to a predefined data model and standardizing it; the predefined data model is a data structure designed according to business requirements, predefining the relationship between various data items;

[0030] After verification, select the output format according to business needs to output the structured data report.

[0031] On the other hand, the present invention also provides a document-type financial report recognition device in a natural scene, and when the device is executed, the steps of any one of the methods mentioned above are implemented, and the device includes:

[0032] A file processing module is used to receive the financial report file to be identified, convert it into an image format, and obtain a financial report image;

[0033] A correction and erasure module, used for correcting the direction of the financial report image and erasing a covering object on the financial report image;

[0034] A table processing module, configured to detect the position of the table on the financial image using a first preset deep learning algorithm and crop a table area; in the table area, detect the unit lines of the table structure using a second preset deep learning algorithm and divide the cells according to the unit lines; and perform centralization processing on each cell to extract information of each cell;

[0035] The information standardization module is used to standardize the extracted information according to subject categories and obtain the recognition results in a unified description form;

[0036] The structured output module is used to output the recognition results in a structured manner according to the business scenario.

[0037] On the other hand, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any one of the methods mentioned above.

[0038] The beneficial effects of the present invention are at least:

[0039] The present invention provides a method and device for document-type financial report recognition in natural scenes, including: receiving a financial report file to be recognized, converting it into an image format, and obtaining a financial report image; performing direction correction on the financial report image, and erasing the covering on the financial report image; using a first preset deep learning algorithm to detect the position of the table on the financial image, and cutting out the table area; in the table area, using a second preset deep learning algorithm to detect the unit lines of the table structure, and dividing the cells according to the unit lines; performing central processing on each cell to extract the information of each cell; standardizing the extracted information according to the subject category, and obtaining the recognition result in a unified description form; and outputting the recognition result in a structured manner according to the business scenario. Compared with the prior art, the method provided by the present invention can better handle the influence of light, angle, occlusion and other problems on financial report recognition, that is, it can detect and recognize key information in financial report documents in complex environments, realize automated document management and processing, improve the accuracy and efficiency of recognition, greatly reduce labor costs, and improve the efficiency of upstream and downstream businesses.

[0040] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0041] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of the present application, and do not constitute a limitation of the present invention. In the drawings:

[0043] Figure 1 The figure is a schematic diagram of the steps of a method for identifying document-type financial reports in a natural scenario in one embodiment of the present invention.

[0044] Figure 2 The figure is a flow chart of a method for identifying document-type financial reports in a natural scenario according to an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0046] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0047] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0048] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0049] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0050] It should be emphasized here that the step marks mentioned below are not intended to limit the order of the steps, but it should be understood that the steps can be executed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be executed simultaneously.

[0051] In order to solve the problems of poor generalization ability of existing financial report recognition technology solutions in natural scenes, dense tables, tables close to table lines, etc., misplaced recognition content, incomplete and incorrect recognition subject information, etc., this paper proposes a document-based financial report recognition method in natural scenes, such as Figure 1 As shown, the method includes the following steps S101 to S105:

[0052] Step S101: receiving a financial report file to be identified, converting it into an image format, and obtaining a financial report image.

[0053] Step S102: Directionally correct the financial report image and erase the covering object on the financial report image.

[0054] Step S103: Use a first preset deep learning algorithm to detect the position of the table on the financial image and crop the table area; in the table area, use a second preset deep learning algorithm to detect the unit lines of the table structure and divide the cells according to the unit lines; perform centralization on each cell to extract the information of each cell.

[0055] Step S104: Standardize the extracted information according to subject categories and obtain recognition results in a unified description form.

[0056] Step S105: Output the recognition results in a structured manner according to the business scenario.

[0057] like Figure 2 As shown, it is a flowchart of the document-type financial report recognition method in a natural scenario.

[0058] In step S101, the user side sends the financial report file to be identified to the identification device server. The financial report file may be in image format, document format, etc. After receiving the financial report file, the server side converts it into an image format and saves it to obtain a financial report image.

[0059] In some embodiments, the financial report image is preprocessed to enhance the image quality. Generally, the preprocessing includes improving brightness, contrast, resolution, etc., and one or more of the preprocessing can be selected according to actual needs. The image can also be further optimized using methods such as image denoising, sharpening, and binarization.

[0060] In step S102, if the financial report image is tilted, its direction is corrected.

[0061] In some embodiments, the financial report image is input into a pre-trained multi-angle classification model to obtain the tilt angle of the financial report image, and the financial report image is coarse-grainedly corrected according to the obtained tilt angle. Preferably, a four-classification algorithm model developed based on the ResNet18 deep learning network architecture is used to detect the tilt angle of the financial report image, wherein the four categories include clockwise 0°, 90°, 180°, and 270°. Exemplarily, if the tilt angle of the financial report image is 90° based on the detection of the four-classification algorithm model, the financial report image is rotated -90° for correction.

[0062] In some embodiments, in addition to using ResNet18, other deep learning models, such as Inception, VGG, MobileNet, etc., or other traditional image processing methods, such as Hough Transform, can also be considered to detect straight lines and edges in the image to determine the direction of the image.

[0063] In some embodiments, perspective transformation and geometric transformation methods can be combined to perform fine-grained correction on financial report images. Perspective transformation can make skewed tables straight by adjusting the angle of the photo, and geometric transformation includes rotating, enlarging, and reducing the image to make the table easier to read.

[0064] After the direction of the financial report image is corrected, it is detected whether there is any covering object on the financial report image, and if so, it is erased. The covering object is usually a watermark, seal, etc.

[0065] In some embodiments, the financial report image is input into a seal detection model obtained based on deep learning training to obtain the position of the obstruction. Preferably, the seal detection model adopts a real-time object detection model developed based on the YOLOv8 deep learning network architecture. Then, the seal erasure model obtained based on the generative adversarial network training is used to erase the obstruction. Preferably, the seal erasure model adopts a DocDiff image restoration model developed based on the generative adversarial network GAN deep learning network.

[0066] In step S103, a first preset deep learning algorithm is used to detect the position of the table on the financial image, and the table area is cropped. In the table area, a second preset deep learning algorithm is used to detect the unit lines of the table structure, and the cells are divided according to the unit lines; each cell is centralized to extract the information of each cell.

[0067] Preferably, the first preset deep learning algorithm uses a real-time object detection model developed based on the YOLOv8 deep learning network architecture to detect the specific location of the table and then crop the table area.

[0068] For dense, wired tables close to table lines, preferably, the second preset deep learning algorithm adopts an end-to-end table structure detection model based on CenterNet to better extract the cell lines of the table structure, divide the cells according to the cell lines, and identify the cells one by one, thereby improving the accuracy of dense table recognition. Among them, CenterNet is an object detection algorithm based on key point detection, which locates the object by detecting the center point and width and height attributes of the object. The table structure refers to the layout and structure of rows, columns, and cells in the table.

[0069] In some embodiments, for table recognition close to table lines, the cells are enlarged and filled with blank spaces at the top, bottom, left and right sides to perform centralization processing to improve recognition accuracy.

[0070] In some embodiments, when determining the cell position, it also includes: calculating the plane angle between the center points of each cell to determine the relative direction; calculating the distance between the center points of each cell to determine the relative position; using the angle between the center points, such as the cosine angle, to measure the relative direction of each cell on the plane to locate and divide the cells.

[0071] In step S104, the extracted information is standardized according to subject categories to obtain recognition results in a unified description form.

[0072] In some embodiments, in response to changes in the subject standards of financial reports and inconsistent descriptions of financial report subjects of different companies, the present invention uses a text classification algorithm, such as FasText, to classify and map the extracted information, standardize different subject descriptions into a unified description form, and speed up the efficiency of downstream businesses. Among them, FastText is a text classification algorithm based on deep learning that can convert text data into vector representations and use these vectors for fast text classification.

[0073] In step S105, key information is extracted from the recognition result and the data type is determined, wherein the key information at least includes the account name, amount and date, and the data type at least includes text, number and date.

[0074] Map key information to predefined data models and standardize them. For example, dates in various formats are unified into the "YYYY-MM-DD" format. Predefined data models are data structures designed according to business needs, and predefine the relationship between each data item. The data structure is usually a table, JSON, database table, etc., which defines the relationship between each data item, such as the relationship between account and amount, the relationship between date and account, etc.

[0075] After verification, select the output format to output the structured data report according to business needs. The output format can be CSV file, Excel table, database table, JSON or XML format, etc.

[0076] Corresponding to the method for identifying document-type financial reports in natural scenes, the present invention also provides a device for identifying document-type financial reports in natural scenes. When the device is executed, the steps of the method for identifying document-type financial reports in natural scenes can be implemented. The device includes:

[0077] The file processing module is used to receive the financial report file to be identified, convert it into an image format, and obtain a financial report image.

[0078] The correction and erasure module is used to correct the direction of the financial report image and erase the covering objects on the financial report image.

[0079] The table processing module is used to use a first preset deep learning algorithm to detect the position of the table on the financial image and crop the table area; in the table area, use a second preset deep learning algorithm to detect the unit lines of the table structure and divide the cells according to the unit lines; and perform centralization processing on each cell to extract the information of each cell.

[0080] The information standardization module is used to standardize the extracted information according to subject categories and obtain the recognition results in a unified description form.

[0081] The structured output module is used to output the recognition results in a structured manner according to the business scenario.

[0082] The present invention also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the document-type financial report recognition method in a natural scenario are implemented.

[0083] Corresponding to the above method, the present invention also provides a device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the device implements the steps of the method described above.

[0084] The embodiment of the present invention also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the aforementioned edge computing server deployment method are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0085] In summary, the present invention provides a method and device for document-type financial report recognition in natural scenes, including: receiving a financial report file to be recognized, converting it into an image format, and obtaining a financial report image; performing direction correction on the financial report image, and erasing the covering on the financial report image; using a first preset deep learning algorithm to detect the position of the table on the financial image, and cropping the table area; in the table area, using a second preset deep learning algorithm to detect the unit lines of the table structure, and dividing the cells according to the unit lines; performing central processing on each cell to extract the information of each cell; standardizing the extracted information according to the subject category, and obtaining the recognition result in a unified description form; and outputting the recognition result in a structured manner according to the business scenario. Compared with the prior art, the method provided by the present invention can better handle the influence of light, angle, occlusion and other problems on financial report recognition, that is, it can detect and recognize key information in financial report documents in complex environments, realize automated document management and processing, improve the accuracy and efficiency of recognition, greatly reduce labor costs, and improve the efficiency of upstream and downstream businesses.

[0086] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0087] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0088] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0089] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for identifying financial reports in natural scenarios, characterized in that: The method comprises the following steps: Receiving a financial report file to be identified, converting it into an image format, and obtaining a financial report image; Directionally correcting the financial report image and erasing a covering object on the financial report image; A first preset deep learning algorithm is used to detect the position of the table on the financial image, and a table area is cropped; in the table area, a second preset deep learning algorithm is used to detect the unit lines of the table structure, and cells are divided according to the unit lines; each cell is centrally processed to extract information of each cell; The extracted information is standardized according to subject categories to obtain identification results in a unified description form; According to the business scenario, the recognition results are output in a structured manner.

2. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: Receiving the financial report file to be recognized, converting it into an image format, and obtaining a financial report image, further comprising: The financial report image is preprocessed to enhance image quality; the preprocessing includes one or more operations of increasing brightness, increasing contrast, and improving resolution.

3. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: Direction correction is performed on the financial report image, including: Inputting the financial report image into a pre-trained multi-angle classification model to obtain a tilt angle of the financial report image, and performing coarse-grained correction on the financial report image according to the obtained tilt angle; The financial report image is corrected in fine grain by combining perspective transformation and geometric transformation methods.

4. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: Erasing any covering over the financial report image, including: Input the financial report image into a seal detection model trained based on deep learning to obtain the position of the covering object; The mask is erased using a stamp erasure model trained based on a generative adversarial network.

5. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: Each cell is centrally processed, including: The cell is enlarged, and white space is filled in the upper, lower, left and right sides of the cell to improve the recognition accuracy.

6. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: In the table area, a second preset deep learning algorithm is used to detect the unit lines of the table structure, and cells are divided according to the unit lines, further comprising: Calculate the plane angle between the center points of each cell to determine the relative direction; calculate the distance between the center points of each cell to determine the relative position; use the angle between the center points to measure the relative direction of each cell on the plane to locate and divide the cells.

7. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: The extracted information is standardized according to subject categories to provide a unified description format, including: A preset text classification algorithm is used to classify and map the extracted information, and standardize descriptions of different subjects into a unified description form.

8. The method for identifying financial reports of documents in natural scenes according to claim 1, characterized in that: According to the business scenario, the recognition results are output in a structured manner, including: Extract key information from the recognition result and determine the data type; wherein the key information at least includes the account name, amount and date; the data type at least includes text, number and date; Mapping the key information to a predefined data model and standardizing it; the predefined data model is a data structure designed according to business requirements, predefining the relationship between various data items; After verification, select the output format according to business needs to output the structured data report.

9. A document-type financial report recognition device in a natural scene, characterized in that: When the device is executed, the steps of the method according to any one of claims 1 to 8 are implemented, and the device comprises: A file processing module is used to receive the financial report file to be identified, convert it into an image format, and obtain a financial report image; A correction and erasure module, used for correcting the direction of the financial report image and erasing a covering object on the financial report image; A table processing module, configured to detect the position of the table on the financial image using a first preset deep learning algorithm and crop a table area; in the table area, detect the unit lines of the table structure using a second preset deep learning algorithm and divide the cells according to the unit lines; and perform centralization processing on each cell to extract information of each cell; The information standardization module is used to standardize the extracted information according to subject categories and obtain the recognition results in a unified description form; The structured output module is used to output the recognition results in a structured manner according to the business scenario.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images