Complex table structured processing method suitable for A-stock financial statements

Abstract

The invention aims to solve the technical problems that the improvement of the data production efficiency is seriously influenced by the defects of the pdf analysis service provided by the existing C + + in combination with an xpdf library, and the maintenance cost is very high. In order to solve the technical problem, the technical scheme of the invention provides a complex table structured processing method suitable for the A-stock financial statement. According to the method, the Python library and the pdfplug library are introduced, so that the development threshold and the cost are reduced; by optimizing the use of the pdfplumber library, the processing efficiency of complex tables and large-scale data is improved; by using an efficient development environment of Python and a flexible API (Application Program Interface) of pdfplug, the period of customized development is shortened; by providing a highly customized table analysis function, the flexibility of the system is enhanced.

Description

Technical Field

[0001] The present invention relates to a complex table structured processing method, and in particular to a structured processing method suitable for A-share financial statements. Background Art

[0002] A-share financial statements contain key financial information of listed companies and are an important basis for investors, regulators and other stakeholders to evaluate the company's financial status and operating performance. However, A-share financial statements are usually presented in complex tables, including multiple financial indicators, data from different time periods, and multiple categories of financial information. The structural complexity of these reports makes it very difficult to manually process these data, especially when dealing with a large number of reports.

[0003] The structured processing of financial statement data usually involves converting these unstructured or semi-structured data into machine-readable structured data for further analysis, storage and query. This process usually requires consideration of multiple dimensions, including table structure recognition, text parsing, data cleaning, and data format conversion. Due to the complexity of the content and rich sources, the processing difficulty is also very high, requiring the comprehensive application of multiple technologies and technical creation. With the development of technologies such as big data analysis and artificial intelligence, more and more research and development work is focused on how to efficiently and accurately achieve structured processing of financial statements.

[0004] The existing technical solutions have obvious stages with the development of technology. The earliest is the manual production stage: PDF table data extraction initially mainly relies on manual operations on the input system, which is not only inefficient but also prone to errors. With the growth of business volume, manual operations can hardly meet the needs of large-scale data processing.

[0005] To improve efficiency, technicians in this field developed a PDF parsing service using C++ combined with the xpdf library. Figure 1 The process shown can efficiently convert PDF documents into HTML format. Business personnel can flexibly configure different table matching rules based on specific table features. The specific processing flow is as follows:

[0006] Step 1: PDF document loading and preprocessing;

[0007] Step 101: Load PDF document: The system first loads the PDF document to be parsed from the database or file system, usually the PDF file of the A-share financial report. This step includes reading the file from the specified directory and performing basic loading preparation;

[0008] Step 102: Document format verification: After loading is complete, the system will verify the format of the PDF document to ensure that the document is not damaged, has a normal number of pages, and conforms to the standardized PDF format. This stage will handle issues such as file corruption, missing content, or abnormal file format. If necessary, the PDF will be repaired or the user will be reminded to correct the format;

[0009] Step 103: Page separation and basic processing: The PDF document is decomposed into multiple independent pages according to the number of pages, which is convenient for subsequent page-by-page analysis. At the same time, the system will perform preliminary content detection to separate different elements such as text, pictures, tables, etc. in the page to prepare for subsequent table analysis.

[0010] Step 2: Table detection and positioning;

[0011] Step 201: Table area detection: The system analyzes the graphic elements (such as lines, frames, etc.) in the PDF to preliminarily detect the area on the page that may contain a table. By identifying horizontal and vertical lines, the outline of the table is determined and the boundaries of the table are marked;

[0012] Step 202: Merge and organize line information: The system will then merge and organize the line information of these table areas. In particular, when the table lines may be broken or discontinuous, the system will use intelligent algorithms to complete and repair them to ensure that all table lines are complete and closed;

[0013] Step 203: Generate table intersection points: After obtaining the complete table lines, the system will calculate the intersection points of each horizontal line and vertical line; these intersection points will be used to generate the cell layout of the table to ensure that subsequent content can be correctly allocated to each cell;

[0014] Step 204: Preliminary generation of table cell frames: Based on the intersection points, the system will delineate the cell frames of each table, generate row and column coordinates, and ensure that the overall structure of the table is restored; this step completes the preliminary frame division of the table and prepares for parsing the content in the table.

[0015] Step 3: Complex table parsing:

[0016] Step 301: Cell generation and row and column list creation: The system creates a row and column list based on the cell frame generated by the intersection point, and identifies the row and column position of each cell. On this basis, the system also processes cells that span rows and columns, such as the merged cells commonly seen in financial statements; this stage ensures the accurate restoration of the table structure;

[0017] Step 302: Filling in character elements: The system will go through each cell one by one, extract character elements (text content) from the PDF, and fill them into the corresponding cells; the text content is extracted based on the coordinate information in the PDF to ensure that the character elements are accurately positioned;

[0018] Step 303: Character sorting and adjustment: To ensure that the text content in the table is presented in a natural reading order, the system will sort the characters in each cell; character sorting is based on the vertical and horizontal coordinates, arranging the content from top to bottom and from left to right to ensure that the text conforms to the reading logic; if necessary, the system will also fine-tune the text, such as solving the problem of overlapping characters or over-segmentation;

[0019] Step 304: Parsing of off-table text and other elements: In addition to the table content, the system also processes text, images or other elements outside the table; especially in A-share financial statements, notes, titles or footer information outside the table also need to be extracted and stored; after the table is parsed, the system will parse and organize these additional contents to ensure the integrity of the financial statement information.

[0020] Step 4: Data verification and optimization:

[0021] Step 401: Checking consistency between table and text: After the content extraction is completed, the system will check the consistency between the data and text in the table to ensure that the parsed table structure matches the actual data; for example, the system will check whether the cell content is correctly filled and whether the cells across rows and columns are consistent with the surrounding data;

[0022] Step 402: Automated data correction: The system uses a pre-trained machine learning model to automatically correct possible parsing errors; for example, during the table segmentation process, if the content of some cells is misjudged or omitted, the system will correct it through the learning model to ensure the accuracy of the table structure and data content;

[0023] Step 403: Table optimization processing: The system will further optimize the parsed table structure, especially for complex table structures (such as nested tables and borderless tables); the optimization process includes adjusting cell sizes, merging duplicate cells, repairing missing cells, etc., so that the table structure is more reasonable and neat.

[0024] Step 5: Data output and storage:

[0025] Step 501: Processing and splicing cross-page tables: For cross-page tables, the system will splice the table contents; the system will merge the table contents from different pages into a complete table based on the table boundary information; in this way, even if the table is paginated in the PDF, the structured data in the final output will still remain consistent;

[0026] Step 502: Data formatting and exporting: After parsing and verifying the table, the system will export the structured data into multiple formats for subsequent use by the user; common export formats include CSV, JSON, Excel tables, and can even be written directly into the database;

[0027] Step 503: Data storage and backup: The system stores the parsed data in a specified folder or database and backs it up as needed; the data storage method is flexible and can be stored locally or uploaded to the cloud for easy sharing and access.

[0028] Step 6: Data display and visualization;

[0029] Step 601: HTML format generation and display: Finally, the system can generate the parsed data in HTML format for front-end display; this step can intuitively present the parsed table data in the system interface, and users can directly view the parsing results in the browser;

[0030] Step 602: Data visualization: The system supports visualization of structured data and generates charts or graphical interfaces to better display the trends and distribution of financial data; users can choose different visualization methods according to their needs, such as line charts, pie charts, bar charts, etc., to further analyze the data.

[0031] The table extraction service automatically performs table extraction based on these HTML content and table matching rules to achieve automatic parsing and extraction of table content. Although xpdf performs well in text extraction, it has limited support for special complex tables and wireless tables. At the same time, since C++ has a certain development threshold, custom development of tables requires a long development cycle, and business tables cannot be quickly rolled out. Specifically, it has the following disadvantages:

[0032] 1. High cost: xpdf is developed based on C++. Although it has powerful performance, its development and maintenance costs are high. The C++ language itself is relatively complex, and developers need to have high programming skills. The debugging and maintenance process of the code is complicated, which increases the manpower and time costs. In addition, customized development requires a deep understanding of the underlying implementation of xpdf, which further increases the threshold and cost of development.

[0033] 2. Low efficiency: In the existing C++ development environment, xpdf is relatively inefficient when processing complex or wireless tables, especially when customized processing is required. The development cycle is long and it is difficult to quickly adapt to changes in business needs. Although pdfplumber is easier to use in the Python environment, its efficiency is still limited by its single-threaded operation mode when processing large-scale data, which may lead to performance bottlenecks in high-frequency data processing scenarios.

[0034] 3. Long development cycle: Due to the complexity of C++ development and customization requirements, the development cycle of using xpdf for table parsing and text extraction is long. Developers need to invest a lot of time in writing and debugging code, especially when faced with diverse financial report formats. The development cycle will be further extended, resulting in slower project delivery and an inability to quickly respond to business needs.

[0035] 4. Lack of flexibility: Although the existing xpdf tool is powerful, it lacks flexibility, especially when dealing with financial statements of different formats and structures, a lot of manual adjustments and configurations are required to obtain accurate parsing results. Although pdfplumber improves the flexibility of development, it still requires a lot of manual intervention in the processing of complex tables and specific formats, and cannot be fully automated, which reduces the flexibility and efficiency of the system. Summary of the invention

[0036] The technical problem to be solved by the present invention is that the defects of the existing PDF parsing service provided by the C++ combined with the xpdf library seriously affect the improvement of the efficiency of data production and bring very large maintenance costs.

[0037] In order to solve the above technical problems, the technical solution of the present invention is to provide a complex table structured processing method applicable to A-share financial statements, which is characterized by comprising the following steps:

[0038] Step 1: Task notification and configuration loading, further including the following steps:

[0039] Step 101: The system receives a task notification from a task center through an HTTP interface, wherein the task notification includes at least relevant information and key parameters of a PDF file to be parsed;

[0040] Step 102: After receiving the task notification, the system verifies the key parameters, saves the task notification that passes the verification to a file and memory, and adds it to the task pool;

[0041] Step 2: Task collection and deduplication, further including the following steps:

[0042] Step 201: The system starts the task receiving thread, regularly obtains the configuration required for the task from the task pool and prepares to receive the task. The task receiving thread pulls the configuration required for the task and checks whether the system is qualified to receive the task. Only when the system is qualified to receive the task can the task be officially received;

[0043] Step 203: The system checks the deduplication table to ensure that the task will not be processed repeatedly;

[0044] Step 204: If the system detects a task primary key conflict, which means that the current task has been pushed repeatedly, the current task will be deleted; if a non-primary key conflict error occurs, the system will still save the task detail file and try to process it;

[0045] Step 3: Automating the parsing thread and PDF file downloading further includes the following steps:

[0046] Step 301: After the task is received, the system starts the automatic parsing thread. Each automatic parsing thread is responsible for loading the unfinished task detail file and marking it as busy to prevent repeated processing;

[0047] Step 302: the automated parsing thread extracts the relevant PDF file download path from the task, and the main process starts downloading the corresponding PDF file;

[0048] Step 4: PDF file parsing and data processing, including the following steps:

[0049] Step 401: Each automatic parsing thread starts a subprocess, which is responsible for parsing a specific PDF file, and specifically includes the following steps:

[0050] Step 4011: The subprocess locates the PDF file according to the task configuration and identifies the areas and tables that need to be parsed;

[0051] Step 4012: PDF form extraction, further comprising the following steps:

[0052] Step 40121: Use the extract_table() method provided by pdfplumber to identify the data in the wired table: If the table data cannot be extracted, proceed directly to step 40127; otherwise, proceed to the next step;

[0053] Step 40122: Collect all line segments of the current page and perform preprocessing to merge and splice the line segments, thereby extracting complete horizontal and vertical lines;

[0054] Step 40123: By analyzing the horizontal and vertical lines obtained in the previous step, the intersections of all line segments are accurately found, and the framework of all cells is constructed using the intersection information and the corresponding line segment data of the horizontal and vertical lines;

[0055] Step 40124: grouping the cells according to the coherence of the cells to obtain a table list, each table in the table list contains detailed cell information;

[0056] Step 40125: According to the positional relationship between each cell and its corresponding horizontal and vertical lines, the row and column positions of each cell are accurately determined, thereby establishing a complete table structure;

[0057] Step 40126: Fill the text content on the page into the corresponding cells in the table structure, so as to achieve complete extraction of the PDF table content, and then proceed to step 4013;

[0058] Step 40127: Obtain basic information of the PDF file according to the pdfplumber library and generate text lines;

[0059] Step 40128: Locate the page and location area where the business table is located according to the business configuration;

[0060] Step 40129: Obtain the header and footer of the page where the business table obtained in the previous step is located and remove them, and cut out the text data belonging to the table area based on the location area where the business table obtained in the previous step is located;

[0061] Step 40130: Obtain the text information corresponding to the text data obtained in the previous step and all the phrases word corresponding to the text information through the pdfplumber library, and obtain the table data rows data_rows and column intervals column_intervals:

[0062] Step 40131: Generate a two-dimensional table according to the data rows data_rows and the column intervals column_intervals;

[0063] Step 40132: fill all phrases word into the two-dimensional table and proceed to step 4013;

[0064] Step 4013: extracting non-derived fields from the PDF file;

[0065] Step 4014: Obtain complete derived fields by calling the derived service;

[0066] Step 4015: insert the data parsed in steps 4012, 4013 and 4015 into the system data table pdfjx and the system data table newsadmin respectively;

[0067] Step 4016: The system performs front-end verification on the inserted data to ensure its correctness and completeness;

[0068] Step 4017: After the transaction submission is successfully verified, the system will submit the transaction inserted into the system data table newsadmin;

[0069] Step 402: Result feedback and task closing, further including the following steps:

[0070] Step 4021: After the PDF file is parsed, the system feeds back the task processing result to the task center through the HTTP interface to notify the task processing status;

[0071] Step 4022: After the subprocess analysis is completed, the system closes the subprocess and releases resources; then, the task analysis is officially completed and marked as completed;

[0072] Step 5: Task record update and deduplication, further including the following steps:

[0073] Step 501: The system updates the processing status of the task according to the records in the deduplication table: if the task center notifies that the task has been completed, the system removes the saved task notification information;

[0074] Step 502: After the analysis is completed, the system inserts the task log into the main table and the deduplication table, and generates an unsubmitted task detail file to ensure the traceability of the task status.

[0075] Preferably, in step 203, the system inserts the new task into the deduplication table record; writes the unsubmitted task detail file; commits the transaction, and saves the unsubmitted task into the file.

[0076] Preferably, in step 40130, the table data rows data_rows and the column intervals column_intervals are obtained by the following method:

[0077] Step 401301: sort all the word phrases according to bottom;

[0078] Step 401302: determine the error threshold according to the page size and the text size, put all the phrases word whose bottom is within the error threshold into a list, and generate table data rows data_rows;

[0079] Step 401303: traverse all phrases word in the data row data_rows, dynamically generate and expand column intervals according to x0 and x1 of the phrase word, and obtain column interval column_intervals.

[0080] Preferably, in step 40132, the filling methods are divided into 7 according to the scenarios, including:

[0081] Scenario 1: If the center point of a word is between the left and right edges of a column, the word belongs to this column.

[0082] Scenario 2: If the coordinates of the center point of a word are within a certain threshold from the left or right edge of a column, the word belongs to this column;

[0083] Scenario 3: If the left coordinate of the phrase word is close enough to the left edge of a column, the phrase word belongs to this column;

[0084] Scenario 4: If the right coordinate of the phrase word is close enough to the right edge of a column, the phrase word belongs to this column;

[0085] Scenario 5: If the left coordinate of the phrase word is less than the left edge of a column and the right coordinate is greater than the right edge of the column, then the phrase word belongs to this column;

[0086] Scenario 6: The left coordinate of the phrase word is between the left edge and the right edge of its closest left column, and the right coordinate of the phrase word is smaller than the left edge of its closest right column, then the phrase word belongs to the closest left column;

[0087] Scenario 7: The right coordinate of the phrase word is between the left edge and the right edge of its closest right column, and the left coordinate of the phrase word is greater than the right edge of its closest left column, then the phrase word belongs to the closest right column.

[0088] In order to solve the defects of the existing technical solutions, the technical solution disclosed in the present invention is:

[0089] 1. The introduction of Python and pdfplumber libraries has lowered the development threshold and cost. The simplicity of the Python language and its broad community support enable developers to quickly get started and effectively carry out development work, reducing dependence on high-end developers, thereby significantly reducing labor costs and time costs.

[0090] 2. By optimizing the use of the pdfplumber library, the processing efficiency of complex tables and large-scale data is improved. By introducing multi-threading or parallel processing technology, the present invention can process a large amount of financial statement data more quickly, meet the needs of high-frequency processing scenarios, and improve the overall performance of the system.

[0091] 3. By utilizing Python's efficient development environment and pdfplumber's flexible API interface, the customized development cycle is shortened. Developers can more quickly define and implement the parsing logic of complex tables, thereby speeding up project delivery, responding quickly to business needs, and ensuring that business tables can be launched and updated in a timely manner.

[0092] 4. Enhanced system flexibility by providing highly customized table parsing capabilities. The flexible API interface of the pdfplumber library enables developers to easily adjust parsing strategies according to different financial report formats, thereby achieving a higher degree of automation, reducing manual intervention, and ensuring that the system can cope with diverse report formats and changing business needs.

[0093] Compared with the existing technical solutions, the core technical elements that reflect the innovativeness of the technical solution disclosed in the present invention include the following points, which bring significant beneficial effects to the structured processing of complex tables including A-share financial statements:

[0094] 1. High-precision processing technology for complex table analysis:

[0095] The present invention is based on the high-precision table parsing function of Python and pdfplumber library, which can effectively process complex table layouts, such as cells across rows and columns, nested tables, wireless tables, etc. The technical point lies in its unique table parsing algorithm and strategy, which can cope with various financial report formats, and automatically identify and process complex table structures, significantly improving the accuracy and consistency of data extraction. Among them, the core technical points further included in complex table processing include:

[0096] 1.1 Supplementary table line function:

[0097] For non-completely closed tables with missing table lines on the left and right sides, or missing horizontal lines in the first and last rows due to page changes, if the table is directly formed based on the line segments of the current page, some rows or columns of the table will be missing. Therefore, in order to extract a complete table, before forming a table based on line segments, we need to insert horizontal and vertical lines to make it a closed table. Whether the vertical lines need to be supplemented is determined based on the position information of the horizontal and vertical lines. When the difference between the x0 of the horizontal line and the x0 of the first vertical line exceeds a certain threshold or the difference between the x1 of the horizontal line and the x1 of the last vertical line exceeds a certain threshold, it is necessary to supplement the vertical lines. Whether the horizontal lines need to be supplemented is also determined based on similar logic. When the difference between the y0 of the vertical line and the y0 of the first horizontal line exceeds a certain threshold or the difference between the y1 of the vertical line and the y1 of the last horizontal line exceeds a certain threshold, it is necessary to supplement the horizontal lines. For the case of missing parts of the horizontal lines, the present invention will first level the horizontal lines according to the position alignment information of the horizontal lines, and then insert two vertical lines.

[0098] 1.2 Cleaning excess lines function:

[0099] For some PDFs, its financial notes are wrapped by a large rectangular frame. Inside the rectangular frame are the financial tables that need to be captured and parsed, and the external large rectangular frame will affect our capture of the tables. Because the large rectangular frame wraps all the tables, all the tables on the page will be judged as a continuous table, and the tables cannot be separated, resulting in incorrect table data information being parsed. Therefore, in order to better parse this type of financial table, we need to clean up the external rectangular frame. The present invention first forms all rectangular cells according to the line segments, and then finds the outermost large rectangular frame and removes it through the relative position information between the rectangular frames. There are also some small miscellaneous lines in some PDF tables, which will affect our line filling and table formation logic, so before forming the table, we need to remove these miscellaneous lines based on information such as the position and length of the line segments.

[0100] 1.3 Enhanced table line detection function:

[0101] Some PDF tables have special table lines, and their line segments cannot be directly extracted using pdfplumber. For example, if the table lines in the table are picture elements, the table will fail to be generated. For this reason, the present invention adds the parsing of picture-type table lines and expands the parsing of PDF line protocols, so that the originally unrecognizable line segments can be used in the table generation, thereby solving the problem of table generation failure.

[0102] 1.4 Character cleaning function in the table:

[0103] Due to the page layer of some PDFs, when we extract the text content of the page, we will get some characters in the hidden layer, which makes the extracted text content inconsistent with what we see with our eyes. For this reason, we increase the protocol range of the clipping area detection, and use the switch to independently select whether to remove the characters in the hidden layer. Some contents in the table will have some corner annotations, which are usually unnecessary. Therefore, the present invention also determines which are the corner mark characters by judging the relative size and position of the characters in the table, and also uses the switch to independently select whether to remove the corner marks.

[0104] 2. Customized parsing framework based on Python and pdfplumber:

[0105] The present invention uses Python and pdfplumber library to build a highly customized table parsing framework. The framework allows developers to flexibly define parsing logic through a concise API interface, thereby quickly adapting to different business needs and report formats. The table parsing framework has flexibility and scalability, especially its customization ability and rapid development advantages. The table parsing framework lowers the development threshold and shortens the development cycle, so that complex table parsing tasks can be completed in a short time.

[0106] 3.Automated table parsing and template matching mechanism:

[0107] The present invention introduces an automated table parsing and template matching mechanism, which can intelligently identify the report format and apply the most suitable parsing strategy to achieve highly automated table data extraction. This reduces manual intervention and improves the automation level of the system. The present invention provides an automated and intelligent parsing mechanism, especially learning and template matching technology based on historical data. These technologies significantly improve the parsing accuracy and efficiency of the system and are an important innovation of the present invention.

[0108] 4. Multi-threaded parallel processing and data flow optimization technology:

[0109] In order to cope with the processing requirements of large-scale financial statements, the present invention introduces multi-threaded parallel processing and data flow optimization technology, which significantly improves the processing efficiency and scalability of the system. This technology ensures that the system can still maintain high performance and stability when processing large amounts of data. The present invention has a parallel processing architecture design and a data flow control mechanism, which can effectively reduce processing time and improve the scalability of the system. These features make the present invention have obvious advantages in high-frequency processing scenarios.

[0110] Since the technical solution disclosed in the present invention has the above core technical elements, the technical solution disclosed in the present invention has the following beneficial effects compared with the existing technical solutions:

[0111] 1. The ability to handle complex tables is significantly enhanced:

[0112] Although the existing technology can handle general table and text extraction, the accuracy and completeness of parsing are poor when facing complex table layouts, such as cells across rows and columns, nested tables and wireless tables. The present invention can effectively handle these complex table structures through optimized table parsing algorithms and intelligent parsing strategies, ensuring the integrity and consistency of data, and greatly improving the accuracy of parsing.

[0113] 2. The development cycle is greatly shortened:

[0114] The existing technology, especially the xpdf tool based on C++, has a high development threshold and a long development cycle. The present invention greatly simplifies the development process by introducing Python language and pdfplumber library. Python's concise syntax and active community support, combined with pdfplumber's flexible API interface, enable developers to implement customized table parsing logic more quickly, thereby significantly shortening the development cycle and quickly responding to business needs.

[0115] 3. Enhanced system flexibility and scalability:

[0116] The existing technology requires a lot of manual adjustments and configurations when dealing with diverse financial report formats, and lacks flexibility and scalability. The present invention builds a highly customized parsing framework and an automated template matching mechanism, which enables the system to flexibly respond to different report formats and changing business needs, while supporting the rapid expansion of business tables, reducing manual intervention and improving the flexibility and scalability of the system.

[0117] 4. Significant improvement in processing efficiency:

[0118] When facing large-scale data processing, the pdfplumber library in the prior art has certain advantages, but the single-threaded operation mode may cause performance bottlenecks in high-frequency processing scenarios. The present invention greatly improves the processing efficiency of the system by introducing multi-threaded parallel processing and data flow optimization technology, enabling it to efficiently process a large amount of financial statement data and meet the needs of high-frequency data processing scenarios.

[0119] 5. Reduced development and maintenance costs:

[0120] The xpdf tool in the prior art has a high maintenance cost due to the high development threshold of C++. The present invention reduces the difficulty and cost of development and maintenance by adopting Python, a language with a low development threshold, combined with the pdfplumber library. At the same time, Python's active community support and rich third-party library resources also help to reduce maintenance costs and improve development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] Figure 1 It shows the existing PDF parsing service process;

[0122] Figure 2 The initialization process of the invented system is illustrated;

[0123] Figure 3 It illustrates the specific modules of loading configuration, processing unfinished tasks, deduplication mechanism, and parallel processing of tasks;

[0124] Figure 4The overall architecture of the system of the present invention is illustrated;

[0125] Figure 5 It shows the specific steps of task notification and configuration loading;

[0126] Figure 6 It shows the specific steps of task collection and deduplication;

[0127] Figure 7 The cell properties are shown;

[0128] Figure 8 It shows the corresponding attributes of the text line;

[0129] Fig. 9 It indicates the corresponding attributes of the phrases;

[0130] Fig.10 A scene illustrating the filling method. DETAILED DESCRIPTION

[0131] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0132] Based on the existing technology, the technical solution disclosed in the present invention aims to solve the following key technical problems:

[0133] 1. Insufficient processing capabilities for special complex forms and wireless forms:

[0134] Technical problems to be solved: When dealing with simple text extraction, it is also necessary to deal with special complex wired tables and wireless tables, to realize the merging of cells across rows and columns, nested tables and wireless tables, etc., to improve the accuracy of data extraction, and even solve the problem of possible data loss or misalignment.

[0135] 2. High threshold and long cycle of custom development:

[0136] Technical issues to be solved: Simplify the development process and shorten the development cycle; using concise and clear API interfaces, developers can flexibly define table parsing logic and achieve highly customized table processing; using a development language with a lower threshold, the team can quickly master the tools and carry out development, thereby significantly shortening the customized development cycle of business forms and accelerating the automated publishing process of business forms.

[0137] 3. Difficulty in rapid expansion of business forms:

[0138] Technical problems to be solved: Development requires simple and flexible languages ​​and tools to support rapid expansion of business tables and rapid expansion of table parsing capabilities; strong open source support to more conveniently obtain the required resources and help, efficiently respond to the needs of different business scenarios, and achieve rapid expansion of business tables.

[0139] 4. The contradiction between development efficiency and maintenance cost:

[0140] Technical issues to be resolved: While ensuring that the development process can efficiently implement development and deployment, it is also necessary to reduce maintenance costs through selection and solution design, especially when facing the ever-changing report format and complexity, the code is easier to understand and maintain; for possible problems, it is necessary to be able to quickly resolve and seek support for solutions, and to iterate and repair.

[0141] After research and comparison, we have chosen Python combined with the pdfplumber library as a new PDF parsing tool. pdfplumber not only supports high-precision text extraction, but also effectively handles complex table layouts. Through the API interface it provides, you can flexibly define parsing logic, achieve highly customized table parsing requirements, and improve the automatic publishing rate and accuracy of business tables. At the same time, due to its concise syntax and active community support, the overall development efficiency is greatly improved, and the business can be rapidly expanded.

[0142] The overall system process is divided into two parts: initialization and execution process.

[0143] The initialization process of the system of the present invention is as follows Figure 2 As shown in the figure, a series of steps are used to ensure the efficient and stable operation of the system. The specific modules include loading configuration, processing unfinished tasks, deduplication mechanism, and parallel processing of tasks. Figure 3 The above diagram ensures the persistence and reliability of the system. The registration, keep-alive and reporting of business data of the service further improve the maintainability and monitorability of the system.

[0144] The execution process of the system of the present invention ensures that tasks can be efficiently received, analyzed and fed back through reasonable thread and process management. The overall architecture is as follows: Figure 4 The system of the present invention makes full use of multithreading and subprocesses to realize concurrent processing of tasks, and combines the accurate analysis, deduplication mechanism and error handling logic of PDF files to make the whole process efficient and robust. It is the core of the whole system and embodies the processing method of complex tables in the technical solution, solving the problems of low accuracy in complex table processing, large consumption of computing resources and low business participation. It specifically includes the following steps:

[0145] Step 1: Task notification and configuration loading, such as Figure 5As shown, the following steps are included:

[0146] Step 101: Task notification interface receives:

[0147] The system receives task notifications from the task center through the HTTP interface; the notifications contain relevant information and key parameters of the PDF files to be parsed;

[0148] Step 102: Request key parameter verification:

[0149] After receiving the task notification, the system will verify the key parameters of the task to ensure the integrity and correctness of the task content and prevent erroneous or invalid tasks from entering the parsing process;

[0150] Step 103: Save task notification:

[0151] After verification, the task notification will be saved to the file and memory as the basis for subsequent task processing, and added to the task pool for further processing.

[0152] Step 2: Task collection and deduplication, such as Figure 6 As shown, the following steps are included:

[0153] Step 201: Task collection thread starts:

[0154] The system starts the task collection thread, regularly obtains the configuration required for the task from the task pool and prepares to collect the task;

[0155] Step 202: Receive task configuration check:

[0156] The task collection thread will pull the configuration required for the task and check whether the system is qualified to collect the task; only when the processing conditions are met can the task be officially collected;

[0157] Step 203: Task deduplication processing;

[0158] The system checks the deduplication table to ensure that tasks are not processed repeatedly. The specific process is as follows: insert the new task into the deduplication table record; write the uncommitted task details file; commit the transaction and save the uncommitted tasks to the file;

[0159] Step 204: Task repetition processing logic:

[0160] If a task primary key conflict is detected, it means that the task has been pushed repeatedly, and the task will be removed; if a non-primary key conflict error occurs, the system will still save the task detail file and try to process it.

[0161] Step 3: Automated parsing thread and PDF download, including the following steps:

[0162] Step 301: Start the automated parsing thread:

[0163] After the task is received, the system will start the automatic parsing thread; each automatic parsing thread is responsible for loading the unfinished task details file and marking it as busy to prevent duplicate processing;

[0164] Step 302: The main process downloads the PDF file:

[0165] The automated parsing thread extracts the relevant PDF file download path from the task, and the main process starts downloading the corresponding PDF file.

[0166] Step 4: PDF parsing and data processing, including the following steps:

[0167] Step 401: Start a subprocess to perform PDF parsing:

[0168] Each automated parsing thread will start a subprocess, which is responsible for the specific PDF parsing work. The parsing process includes the following steps:

[0169] Step 402: PDF positioning:

[0170] The child process first locates the PDF file according to the task configuration and identifies the areas and tables that need to be parsed;

[0171] Step 403: PDF form extraction, further comprising the following steps:

[0172] Step 40301: First, use the extract_table() method provided by pdfplumber to identify the data in the wired table: If the table data cannot be extracted, proceed directly to step 40307; otherwise, proceed to the next step;

[0173] Step 40302: Collect all line segments of the current page and perform preprocessing to merge and splice the line segments, thereby extracting complete horizontal and vertical lines;

[0174] Step 40303: By analyzing the horizontal and vertical lines obtained in the previous step, the intersections of all line segments are accurately found. The framework of all rectangular cells is constructed using the intersection information and the corresponding line segment data. The properties of the rectangular cells are shown in Figure 7 As shown;

[0175] Step 40304: According to the continuity of the rectangular cells, the rectangular cells are reasonably grouped to obtain a table list, each table in the table list contains detailed rectangular cell information;

[0176] Step 40305: According to the positional relationship between the rectangular cells, horizontal lines, and vertical lines, the row position (rows) and column position (columns) of each rectangular cell are accurately determined, thereby establishing a complete table structure;

[0177] Step 40306: accurately fill the text content on the page into the corresponding rectangular cells, so as to achieve complete extraction of the table content, and then proceed to step 404;

[0178] Step 4037: If the data cannot be directly extracted, the basic information of the PDF is obtained according to the pdfplumber library to generate text lines. The corresponding attributes of the text lines are shown in Figure 8 As shown;

[0179] Step 4038: Locate the page and location area where the business table is located according to the business configuration;

[0180] Step 4039: Get the header and footer and remove them, cutting out the text data belonging to the table area;

[0181] Step 40310: Obtain the text information and phrase word of the current page through the pdfplumber library. The corresponding properties of the phrase word are shown in Fig. 9 As shown, further comprising the following steps:

[0182] Step 403101: sort all the word phrases according to bottom;

[0183] Step 403102: determine the error threshold according to the page size and the text size, put all the phrases word whose bottom is within the error threshold into a list, and generate table data rows data_rows;

[0184] Step 403103: traverse all phrases word in the data row data_rows, and dynamically generate and expand the column interval according to x0 and x1 of the phrase word;

[0185] Step 403104: Generate a column_intervals column interval list (elements in it are represented as column_interval(left,right));

[0186] Step 40311: Generate a two-dimensional table according to the data rows data_rows and the column intervals column_intervals;

[0187] Step 40312: Fill all the word phrases into the two-dimensional table. The filling method is divided into 7 types according to the scenario. Fig.10 As shown, the process proceeds to step 404, which includes:

[0188] Scenario 1: If the center point of a word is between the left and right edges of a column, the word belongs to this column.

[0189] Scenario 2: If the coordinates of the center point of the phrase word are within a certain threshold (the default value in the present invention is 3) from the left edge or the right edge of a column, the phrase word belongs to this column;

[0190] Scenario 3: If the left coordinate of the phrase word is close enough to the left edge of a column, the phrase word belongs to this column;

[0191] Scenario 4: If the right coordinate of the phrase word is close enough to the right edge of a column, the phrase word belongs to this column;

[0192] Scenario 5: If the left coordinate of the phrase word is less than the left edge of a column and the right coordinate is greater than the right edge of the column, then the phrase word belongs to this column;

[0193] Scenario 6: The left coordinate of the phrase word is between the left edge and the right edge of its closest left column, and the right coordinate of the phrase word is smaller than the left edge of its closest right column, then the phrase word belongs to the closest left column;

[0194] Scenario 7: The right coordinate of the phrase word is between the left edge and the right edge of its closest right column, and the left coordinate of the phrase word is greater than the right edge of its closest left column, then the phrase word belongs to the closest right column;

[0195] Step 404: Non-derived field mapping:

[0196] Extract non-derived fields from PDF files and map them to system data tables;

[0197] Step 405: Derived fields supplement:

[0198] By calling the derivative service, the complete derivative fields are obtained and added to the system data table;

[0199] Step 406: Insert data table:

[0200] Insert the data parsed in step 403, step 404 and step 405 into the system data table pdfjx and the system data table newsadmin respectively;

[0201] Step 407: Front-end verification:

[0202] The system performs front-end verification on the inserted data to ensure its correctness and completeness;

[0203] Step 408: After the transaction submission is successfully verified, the system will submit the transaction inserted into the system data table newsadmin.

[0204] Step 5: Result feedback and task closing, further including the following steps:

[0205] Step 501: Task processing result callback:

[0206] After the PDF file is parsed, the system will feed back the task processing results to the task center through the HTTP interface to notify the task processing status;

[0207] Step 502: Closing the subprocess and parsing ends:

[0208] After the subprocess is parsed, the system will close the subprocess and release resources; then, the task parsing is officially completed and marked as completed.

[0209] Step 6: Task record update and deduplication, further including the following steps:

[0210] Step 601: Task information update:

[0211] The system updates the task processing status based on the records in the deduplication table: if the task center notifies that the task has been completed, the system removes the saved task notification information;

[0212] Step 602: Task log recording and saving:

[0213] After the analysis is completed, the system will insert the task log into the main table and the deduplication table, and generate a detailed file of unsubmitted tasks to ensure the traceability of the task status.

Claims

1. A complex table structured processing method applicable to A-share financial statements, characterized in that: The following steps are involved: Step 1: Task notification and configuration loading, further including the following steps: Step 101: The system receives a task notification from a task center through an HTTP interface, wherein the task notification includes at least relevant information and key parameters of a PDF file to be parsed; Step 102: After receiving the task notification, the system verifies the key parameters, saves the task notification that passes the verification to a file and memory, and adds it to the task pool; Step 2: Task collection and deduplication, further including the following steps: Step 201: The system starts the task receiving thread, regularly obtains the configuration required for the task from the task pool and prepares to receive the task. The task receiving thread pulls the configuration required for the task and checks whether the system is qualified to receive the task. Only when the system is qualified to receive the task can the task be officially received; Step 203: The system checks the deduplication table to ensure that the task will not be processed repeatedly; Step 204: If the system detects a task primary key conflict, which means that the current task has been pushed repeatedly, the current task will be deleted; if a non-primary key conflict error occurs, the system will still save the task detail file and try to process it; Step 3: Automating the parsing thread and PDF file downloading further includes the following steps: Step 301: After the task is received, the system starts the automatic parsing thread. Each automatic parsing thread is responsible for loading the unfinished task detail file and marking it as busy to prevent repeated processing; Step 302: the automated parsing thread extracts the relevant PDF file download path from the task, and the main process starts downloading the corresponding PDF file; Step 4: PDF file parsing and data processing, including the following steps: Step 401: Each automatic parsing thread starts a subprocess, which is responsible for parsing a specific PDF file, and specifically includes the following steps: Step 4011: The subprocess locates the PDF file according to the task configuration and identifies the area and table that need to be parsed; Step 4012: PDF form extraction, further comprising the following steps: Step 40121: Use the extract_table() method provided by pdfplumber to identify the data in the wired table: If the table data cannot be extracted, proceed directly to step 40127; otherwise, proceed to the next step; Step 40122: Collect all line segments of the current page and perform preprocessing to merge and splice the line segments, thereby extracting complete horizontal and vertical lines; Step 40123: By analyzing the horizontal and vertical lines obtained in the previous step, the intersections of all line segments are accurately found, and the framework of all cells is constructed using the intersection information and the corresponding line segment data of the horizontal and vertical lines; Step 40124: grouping the cells according to the coherence of the cells to obtain a table list, each table in the table list contains detailed cell information; Step 40125: According to the positional relationship between each cell and its corresponding horizontal and vertical lines, the row and column positions of each cell are accurately determined, thereby establishing a complete table structure; Step 40126: Fill the text content on the page into the corresponding cells in the table structure, so as to achieve complete extraction of the PDF table content, and then proceed to step 4013; Step 40127: Obtain basic information of the PDF file according to the pdfplumber library and generate text lines; Step 40128: Locate the page and location area where the business table is located according to the business configuration; Step 40129: Obtain the header and footer of the page where the business table obtained in the previous step is located and remove them, and cut out the text data belonging to the table area based on the location area where the business table obtained in the previous step is located; Step 40130: Obtain the text information corresponding to the text data obtained in the previous step and all the phrases word corresponding to the text information through the pdfplumber library, and obtain the table data rows data_rows and column intervals column_intervals: Step 40131: Generate a two-dimensional table according to the data rows data_rows and the column intervals column_intervals; Step 40132: Fill all phrases word into the two-dimensional table and go to step 4013; Step 4013: extracting non-derived fields from the PDF file; Step 4014: Obtain complete derived fields by calling the derived service; Step 4015: insert the data parsed in steps 4012, 4013 and 4015 into the system data table pdfjx and the system data table newsadmin respectively; Step 4016: The system performs front-end verification on the inserted data to ensure its correctness and completeness; Step 4017: After the transaction submission is successfully verified, the system will submit the transaction inserted into the system data table newsadmin; Step 402: Result feedback and task closing, further including the following steps: Step 4021: After the PDF file is parsed, the system feeds back the task processing result to the task center through the HTTP interface to notify the task processing status; Step 4022: After the subprocess analysis is completed, the system closes the subprocess and releases resources; then, the task analysis is officially completed and marked as completed; Step 5: Task record update and deduplication, further including the following steps: Step 501: The system updates the processing status of the task according to the records in the deduplication table: if the task center notifies that the task has been completed, the system removes the saved task notification information; Step 502: After the analysis is completed, the system inserts the task log into the main table and the deduplication table, and generates an unsubmitted task detail file to ensure the traceability of the task status.

2. A complex table structured processing method applicable to A-share financial statements as claimed in claim 1, characterized in that: In step 203, the system inserts the new task into the deduplication table record; writes the unsubmitted task detail file; commits the transaction, and saves the unsubmitted task into the file.

3. A complex table structure processing method applicable to A-share financial statements as claimed in claim 1, characterized in that: In step 40130, the table data rows data_rows and the column intervals column_intervals are obtained by the following method: Step 401301: sort all the word phrases according to bottom; Step 401302: determine the error threshold according to the page size and the text size, put all the phrases word whose bottom is within the error threshold into a list, and generate table data rows data_rows; Step 401303: traverse all phrases word in the data row data_rows, dynamically generate and expand column intervals according to x0 and x1 of the phrase word, and obtain column interval column_intervals.

4. A complex table structure processing method applicable to A-share financial statements as claimed in claim 1, characterized in that: In step 40132, the filling methods are divided into 7 according to the scene, including: Scenario 1: If the center point of a word is between the left and right edges of a column, the word belongs to this column. Scenario 2: If the coordinates of the center point of a word are within a certain threshold from the left or right edge of a column, the word belongs to this column; Scenario 3: If the left coordinate of the phrase word is close enough to the left edge of a column, the phrase word belongs to this column; Scenario 4: If the right coordinate of the phrase word is close enough to the right edge of a column, the phrase word belongs to this column; Scenario 5: If the left coordinate of the phrase word is less than the left edge of a column and the right coordinate is greater than the right edge of the column, then the phrase word belongs to this column; Scenario 6: The left coordinate of the phrase word is between the left edge and the right edge of its closest left column, and the right coordinate of the phrase word is smaller than the left edge of its closest right column, then the phrase word belongs to the closest left column; Scenario 7: The right coordinate of the phrase word is between the left edge and the right edge of its closest right column, and the left coordinate of the phrase word is greater than the right edge of its closest left column, then the phrase word belongs to the closest right column.

Images