File parsing method and system applied to ofd reader
By splitting and clustering OFD files into pages and optimizing the parsing order based on feature scoring, more efficient file parsing was achieved, solving the problem of slow parsing speed and improving user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- UWAYSOFT BEIJING INFORMATION TECH CO LTD
- Filing Date
- 2026-02-04
- Publication Date
- 2026-05-12
AI Technical Summary
ofd files can be slow to parse on different devices, affecting user experience and potentially causing users to mistakenly believe the file is corrupted and delete it.
The OFD file is split into pages, and object analysis and clustering are performed to construct a content distribution array. Based on structural and content features, the parsing score is determined, and a gradient parsing process is executed.
It improves the parsing efficiency of OFD files, allowing users to monitor the parsing process in real time, thus enhancing the user experience.
Smart Images
Figure CN121640501B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of file parsing technology, specifically a file parsing method and system applied to OFD readers. Background Technology
[0002] OFD (Open Fixed-layout Document) is a well-established and publicly used open format document standard, primarily used for electronic documents, archive storage and exchange. It provides a document format independent of specific software or hardware, ensuring consistent display across different devices and platforms. OFD files not only contain structured data but also support rich graphical elements, allowing for precise definition of document layout and content, including text, images, vector graphics, and annotations. Due to its widespread applicability, this type of text is used in most interactive scenarios. However, OFD files also have some drawbacks. Displaying OFD files on different devices requires parsing first, which can be slow when the file is large, impacting user experience. Sometimes, users unfamiliar with this type of file may mistake slow parsing for corrupted files and delete them. Therefore, improving the parsing speed of OFD files is the technical problem this invention aims to solve. Summary of the Invention
[0003] The purpose of this invention is to provide a file parsing method and system for OFD readers, so as to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, the present invention provides the following technical solution:
[0005] A file parsing method and system applied to an OFD reader, the method comprising:
[0006] Receive the OFD file, split the OFD file into pages, perform object analysis on the content of each page, and construct a content distribution array; the content distribution array is used to represent the data location and data volume of different data structures;
[0007] The pages are clustered in a first-level manner based on the content distribution array. For each page after the first-level clustering, the structural features and content features of the page are obtained.
[0008] The pages are clustered into two levels based on structural and content features. Simultaneously, a parsing score for each page is determined based on the first and second level clustering processes. The parsing score is used to characterize the parsing priority of the page.
[0009] Pages are selected based on the parsing score, and the parsing process is executed. A simplified parsing process is constructed based on the parsed content and the secondary clustering of similar pages. Gradient parsing is then performed on each page based on the simplified parsing process.
[0010] As a further aspect of the present invention: the steps of receiving the OFD file, splitting the OFD file by page, performing object analysis on the content of each page, and constructing a content distribution array include:
[0011] Receive the ofd file, traverse the ofd file, and locate the Page node in the ofd file;
[0012] The ofd file is split based on the Page node to obtain pages containing page order;
[0013] Object recognition is performed on each page to determine the object type and location; the object type includes images and text.
[0014] The position is mapped to the row and column positions of a matrix, and the object type is statistically analyzed based on the row and column positions to obtain a content distribution array; the content distribution array is a two-dimensional matrix.
[0015] As a further aspect of the present invention: the step of performing first-level clustering of pages based on the content distribution array, and obtaining the structural and content features of each page after first-level clustering, includes:
[0016] Compare the content distribution arrays of different pages and calculate the array distance;
[0017] The page distance is determined based on the array distance and the page order difference.
[0018] Based on page distance, the pages are clustered into a first-level cluster. For each page in the first-level cluster, the structural and content features of the page are obtained.
[0019] The structural features include page text density, effective text ratio, character confidence, and structural complexity; the content features include page keyword concentration and semantic topic consistency score.
[0020] As a further aspect of the present invention: the step of performing secondary clustering of pages based on structural and content features, and simultaneously determining the analytical score of each page based on the primary and secondary clustering processes, includes:
[0021] For each page after the first-level clustering, compare the structural and content features of any two pages and calculate the information distance.
[0022] Secondary clustering is performed on each page after the primary clustering based on information distance;
[0023] Read the total number of pages in each cluster after secondary clustering, and determine the benchmark score based on the total number of pages;
[0024] The floating score for each page is determined based on structural and content characteristics;
[0025] The baseline score and the floating score are accumulated to obtain the analytical score for each page.
[0026] As a further aspect of the present invention: the process of obtaining the structural features is as follows:
[0027] Page text density is the ratio of the area occupied by text objects on the page to the total display area of the page;
[0028] The effective text percentage is the ratio of the number of consecutively concatenated text characters to the total number of text characters; where the number of consecutively concatenated text characters is the number of characters whose character spacing and line spacing are less than a preset threshold condition;
[0029] The confidence level of a character is determined by the inverse ratio of the confidence level to that of an outlier character.
[0030] The structural complexity is directly proportional to the number of special formats; these special formats include tables, comments, footnotes, and rotated text.
[0031] The process of obtaining the content features is as follows:
[0032] The process of determining the keyword concentration of a page is as follows: extract keywords based on TF-IDF values, calculate the total number of extracted keywords, and calculate the ratio of the total number of keywords to the total number of words on the page as the keyword concentration of the page;
[0033] The process for determining the semantic topic consistency score is as follows:
[0034] Locate the text blocks on the page, convert the text blocks into semantic vectors, compare the semantic vectors of the text blocks pairwise, calculate the semantic similarity, and calculate the mean of all semantic similarities as the semantic topic consistency score.
[0035] The process for determining the floating score is as follows:
[0036] Based on the preset weighted statistical structure features and content features, a floating score is obtained; among them, the floating score is positively proportional to the page text density, effective text ratio, character confidence, page keyword concentration and semantic theme consistency score, and inversely proportional to the structural complexity.
[0037] As a further aspect of the present invention: the steps of selecting pages based on the parsing score, executing the parsing process, constructing a simplified parsing process for similar pages after secondary clustering based on the parsed content, and performing gradient parsing on each page based on the simplified parsing process include:
[0038] Select a page based on the parsing score, execute the parsing process, and record the parsing parameters;
[0039] Each time the parsing process is executed, the number of pages in each cluster after the second clustering is obtained;
[0040] The baseline score is recursively updated based on the number of executions; the parsed score is updated accordingly after the baseline score is updated.
[0041] For each page after secondary clustering, randomly query the parsing parameters corresponding to a parsed page to construct a simplified parsing process;
[0042] The simplified parsing process is run in parallel with the original parsing process to pre-parse pages of the same type after secondary clustering.
[0043] The present invention also provides a file parsing system for OFD readers, the system comprising:
[0044] The distribution determination module receives the OFD file, splits the OFD file by page, performs object analysis on the content of each page, and constructs a content distribution array; the content distribution array is used to represent the data location and data volume of different data structures.
[0045] The page feature extraction module is used to perform first-level clustering of pages based on the content distribution array, and to obtain the structural and content features of each page after first-level clustering.
[0046] The parsing score determination module is used to perform secondary clustering of pages based on structural and content features, and simultaneously determine the parsing score of each page based on the primary and secondary clustering processes; the parsing score is used to characterize the parsing priority of the page;
[0047] The gradient processing module is used to select pages based on the parsing score, execute the parsing process, construct a simplified parsing process for similar pages after secondary clustering based on the parsed content, and perform gradient parsing on each page based on the simplified parsing process.
[0048] As a further aspect of the present invention: the distribution determination module includes:
[0049] The node positioning unit is used to receive the ofd file, traverse the ofd file, and locate the Page node in the ofd file;
[0050] The file splitting unit is used to split the ofd file based on the Page node to obtain pages containing page order;
[0051] An object recognition unit is used to recognize objects on each page and determine the object type and location; the object type includes images and text.
[0052] An array generation unit is used to map positions to row and column positions of a matrix, and to obtain a content distribution array based on the object type of the row and column positions; the content distribution array is a two-dimensional matrix.
[0053] As a further aspect of the present invention: the page feature extraction module includes:
[0054] The array comparison unit is used to compare the content distribution arrays of different pages and calculate the array distance;
[0055] The page distance determination unit is used to determine the page distance based on the array distance and the page order difference.
[0056] The extraction execution unit is used to perform first-level clustering of pages based on page distance, and to obtain the structural and content features of each page after first-level clustering.
[0057] The structural features include page text density, effective text ratio, character confidence, and structural complexity; the content features include page keyword concentration and semantic topic consistency score.
[0058] As a further aspect of the present invention: the analytical scoring determination module includes:
[0059] The information distance calculation unit is used to compare the structural and content features of any two pages for each class of pages after the first-level clustering and calculate the information distance.
[0060] Secondary clustering units are used to perform secondary clustering on each page after primary clustering based on information distance;
[0061] The benchmark score determination unit is used to read the total number of pages in each class after secondary clustering and determine the benchmark score based on the total number of pages.
[0062] The floating score determination unit is used to determine the floating score of each page based on structural and content characteristics.
[0063] The rating accumulation output unit is used to accumulate the baseline rating and the floating rating to obtain the parsed rating for each page.
[0064] Compared with the prior art, the beneficial effects of the present invention are:
[0065] This invention performs page segmentation on OFD files, initial page identification, and clustering of pages based on the initial identification results. Building upon the traditional parsing process, it performs a simplified pre-parsing operation on similar pages based on the parsing parameters of the already processed pages. This allows for the early acquisition of preliminary content with some accuracy but existing parsing results. On the display end, the parsed content can be continuously displayed, which improves parsing efficiency and allows users to observe the parsing changes in real time, greatly enhancing the user experience. Attached Figure Description
[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention.
[0067] Figure 1 The overall flowchart of the file parsing method applied to the OFD reader is shown.
[0068] Figure 2 A structural diagram of the file parsing system applied to the OFD reader is shown. Detailed Implementation
[0069] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.
[0070] Figure 1 The present invention provides a file parsing method and system for an OFD reader, comprising the following steps: (See the overall flowchart of the method and system for OFD reader.)
[0071] Step S100: Receive the ofd file, split the ofd file by page, perform object analysis on the content of each page, and construct a content distribution array; wherein, the content distribution array is used to represent the data location and data volume of different data structures;
[0072] OFD (Open Fixed-layout Document) is a well-established and publicly used open-format document standard, primarily used for electronic documents, archive storage, and exchange. It provides a document format independent of specific software or hardware, ensuring consistent display across different devices and platforms. OFD files not only contain structured data but also support rich graphical elements, precisely defining document layout and content, including text, images, vector graphics, and annotations. OFD files contain page separators; splitting the file based on these separators yields multiple pages. Identifying the content on each page determines its distribution, stored as a numerical parameter called the content distribution array. This initial identification process identifies which types of content are present in which locations. It doesn't require further identification of specific content, such as whether a particular element is an image or text; further identification occurs later.
[0073] Step S200: Perform first-level clustering of the pages based on the content distribution array, and obtain the structural features and content features of each page after first-level clustering;
[0074] The content distribution array represents the content distribution of each page. It is an initial structural information. By comparing the content distribution arrays of different pages, we can determine the initial differences in page structure. Based on the known differences, we can execute existing clustering algorithms to cluster the pages, so that pages that are initially similar can be grouped into one category. This process only requires a quick traversal of each page, and the operation speed is extremely fast. This is also a primary clustering process, called the first-level clustering process.
[0075] For each page after the first-level clustering, the content of the page is further identified. This identification process is slightly more refined, but it is still not enough to achieve the level of parsing. It will obtain the specific structural and content information of the page, thereby extracting the structural and content features of the page.
[0076] Step S300: Perform secondary clustering on the pages based on structural and content features, and simultaneously determine the parsing score of each page based on the primary and secondary clustering processes; the parsing score is used to characterize the parsing priority of the page;
[0077] Structural and content features are used as page characteristics. By comparing the differences between pages, which represent deeper information about each page, secondary clustering can be performed. After secondary clustering, pages of the same type have a higher similarity. Combining the primary and secondary clustering processes, a parsing score can be determined for each page. This parsing score is used to characterize the parsing priority of the page, and its practical meaning represents the value of the page being parsed, which is used to adjust the order of page parsing processes.
[0078] Step S400: Select pages based on the parsing score, execute the parsing process, construct a simplified parsing process for similar pages after secondary clustering based on the parsed content, and perform gradient parsing on each page based on the simplified parsing process;
[0079] In conventional logic (which is also the logic used in the example of the technical solution of this invention), pages are selected in descending order of parsing scores. Then, for each page, a parsing process is executed. This parsing process is existing technology and belongs to the standardized operation of OFD files. It is performed in an independent process, which can be executed when opening an OFD file or after performing steps S100 to S300. The special feature of the technical solution of this invention is that, based on the execution of the parsing process (existing solutions), a simplified parsing process is directly applied to similar pages based on the parsed content. That is, for any page, the parsing operations in the parsing process are recorded, and the parsing operations are directly applied to similar files. Since the parsing process is essentially a decoding process of some data formats, in similar pages... While migration applications may introduce some errors, when the structure and content are highly similar (similar pages), some parsing results may still be correct. In extreme cases, the parsing operation is identical for two identical pages. This direct application parsing process eliminates the need for prior page content identification and parsing method determination, saving on preprocessing and thus increasing efficiency. However, as mentioned above, its accuracy is not high, therefore it can only serve as a simplified parsing process. Gradient parsing of each page based on the simplified parsing process means adding a simplified parsing process based on already processed content to the original parsing process. This is equivalent to a highly efficient pre-parsing process, hence the name gradient parsing process.
[0080] Regarding step S100, the steps of receiving the OFD file, splitting the OFD file by page, performing object analysis on the content of each page, and constructing a content distribution array include:
[0081] Receive the ofd file, traverse the ofd file, and locate the Page node in the ofd file;
[0082] The ofd file is split based on the Page node to obtain pages containing page order;
[0083] Object recognition is performed on each page to determine the object type and location; the object type includes images and text.
[0084] The position is mapped to the row and column positions of a matrix, and the object type is statistically analyzed based on the row and column positions to obtain a content distribution array; the content distribution array is a two-dimensional matrix.
[0085] The process of determining the content distribution array involves receiving an OFD file, traversing the OFD file, locating Page nodes within it, and splitting the OFD file based on these Page nodes to obtain pages with page order. This node-based splitting operation is extremely simple. On top of this, object recognition is performed on each page to determine the object type and location. This is a very basic operation, only requiring identification of which content formats exist at which locations. In extreme cases, where only text and image formats exist, this method is even simpler. For example, the content can be directly located on the page. For images, this is generally an address indicating the image's storage location. For non-address content, it can be considered text. This method determines the object type, and subsequent location markers will also be present. The signature indicates the object type and location. Based on this, a matrix is constructed using the preset page size (the displayed interface is essentially a matrix). The matrix counts the content format corresponding to each location. If only images and text are distinguished, 0 and 1 can be used directly (if there are many types, other values are used; the object types in an OFD file are limited to a few). In this case, the matrix is a zero-one matrix, with 0 and 1 representing images and text respectively. For ease of understanding, this matrix is defined as follows: for a matrix, its size is the same as the size of the future displayed page, where each position corresponds to a pixel, and the value of the position represents its corresponding object type. The matrix mentioned above is called the content distribution array, representing the format of data at different locations on a page.
[0086] Regarding step S200, the step of performing first-level clustering of pages based on the content distribution array, and obtaining the structural and content features of each page after first-level clustering, includes:
[0087] Compare the content distribution arrays of different pages and calculate the array distance;
[0088] The page distance is determined based on the array distance and the page order difference.
[0089] The pages are clustered at the first level based on page distance. For each page in the first-level cluster, the structural and content features of the page are obtained.
[0090] In one example of the technical solution of this invention, the clustering process and feature extraction process are described. The content distribution arrays of different pages are compared, and the array distance is calculated. If it is a zero-one matrix, AND or XOR operations can be used to quickly determine the difference positions, and then divided by the total number to obtain the array distance. If it is a multi-value matrix, conventional matrix distance methods, such as Euclidean distance, can be used. Meanwhile, in actual OFD files, the similarity of adjacent pages may correspond to the same topic, and their similarity may be higher. Therefore, when determining page differences, the page order parameter needs to be introduced. The page distance is determined based on the array distance and the difference in page order. The page distance is directly proportional to the array distance and the page order difference. Pages are clustered based on the page distance; this is the first clustering, hence called first-level clustering. For each page after first-level clustering, the structural features and content features of the page are obtained. The structural features include page text density, effective text ratio, character confidence, and structural complexity; the content features include page keyword concentration and semantic topic consistency score.
[0091] Regarding step S300, the step of performing secondary clustering of the page based on structural and content features, and simultaneously determining the parsing score of each page based on the primary and secondary clustering processes, includes:
[0092] For each page after the first-level clustering, compare the structural and content features of any two pages and calculate the information distance.
[0093] Secondary clustering is performed on each page after the primary clustering based on information distance;
[0094] Read the total number of pages in each cluster after secondary clustering, and determine the benchmark score based on the total number of pages;
[0095] The floating score for each page is determined based on structural and content characteristics;
[0096] The baseline score and the floating score are accumulated to obtain the analytical score for each page.
[0097] Based on the completion of the first-level clustering, the process of determining the second-level clustering and the analytical score is explained. For each page after the first-level clustering, the structural and content features of any two pages are compared, and the difference between them is calculated. The resulting parameter is called the information distance, which represents the difference in information within each page. Second-level clustering is then performed on each page after the first-level clustering based on the information distance. After the second-level clustering process is completed, the total number of pages in each page after the second-level clustering is read. The higher the total number of pages, the more content related to the corresponding information is in the entire OFD file, and correspondingly, the more important the content is. A parameter called the baseline score is determined by the total number of pages. Furthermore, the content of each page is identified by combining structural and content features to obtain a floating score for each page. The baseline score and the floating score are accumulated to obtain a parameter for each page that reflects its importance, called the analytical score.
[0098] As an example of the technical solution of this invention, the technical details of the above content include at least the process of determining structural features, the process of determining content features, the process of determining floating scores, and the specific operations of clustering. The explanation of these technical details is as follows:
[0099] The process of obtaining the structural features is as follows:
[0100] Page text density is the ratio of the area occupied by text objects on the page to the total display area of the page;
[0101] The effective text percentage is the ratio of the number of consecutively concatenated text characters to the total number of text characters; where the number of consecutively concatenated text characters is the number of characters whose character spacing and line spacing are less than a preset threshold condition;
[0102] The confidence level of a character is determined by the inverse ratio of the confidence level to that of an outlier character.
[0103] The structural complexity is directly proportional to the number of special formats; these special formats include tables, comments, footnotes, and rotated text.
[0104] The process of obtaining the content features is as follows:
[0105] The process of determining the keyword concentration of a page is as follows: extract keywords based on TF-IDF values, calculate the total number of extracted keywords, and calculate the ratio of the total number of keywords to the total number of words on the page as the keyword concentration of the page;
[0106] The process for determining the semantic topic consistency score is as follows:
[0107] Locate the text blocks on the page, convert the text blocks into semantic vectors, compare the semantic vectors of the text blocks pairwise, calculate the semantic similarity, and calculate the mean of all semantic similarities as the semantic topic consistency score.
[0108] The process for determining the floating score is as follows:
[0109] Based on the preset weighted statistical structure features and content features, a floating score is obtained; among them, the floating score is positively proportional to the page text density, effective text ratio, character confidence, page keyword concentration and semantic theme consistency score, and inversely proportional to the structural complexity.
[0110] Regarding the above calculation process, the meaning of each parameter and its relationship with the floating score are explained as follows:
[0111] The meanings of page text density are as follows: High text density indicates high information capacity, while low text density indicates a higher probability of cover, illustration, and decorative pages. The meaning of effective text percentage is as follows: High effective text percentage indicates continuous and highly readable text, while low percentage indicates more watermarks, noisy characters, and discrete annotations. The meaning of character confidence is as follows: High character confidence indicates reliable text, while low character confidence indicates more OCR errors or encoding anomalies. The meaning of page keyword concentration is as follows: Concentrated keywords indicate a clear theme, while scattered keywords indicate a higher probability of being a table of contents, index, or description page. The meaning of semantic theme consistency score is as follows: High consistency... The parameters indicate whether a page is a single-topic page (content page) or a page with low consistency (table page, mixed page, or description page). These parameters all indicate that the page is more suitable to be parsed first and as a template to simplify the parsing of similar pages. The calculated floating score is directly proportional to these parameters. Conversely, structural complexity means that a complex structure indicates high parsing cost and high failure risk, while a simple structure indicates stable parsing and high reuse value. Although a complex structure may be more important in terms of content, it does not utilize the reuse process, meaning it is not suitable as a template to parse other pages. Therefore, the calculated floating score is inversely proportional to this complexity.
[0112] In addition, regarding the clustering process, the technical solution of this invention preferably adopts a clustering algorithm that does not limit the number of clusters. Given a fixed distance, it is not difficult to use existing clustering algorithms.
[0113] Regarding step S400, the steps of selecting pages based on the parsing score, executing the parsing process, constructing a simplified parsing process for similar pages after secondary clustering based on the parsed content, and performing gradient parsing on each page based on the simplified parsing process include:
[0114] Select a page based on the parsing score, execute the parsing process, and record the parsing parameters;
[0115] Each time the parsing process is executed, the number of pages in each cluster after the second clustering is obtained;
[0116] The baseline score is recursively updated based on the number of executions; the parsed score is updated accordingly after the baseline score is updated.
[0117] For each page after secondary clustering, randomly query the parsing parameters corresponding to a parsed page to construct a simplified parsing process;
[0118] The simplified parsing process is run in parallel with the original parsing process to pre-parse pages of the same type after secondary clustering.
[0119] In one example of the technical solution of this invention, the parsing process is specifically described. Pages are selected sequentially according to the descending order of the parsing scores, and the parsing process is executed. During the execution of the parsing process, parsing parameters are recorded, which represent the operation information used in the parsing process. Each time the parsing process is executed, the number of pages in each cluster after secondary clustering is obtained. The more pages that have been executed, the more parsing parameters can be used to simplify the parsing among the pages of the same type. At this time, the benchmark score can be lowered, and correspondingly, the parsing score of that page will also decrease (the floating score remains unchanged). In subsequent processes, the parsing order of pages of the same type will be placed later. This is a dynamic adjustment process. In addition, for each page after secondary clustering, the parsing parameters corresponding to a parsed page are randomly queried to construct a simplified parsing process. The simplified parsing process applies the selected parsing parameters to other pages of the same type for a more efficient but less accurate parsing process. The simplified parsing process is a process that runs in parallel with the original parsing process to pre-parse the pages of the same type after secondary clustering. It should be noted that this process is parallel and does not affect the original parsing process, which is executed sequentially.
[0120] Figure 2 A structural diagram of a file parsing system applied to an OFD reader is shown. In a preferred embodiment of the technical solution of the present invention, a file parsing system applied to an OFD reader is also provided, the system 10 comprising:
[0121] The distribution determination module 11 is used to receive the ofd file, split the ofd file by page, perform object analysis on the content of each page, and construct a content distribution array; wherein, the content distribution array is used to represent the data location and data volume of different data structures;
[0122] The page feature extraction module 12 is used to perform first-level clustering of pages based on the content distribution array, and to obtain the structural features and content features of each page after first-level clustering.
[0123] The parsing score determination module 13 is used to perform secondary clustering of pages based on structural and content features, and simultaneously determine the parsing score of each page based on the primary and secondary clustering processes; the parsing score is used to characterize the parsing priority of the page;
[0124] The gradient processing module 14 is used to select pages based on the parsing score, execute the parsing process, construct a simplified parsing process for similar pages after secondary clustering based on the parsed content, and perform gradient parsing on each page based on the simplified parsing process.
[0125] Furthermore, the distribution determination module 11 includes:
[0126] The node positioning unit is used to receive the ofd file, traverse the ofd file, and locate the Page node in the ofd file;
[0127] The file splitting unit is used to split the ofd file based on the Page node to obtain pages containing page order;
[0128] An object recognition unit is used to recognize objects on each page and determine the object type and location; the object type includes images and text.
[0129] An array generation unit is used to map positions to row and column positions of a matrix, and to obtain a content distribution array based on the object type of the row and column positions; the content distribution array is a two-dimensional matrix.
[0130] Specifically, the page feature extraction module 12 includes:
[0131] The array comparison unit is used to compare the content distribution arrays of different pages and calculate the array distance;
[0132] The page distance determination unit is used to determine the page distance based on the array distance and the page order difference.
[0133] The extraction execution unit is used to perform first-level clustering of pages based on page distance, and to obtain the structural and content features of each page after first-level clustering.
[0134] The structural features include page text density, effective text ratio, character confidence, and structural complexity; the content features include page keyword concentration and semantic topic consistency score.
[0135] Furthermore, the analytical scoring determination module 13 includes:
[0136] The information distance calculation unit is used to compare the structural and content features of any two pages for each class of pages after the first-level clustering and calculate the information distance.
[0137] Secondary clustering units are used to perform secondary clustering on each page after primary clustering based on information distance;
[0138] The benchmark score determination unit is used to read the total number of pages in each class after secondary clustering and determine the benchmark score based on the total number of pages.
[0139] The floating score determination unit is used to determine the floating score of each page based on structural and content characteristics.
[0140] The rating accumulation output unit is used to accumulate the baseline rating and the floating rating to obtain the parsed rating for each page.
[0141] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.
Claims
1. A file parsing method applied to an OFD reader, characterized in that, The method includes: Receive the OFD file, split the OFD file into pages, perform object analysis on the content of each page, and construct a content distribution array; the content distribution array is used to represent the data location and data volume of different data structures; Compare the content distribution arrays of different pages and calculate the array distance; The page distance is determined based on the array distance and the page order difference. Based on page distance, the pages are clustered into a first-level cluster. For each page in the first-level cluster, the structural and content features of the page are obtained. Among them, structural features include page text density, effective text ratio, character confidence, and structural complexity; content features include page keyword concentration and semantic topic consistency score. For each page after the first-level clustering, compare the structural and content features of any two pages and calculate the information distance. Secondary clustering is performed on each page after the primary clustering based on information distance; Read the total number of pages in each cluster after secondary clustering, and determine the benchmark score based on the total number of pages; The floating score for each page is determined based on structural and content characteristics; The baseline score and the floating score are accumulated to obtain the parsing score for each page; the parsing score is used to represent the parsing priority of the page. Select a page based on the parsing score, execute the parsing process, and record the parsing parameters; Each time the parsing process is executed, the number of pages in each cluster after the second clustering is obtained; The baseline score is recursively updated based on the number of executions; the parsed score is updated accordingly after the baseline score is updated. For each page after secondary clustering, randomly query the parsing parameters corresponding to a parsed page to construct a simplified parsing process; The simplified parsing process is run in parallel with the original parsing process to pre-parse pages of the same type after secondary clustering.
2. The file parsing method applied to an OFD reader according to claim 1, characterized in that, The steps of receiving the OFD file, splitting the OFD file by page, performing object analysis on the content of each page, and constructing a content distribution array include: Receive the ofd file, traverse the ofd file, and locate the Page node in the ofd file; The ofd file is split based on the Page node to obtain pages containing page order; Object recognition is performed on each page to determine the object type and location; the object type includes images and text. The position is mapped to the row and column positions of a matrix, and the object type is statistically analyzed based on the row and column positions to obtain a content distribution array; the content distribution array is a two-dimensional matrix.
3. The file parsing method applied to an OFD reader according to claim 1, characterized in that, The process of obtaining the structural features is as follows: Page text density is the ratio of the area occupied by text objects on the page to the total display area of the page; The effective text percentage is the ratio of the number of consecutively concatenated text characters to the total number of text characters; where the number of consecutively concatenated text characters is the number of characters whose character spacing and line spacing are less than a preset threshold condition; The confidence level of a character is determined by the inverse ratio of the confidence level to that of an outlier character. The structural complexity is directly proportional to the number of special formats; these special formats include tables, comments, footnotes, and rotated text. The process of obtaining the content features is as follows: The process of determining the keyword concentration of a page is as follows: extract keywords based on TF-IDF values, calculate the total number of extracted keywords, and calculate the ratio of the total number of keywords to the total number of words on the page as the keyword concentration of the page; The process for determining the semantic topic consistency score is as follows: Locate the text blocks on the page, convert the text blocks into semantic vectors, compare the semantic vectors of the text blocks pairwise, calculate the semantic similarity, and calculate the mean of all semantic similarities as the semantic topic consistency score. The process for determining the floating score is as follows: Based on the preset weighted statistical structure features and content features, a floating score is obtained; among them, the floating score is positively proportional to the page text density, effective text ratio, character confidence, page keyword concentration and semantic theme consistency score, and inversely proportional to the structural complexity.
4. A file parsing system for OFD readers, characterized in that, The system includes: The distribution determination module receives the OFD file, splits the OFD file by page, performs object analysis on the content of each page, and constructs a content distribution array; the content distribution array is used to represent the data location and data volume of different data structures. The page feature extraction module is used to perform first-level clustering of pages based on the content distribution array, and to obtain the structural and content features of each page after first-level clustering. The parsing score determination module is used to perform secondary clustering of pages based on structural and content features, and simultaneously determine the parsing score of each page based on the primary and secondary clustering processes; the parsing score is used to characterize the parsing priority of the page; The gradient processing module is used to select pages based on the parsing score, execute the parsing process, construct a simplified parsing process for similar pages after secondary clustering based on the parsed content, and perform gradient parsing on each page based on the simplified parsing process. The page feature extraction module includes: The array comparison unit is used to compare the content distribution arrays of different pages and calculate the array distance; The page distance determination unit is used to determine the page distance based on the array distance and the page order difference. The extraction execution unit is used to perform first-level clustering of pages based on page distance, and to obtain the structural and content features of each page after first-level clustering. Among them, structural features include page text density, effective text ratio, character confidence, and structural complexity; content features include page keyword concentration and semantic topic consistency score. The analytical scoring determination module includes: The information distance calculation unit is used to compare the structural and content features of any two pages for each class of pages after the first-level clustering and calculate the information distance. Secondary clustering units are used to perform secondary clustering on each page after primary clustering based on information distance; The benchmark score determination unit is used to read the total number of pages in each class after secondary clustering and determine the benchmark score based on the total number of pages. The floating score determination unit is used to determine the floating score of each page based on structural and content characteristics. The rating accumulation output unit is used to accumulate the baseline rating and the floating rating to obtain the parsed rating for each page; The process of selecting pages based on parsing scores, executing the parsing process, constructing a simplified parsing process for similar pages after secondary clustering based on the parsed content, and performing gradient parsing on each page based on the simplified parsing process includes: Select a page based on the parsing score, execute the parsing process, and record the parsing parameters; Each time the parsing process is executed, the number of pages in each cluster after the second clustering is obtained; The baseline score is recursively updated based on the number of executions; the parsed score is updated accordingly after the baseline score is updated. For each page after secondary clustering, randomly query the parsing parameters corresponding to a parsed page to construct a simplified parsing process; The simplified parsing process is run in parallel with the original parsing process to pre-parse pages of the same type after secondary clustering.
5. The file parsing system applied to an OFD reader according to claim 4, characterized in that, The distribution determination module includes: The node positioning unit is used to receive the ofd file, traverse the ofd file, and locate the Page node in the ofd file; The file splitting unit is used to split the ofd file based on the Page node to obtain pages containing page order; An object recognition unit is used to recognize objects on each page and determine the object type and location; the object type includes images and text. An array generation unit is used to map positions to row and column positions of a matrix, and to obtain a content distribution array based on the object type of the row and column positions; the content distribution array is a two-dimensional matrix.