Standard data file analysis method and device, electronic equipment and storage medium

Abstract

The invention discloses a standard data file analysis method and device, electronic equipment and a storage medium, and relates to the field of data processing or other related technical fields, and the method comprises the steps: obtaining a standard data file, preprocessing the standard data file, and converting the content in the standard data file into a uniform format; constructing a heterogeneous graph for the preprocessed standard data file, and performing hierarchical analysis and entry recognition based on the heterogeneous graph to obtain a plurality of data units; performing content identification on each data unit, and calibrating an identification result based on a knowledge graph of a field to which the standard data file belongs to obtain a standard data unit; and performing structured extraction on the standard data unit, constructing a semantic structure chart, and generating a structured data file based on the semantic structure chart. According to the method and the device, the technical problems that the standard data file depends on manual analysis and calibration, the analysis efficiency is relatively low, and the accuracy of an analysis result is relatively low in the prior art are solved.

Description

Technical Field

[0001] This invention relates to the field of data processing or other related technical fields. Specifically, it relates to a method, apparatus, electronic device, and storage medium for parsing standard data files. Background Technology

[0002] With the increasing demands for digitalization and regulatory compliance across industries, the number of standard data files (including terminology, field definitions, data types, constraints, value ranges, code sets, relationships, and message templates) is growing rapidly. Standard data files, as key tools for ensuring data consistency, accuracy, and interoperability, play a central role in multiple fields such as finance, government affairs, and healthcare. However, the diversity and complexity of data standard files, including terminology, field definitions, data types, constraints, value ranges, code sets, relationships, and message templates, as well as their multimodal presentation formats such as Word, PDF, scanned documents, web page exports, embedded tables, and mixed text, pose significant obstacles to efficient and accurate data standard parsing and conversion.

[0003] In related technologies, manual parsing and calibration are relied upon to convert standard data files from various fields into universal data files, ensuring data readability. However, this method suffers from low parsing efficiency and is prone to errors that lead to inaccurate parsing results when faced with massive amounts of standard data files.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for parsing standard data files, to at least solve the technical problems in related technologies where standard data files rely on manual parsing and calibration, resulting in low parsing efficiency and low accuracy of parsing results.

[0006] According to one aspect of the present invention, a method for parsing a standard data file is provided, comprising: acquiring a standard data file and preprocessing the standard data file to convert the content of the standard data file into a unified format, wherein the standard data file represents a document formulated based on rules of a domain; constructing a heterogeneous graph for the preprocessed standard data file, and performing hierarchical parsing and entry recognition based on the heterogeneous graph to obtain multiple data units; performing content recognition on each data unit, and calibrating the recognition results based on a knowledge graph of the domain to which the standard data file belongs to obtain canonical data units; performing structured extraction on the canonical data units, constructing a semantic structure graph, and generating a structured data file based on the semantic structure graph.

[0007] Further, the step of converting the content of the standard data file into a unified format includes: scanning the standard data file to identify different types of data content in the standard data file, wherein the types of data content include at least one of the following: image data, text data, and embedded tables; extracting the image data from the standard data file and converting the extracted image data into ordered page images in a unified format; extracting the text data from the standard data file and converting the extracted text data into content text in a unified format; and extracting the embedded tables from the standard data file and converting the extracted embedded tables into cell grids in a unified format.

[0008] Further, the step of constructing a heterogeneous graph for the preprocessed standard data file includes: dividing the preprocessed standard data file according to physical pages to obtain multiple independent processing units; identifying feature elements within the independent processing units, and constructing an initial heterogeneous graph using the feature elements as nodes, wherein the feature elements include at least one of the following: text blocks, image blocks, cell grids, titles, headers, and footers; obtaining the association relationships between each feature element in the initial heterogeneous graph, and constructing the heterogeneous graph using the association relationships between the feature elements as edges.

[0009] Further, the step of performing hierarchical parsing and entry identification based on the heterogeneous graph to obtain multiple data units includes: analyzing the hierarchical structure of the standard data file based on the header nodes in the heterogeneous graph; dividing the standard data file based on the hierarchical structure to obtain multiple initial data units; identifying entries in each initial data unit; and segmenting the initial data unit based on the identified entries to obtain multiple data units.

[0010] Furthermore, after obtaining multiple data units, the method further includes: for each data unit, identifying cross-page anchor points in the data unit; and restoring the separated data content based on the cross-page anchor points to obtain new data units.

[0011] Further, the steps of performing content recognition on each of the data units and calibrating the recognition results based on the knowledge graph of the domain to which the standard data file belongs include: using optical character recognition to perform content recognition on image blocks in each of the data units to obtain image content recognition results; using semantic recognition to perform content recognition on text blocks in each of the data units to obtain text content recognition results; using table recognition to perform content recognition on cell grids in each of the data units to obtain grid content recognition results; obtaining the knowledge graph of the domain to which the standard data file belongs, matching the image content recognition results, the content recognition results, and the grid content recognition results respectively, and calibrating the content within the data units based on the matching results.

[0012] Further, the steps of performing structured extraction of the standardized data units, constructing a semantic structure graph, and generating a structured data file based on the semantic structure graph include: performing semantic recognition on each standardized data unit, and extracting entities from the standardized data unit based on the semantic recognition results, wherein the entity includes at least a target field; constructing the semantic structure graph with the entity as a node and the relationship between entities as edges; determining the target data format to be converted, and converting the semantic structure graph into the structured data file based on the target data format, and recording the conversion rules.

[0013] According to another aspect of the present invention, a standard data file parsing apparatus is also provided, comprising: an acquisition unit, configured to acquire a standard data file and preprocess the standard data file to convert the content of the standard data file into a unified format, wherein the standard data file represents a document formulated based on rules of a domain; a parsing unit, configured to construct a heterogeneous graph for the preprocessed standard data file and perform hierarchical parsing and entry recognition based on the heterogeneous graph to obtain multiple data units; a calibration unit, configured to perform content recognition on each of the data units and calibrate the recognition results based on a knowledge graph of the domain to which the standard data file belongs to obtain standardized data units; and a generation unit, configured to perform structured extraction on the standardized data units, construct a semantic structure graph, and generate a structured data file based on the semantic structure graph.

[0014] Further, the acquisition unit includes: a first scanning module, used to scan the standard data file and identify different types of data content in the standard data file, wherein the types of data content include at least one of the following: image data, text data, and embedded tables; a first extraction module, used to extract the image data from the standard data file and convert the extracted image data into an ordered page image in a unified format; a second extraction module, used to extract the text data from the standard data file and convert the extracted text data into content text in a unified format; and a third extraction module, used to extract the embedded tables from the standard data file and convert the extracted embedded tables into a cell grid in a unified format.

[0015] Further, the parsing unit includes: a first segmentation module, used to segment the preprocessed standard data file according to physical pages to obtain multiple independent processing units; a first construction module, used to identify feature elements within the independent processing units and construct an initial heterogeneous graph using the feature elements as nodes, wherein the feature elements include at least one of the following: text blocks, image blocks, cell grids, titles, headers, and footers; and a second construction module, used to obtain the association relationships between each feature element in the initial heterogeneous graph and construct the heterogeneous graph using the association relationships between the feature elements as edges.

[0016] Further, the parsing unit includes: a first analysis module, used to analyze the hierarchical structure of the standard data file based on the header nodes in the heterogeneous graph; a first partitioning module, used to partition the standard data file based on the hierarchical structure to obtain multiple initial data units; a first identification module, used to identify entries in each initial data unit; and a first segmentation module, used to segment the initial data unit based on the identified entries to obtain multiple data units.

[0017] Furthermore, the parsing device for the standard data file further includes: a second identification module, used to identify cross-page anchor points in each of the data units; and a first recovery module, used to recover the separated data content based on the cross-page anchor points to obtain new data units.

[0018] Further, the calibration module includes: a third recognition module, used to perform content recognition on image blocks in each data unit using optical character recognition to obtain image content recognition results; a fourth recognition module, used to perform content recognition on text blocks in each data unit using semantic recognition to obtain text content recognition results; a fifth recognition module, used to perform content recognition on cell grids in each data unit using table recognition to obtain grid content recognition results; and a first matching module, used to obtain a knowledge graph of the domain to which the standard data file belongs, and to match the image content recognition results, the content recognition results, and the grid content recognition results respectively, and to calibrate the content within the data unit based on the matching results.

[0019] Further, the generation unit includes: a first extraction module, used to perform semantic recognition on each of the standardized data units, and extract entities from the standardized data units based on the semantic recognition results, wherein the entities include at least: target fields; a third construction module, used to construct the semantic structure graph with the entities as nodes and the relationships between entities as edges; and a first conversion module, used to determine the target data format to be converted, and convert the semantic structure graph into the structured data file based on the target data format, and record the conversion rules.

[0020] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute any of the above-described standard data file parsing methods.

[0021] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described standard data file parsing methods.

[0022] In this application, the following steps are performed: obtaining a standard data file and preprocessing the standard data file to convert the content in the standard data file into a unified format, wherein the standard data file represents a document formulated based on the rules of its respective domain; constructing a heterogeneous graph for the preprocessed standard data file; performing hierarchical parsing and entry recognition based on the heterogeneous graph to obtain multiple data units; then performing content recognition on each data unit; calibrating the recognition results based on the knowledge graph of the domain to which the standard data file belongs to obtain canonical data units; finally, performing structured extraction on the canonical data units to construct a semantic structure graph; and generating a structured data file based on the semantic structure graph.

[0023] In this application, an automated parsing process for standard data files is constructed. First, the content of the file is converted into a unified format. Then, a heterogeneous graph is constructed for layout parsing. Next, hierarchical parsing and item recognition are performed to split the data file into multiple data units. For each data unit, semantic parsing is performed, and knowledge graphs are used for content calibration to ensure readability. Finally, structured extraction is performed to automatically generate structured data files. This achieves automatic conversion of standard data files across different domains, improving the parsing efficiency of standard data files and ensuring the accuracy of the parsing results. This solves the technical problem in related technologies where standard data files rely on manual parsing and calibration, resulting in low parsing efficiency and low accuracy of the parsing results. Attached Figure Description

[0024] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0025] Figure 1 This is a flowchart of an optional standard data file parsing method according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic diagram of an optional standard data file parsing process according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic diagram of an optional standard data file parsing device according to an embodiment of the present invention;

[0028] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) that performs a method for parsing standard data files according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0032] JSON, short for JavaScript Object Notation, is a lightweight data interchange format that is easy for humans to read and write, and also easy for machines to parse and generate.

[0033] XML, short for eXtensible Markup Language, is a markup language-based data format used for transmitting and storing structured data.

[0034] CSV, short for Comma-Separated Values, is a commonly used data file format for storing tabular data.

[0035] It should be noted that the standard data file parsing method and apparatus in this application can be used in the field of data processing to parse standard data files, and can also be used in any field other than data processing to parse standard data files. This application does not limit the application field of the standard data file parsing method and apparatus.

[0036] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of such data all comply with the relevant laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding operation entry points are provided for users to choose to authorize or refuse. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0037] It should be noted that in this application, when collecting and analyzing customer information, users are provided with corresponding operation entry points to choose whether to agree to or reject the automated decision-making results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0038] The following embodiments of the present invention can be applied to various standard data file parsing systems / applications / devices. The present invention constructs an automated parsing process for standard data files, achieving robust parsing of complex formats and recovery from page truncation, structured extraction and semantic alignment of fields / types / value ranges / constraints / sources; reversible cross-standard conversions and type-preserving mapping, enabling automated parsing of standard data files, making the data readable, and improving the efficiency and accuracy of parsing standard data files.

[0039] The present invention will now be described in detail with reference to various embodiments.

[0040] Example 1

[0041] According to an embodiment of the present invention, an embodiment of a method for parsing a standard data file is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0042] Figure 1 This is a flowchart of an optional standard data file parsing method according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0043] Step S101: Obtain the standard data file and preprocess it to convert the content of the standard data file into a unified format.

[0044] In step S101 above, the standard data file represents a document formulated based on rules of its respective domain. Compiled according to rules of a specific industry or field, it standardizes the definition, format, and usage of data, ensuring data consistency and interoperability. Different industries or application scenarios have their own data standards. Parsing standard data files within a specific industry can bridge various different standards, enabling reversible and type-preserving data conversion. This means that even if the data format or standard changes, the integrity and consistency of the data information can be guaranteed. Secondly, through parsing standard data files, the system can clearly display information such as field definitions, data types, value ranges, and constraints in the data standard file, transforming it into a structured, easy-to-understand, and easy-to-operate format. The parsed and transformed standard data files can flow freely between different organizations, platforms, or systems without additional format adjustments or interpretations, greatly promoting the effective integration and utilization of data resources and supporting a data-driven business ecosystem.

[0045] The system receives standard data files, such as financial regulatory guidelines issued by various financial institutions, terminology documents in the medical industry, or other industry-specific standard documents. It then initiates a preprocessing workflow, which includes converting various file types, such as Word documents, PDF files, and scanned copies, into consistent and easily processed image sequences or text formats.

[0046] Furthermore, the steps for converting the content of the standard data file into a unified format include: scanning the standard data file to identify different types of data content, wherein the data content types include at least one of the following: image data, text data, and embedded tables; extracting image data from the standard data file and converting the extracted image data into ordered page images in a unified format; extracting text data from the standard data file and converting the extracted text data into content text in a unified format; and extracting embedded tables from the standard data file and converting the extracted embedded tables into cell grids in a unified format.

[0047] Specifically, the first step is to perform a comprehensive scan of the standard data file to identify the different types of data content it contains. This data content refers to image data, text data, and embedded tables within the file. Computer vision and pattern recognition algorithms can be used to accurately distinguish and locate data elements in the file, providing a data foundation for the next step of targeted extraction and transformation.

[0048] For the identified image data, the system activates the image processing module to accurately extract image information from standard data files. The extraction process takes into account the image's position, size, and layout features to ensure image integrity and contextual accuracy. Subsequently, an image conversion tool is used to convert the extracted image data into ordered page images in a uniform format. By setting consistent resolution and color modes, all images are guaranteed to have the same display quality in subsequent processing, facilitating analysis and comparison.

[0049] For text data processing, the system employs a text extraction component to accurately extract the text content from the file, while ignoring irrelevant formatting and layout information. The extracted text data is then converted into content text in a uniform format to facilitate subsequent natural language processing and structured analysis. In this step, the system may apply text cleaning algorithms to remove whitespace, special characters, and unnecessary layout marks, retaining only pure text information.

[0050] For complex embedded tables in the file, the system employs form analysis and extraction technology to accurately identify table boundaries, cell structure, and data filling. The extracted table data is reorganized into a uniformly formatted grid. The grid conversion process emphasizes the data arrangement order and relationships to ensure that the information extracted from the table can be correctly interpreted and used.

[0051] Step S102 involves constructing a heterogeneous graph for the preprocessed standard data file, and performing hierarchical parsing and entry identification based on the heterogeneous graph to obtain multiple data units.

[0052] In step S102 above, a heterogeneous graph is constructed to analyze the layout of the standard data file. Based on this graph, hierarchical parsing and item identification are performed. The purpose of hierarchical parsing is to identify document elements at different levels and determine their hierarchical relationships, such as which paragraphs belong to a certain heading, or which row in a table corresponds to a specific data field. Combining layout features and hierarchical structure, the system can identify and separate specific items in the standard data file, such as data field definitions, constraints, and code set lists. Through hierarchical analysis and item identification, the standard data file is divided into multiple data units, facilitating more accurate content parsing in the subsequent process.

[0053] Furthermore, the steps for constructing a heterogeneous graph for the preprocessed standard data file include: dividing the preprocessed standard data file according to physical pages to obtain multiple independent processing units; identifying feature elements within the independent processing units and constructing an initial heterogeneous graph using the feature elements as nodes, wherein the feature elements include at least one of the following: text blocks, image blocks, cell grids, titles, headers, and footers; obtaining the relationships between each feature element in the initial heterogeneous graph and constructing a heterogeneous graph using the relationships between feature elements as edges.

[0054] Specifically, the preprocessed standard data file is first segmented according to its original physical pages, dividing the entire file into multiple independent processing units. This segmentation process is automatically completed based on page boundaries and layout features, ensuring that each unit contains complete page content, facilitating accurate processing in subsequent steps. The content within each independent processing unit is analyzed in depth to identify key feature elements constituting the page information, including text blocks, image blocks, cell grids (i.e., the extracted table structure), titles, headers, and footers. These feature elements will serve as nodes in the heterogeneous graph in subsequent steps. Based on the identified feature elements, an initial heterogeneous graph structure is created. In the graph, each feature element is represented as a node; initially, the heterogeneous graph only reflects the information of the feature elements themselves, without considering the relationships between them. Subsequently, the logical and spatial relationships between the feature elements are analyzed, such as the correspondence between text blocks and titles, the association between image blocks and explanatory text, and the link between cell grids and footer information. These relationships become edges in the heterogeneous graph, used to express the interdependence and interaction between nodes. After identifying the relationships between feature elements, these relationships are used as edges in the graph to connect the nodes, thus constructing a complete heterogeneous graph. The heterogeneous graph comprehensively reflects the page content structure and information flow, serving as a crucial foundation for subsequent in-depth data analysis and information extraction.

[0055] Furthermore, the steps of performing hierarchical parsing and entry identification based on the heterogeneous graph to obtain multiple data units include: analyzing the hierarchical structure of the standard data file based on the header nodes in the heterogeneous graph; dividing the standard data file based on the hierarchical structure to obtain multiple initial data units; identifying the entries in each initial data unit; and segmenting the initial data unit based on the identified entries to obtain multiple data units.

[0056] Specifically, after the heterogeneous graph is constructed, the system immediately begins in-depth hierarchical structure analysis based on the title nodes in the graph, thereby precisely dividing the standard data file, identifying key entries, and further segmenting it into multiple data units. First, the system focuses on the title nodes in the heterogeneous graph, as titles often indicate chapters or sections of the document. The algorithm reconstructs the hierarchical structure of the standard data file by analyzing the level, numbering, and position information of these titles. This analysis is based on the hierarchical relationships between title nodes, such as the nesting of main and subheadings, and the proximity of titles and content. Based on the reconstructed hierarchical structure, the system divides the standard data file into multiple smaller, logically related initial data units. Each unit typically contains a title and its subordinate content; this division facilitates refined processing and reduces the possibility of information confusion. For each initial data unit, the system uses deep learning techniques and natural language processing methods to identify key information within the unit, i.e., entries. Entries may include field definitions, data type descriptions, constraints, value ranges, etc. Subsequently, based on the boundaries and contextual information of the entries, the initial data units are further segmented, ultimately resulting in a series of independent and complete data units. Each data unit is closely tied to a single theme or function, facilitating subsequent semantic analysis and standardization.

[0057] Furthermore, after obtaining multiple data units, the process also includes: for each data unit, identifying cross-page anchors within the data unit; and restoring the separated data content based on the cross-page anchors to obtain new data units.

[0058] Specifically, after completing the item identification and data unit segmentation based on heterogeneous graphs, cross-page anchor point recovery is introduced to further improve the integrity and accuracy of data units, solving the data content segmentation problem caused by physical page limitations. Each segmented data unit is analyzed in depth to identify the segmented items, including those at physical page boundaries, those at image or table boundaries, or those related across different physical pages. By analyzing contextual information and potential item integrity, the system can identify cross-page anchor points. After identifying cross-page anchor points, the system automatically splices these segmented data contents, ensuring the integrity of each item's information. This recovery process, based on the logical order and semantic coherence of the items, achieves accurate reconstruction of data units by matching the context and features of cross-page anchor points. Through cross-page anchor point recovery, the originally segmented data content is merged to form more complete and coherent new data units. These new units not only contain all the necessary information but also maintain the internal logical coherence of the items, improving the accuracy and efficiency of data processing.

[0059] Step S103: Content recognition is performed on each data unit, and the recognition results are calibrated based on the knowledge graph of the domain to which the standard data file belongs, to obtain the standardized data unit.

[0060] In step S103 above, each data unit undergoes deep content recognition and is calibrated using a knowledge graph from the relevant domain to ensure the standardization and accuracy of the data units. Through content recognition, domain knowledge graph calibration, and the generation of standardized data units, gaps are filled in the standard data file.

[0061] Furthermore, the steps of performing content recognition on each data unit and calibrating the recognition results based on the knowledge graph of the domain to which the standard data file belongs include: using optical character recognition to perform content recognition on image blocks in each data unit to obtain image content recognition results; using semantic recognition to perform content recognition on text blocks in each data unit to obtain text content recognition results; using table recognition to perform content recognition on cell grids in each data unit to obtain grid content recognition results; obtaining the knowledge graph of the domain to which the standard data file belongs, matching the image content recognition results, content recognition results, and grid content recognition results respectively, and calibrating the content within the data unit based on the matching results.

[0062] Specifically, in the process of content recognition and calibration of each data unit of the standard data file, the system adopts a multimodal information processing method to recognize images, text and tables respectively, and uses domain knowledge graphs for calibration to ensure the accuracy and standardization of the recognition results.

[0063] For image blocks within a data unit, the system employs optical character recognition (OCR) technology to perform content recognition, converting the text in the image into text format to obtain the image content recognition result. OCR can identify and extract character and text information from images, providing a high recognition rate even against complex backgrounds or with non-standard fonts.

[0064] Deep semantic recognition is performed on text blocks within data units. In this stage, the system not only focuses on the surface information of the text but also delves into its underlying meaning, such as field definitions and data type descriptions, to obtain text content recognition results. Semantic recognition utilizes natural language processing techniques and deep learning models to parse the semantic structure of text and identify and classify information blocks.

[0065] For the cell grid within the data unit, the system employs specialized table recognition technology to extract table structure and cell information, including row and column boundaries, cell content, and table headers, to obtain the grid content recognition results. This table recognition technology can accurately parse the table structure and identify the data in different cells and their corresponding headers.

[0066] A knowledge graph related to the domain of the standard data file is acquired, and the image content recognition results, text content recognition results, and grid content recognition results are matched and calibrated with the entries in the knowledge graph. This process utilizes domain terminology, conceptual relationships, and standard definitions in the knowledge graph to correct errors in the recognition results, supplement missing information, standardize terminology, and ensure the standardization and accuracy of the recognition results.

[0067] Step S104: Extract the standardized data units in a structured manner, construct a semantic structure graph, and generate a structured data file based on the semantic structure graph.

[0068] In step S104 above, these standardized data units are further transformed into structured data, semantic analysis is performed on the standardized data units, a detailed semantic structure diagram is constructed, and a structured data file in the target data format is generated based on the semantic structure diagram, thereby realizing the conversion of cross-domain standard data files.

[0069] Furthermore, the steps of extracting structured data units from the standardized data units, constructing a semantic structure graph, and generating a structured data file based on the semantic structure graph include: performing semantic recognition on each standardized data unit, and extracting entities from the standardized data units based on the semantic recognition results, wherein the entities include at least the target field; constructing a semantic structure graph with entities as nodes and the relationships between entities as edges; determining the target data format to be converted, converting the semantic structure graph into a structured data file based on the target data format, and recording the conversion rules.

[0070] Specifically, deep semantic recognition is performed on each standardized data unit to identify and extract key entities. These entities include at least the target field, but may also include data types, constraints, code sets, and other metadata. This process is the foundation of structured extraction, ensuring the system accurately understands the meaning and components of each data unit. The extracted entities appear as nodes in the semantic structure graph, and the relationships between entities are connected by edges. For example, the correspondence between a field and a data type is represented by an edge connecting the field node and the data type node. The purpose of constructing the semantic structure graph is to capture and visualize the logical structure and entity relationships within the data unit, providing clear guidance for subsequent data file generation. The system needs to specify the format of the structured data files to be generated, such as JSON, XML, CSV, or any other format that can be read and processed by an automated system. The choice of target data format depends on specific application requirements and system compatibility. Based on the selected target data format and semantic structure graph, the system performs transformation operations to generate structured data files. Simultaneously, the system records the specific rules for each transformation, including how to extract information from the graph and how to map entity relationships to the new data format. These rules are crucial for subsequent data inverse transformation, maintenance, and auditing.

[0071] In this embodiment of the invention, automated parsing and format conversion of standard data files significantly improves the efficiency and accuracy of converting data units into structured data. By constructing a semantic structure graph, not only is the inherent logic of the data units captured, but the information is also clearly expressed in the structured data file. Furthermore, the recording of conversion rules ensures the reversibility of the standard data file conversion.

[0072] Through the above steps, a standard data file is obtained and preprocessed to transform its content into a unified format. The standard data file represents a document based on rules defined in its domain. A heterogeneous graph is constructed for the preprocessed standard data file, and hierarchical parsing and entry recognition are performed based on the heterogeneous graph to obtain multiple data units. Then, content recognition is performed on each data unit, and the recognition results are calibrated based on the knowledge graph of the domain to which the standard data file belongs to obtain canonical data units. Finally, the canonical data units are structurally extracted to construct a semantic structure graph, and a structured data file is generated based on the semantic structure graph.

[0073] In this embodiment, an automated parsing process for standard data files is constructed. First, the content of the file is converted into a unified format. Then, a heterogeneous graph is constructed for layout parsing. Next, hierarchical parsing and item recognition are performed to split the data file into multiple data units. For each data unit, semantic parsing is performed, and knowledge graphs are used for content calibration to ensure readability. Finally, structured extraction is performed to automatically generate structured data files. This achieves automatic conversion of standard data files across different domains, improving the parsing efficiency of standard data files and ensuring the accuracy of the parsing results. This solves the technical problem in related technologies where standard data files rely on manual parsing and calibration, resulting in low parsing efficiency and low accuracy of the parsing results.

[0074] The following describes in detail another optional implementation method.

[0075] Figure 2 This is a schematic diagram of an optional standard data file parsing process according to an embodiment of the present invention, such as... Figure 2 As shown, the standard data file parsing process specifically includes:

[0076] Step 1, Begin;

[0077] Step two, ensure consistent formatting;

[0078] Convert Word documents, PDFs, and scanned documents into ordered page images and usable text layers, and extract embedded tables into a cell grid structure.

[0079] Step 3: Hierarchy and Item Resolution;

[0080] Construct a heterogeneous diagram of page-block-cell structure, combine title numbering, table header level, and footer / page span markers to perform hierarchical parsing and item identification, and output multiple data units.

[0081] Step 4: Restore cross-page anchor points;

[0082] Step 5: Knowledge graph standardization;

[0083] The optical character recognition candidate is subjected to language model-in-the-loop constrained decoding, which is limited by industry dictionaries, code sets, types and value ranges. The standardized data units are recovered by knowledge graph matching.

[0084] Step six, structured extraction;

[0085] Based on a preset template, semantic content such as field definition, type, value range, constraint, remarks, and source is extracted in a structured manner and mapped to nodes and edges (field-type, field-constraint, field-code set, field-source, field-relationship) in a semantic structure graph.

[0086] Step 7, cross-standard reversible mapping;

[0087] Record conversion rules to enable reversible conversion and playback verification of standard data files.

[0088] Step 8: Export and save;

[0089] Export the results as structured JSON / database tables and combine them with the transformation records to generate a chain of evidence archive.

[0090] Step nine, end.

[0091] This invention constructs an automated parsing process for standard data files, achieving robust parsing of complex layouts and recovery from page truncation, structured extraction and semantic alignment of fields / types / value domains / constraints / sources; and reversible, type-preserving mapping across standards, enabling automated parsing of standard data files, making the data readable, and improving the efficiency and accuracy of parsing standard data files.

[0092] The following is a detailed description with reference to another embodiment.

[0093] Example 2

[0094] The standard data file parsing device provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in the above embodiment one. The specific implementation method and beneficial effects can be referred to the foregoing method embodiment, and will not be repeated here.

[0095] Figure 3 This is a schematic diagram of an optional standard data file parsing device according to an embodiment of the present invention, such as... Figure 3 As shown, the parsing device for the standard data file may include: an acquisition unit 31, a parsing unit 32, a calibration unit 33, and a generation unit 34, wherein,

[0096] The acquisition unit 31 is used to acquire a standard data file and preprocess the standard data file to convert the content of the standard data file into a unified format. The standard data file represents a document formulated based on the rules of the relevant domain.

[0097] The parsing unit 32 is used to construct a heterogeneous graph for the preprocessed standard data file, and perform hierarchical parsing and entry identification based on the heterogeneous graph to obtain multiple data units;

[0098] The calibration unit 33 is used to perform content recognition on each data unit and calibrate the recognition results based on the knowledge graph of the domain to which the standard data file belongs, so as to obtain the standardized data unit.

[0099] The generation unit 34 is used to extract the standardized data units in a structured manner, construct a semantic structure graph, and generate a structured data file based on the semantic structure graph.

[0100] The aforementioned standard data file parsing device acquires the standard data file through acquisition unit 31 and preprocesses it to convert the content into a unified format. The standard data file represents a document based on rules of its domain. Parsing unit 32 constructs a heterogeneous graph for the preprocessed standard data file and performs hierarchical parsing and entry recognition based on the heterogeneous graph to obtain multiple data units. Calibration unit 33 performs content recognition on each data unit and calibrates the recognition results based on the knowledge graph of the domain to which the standard data file belongs to, obtaining standardized data units. Generation unit 34 performs structured extraction on the standardized data units, constructs a semantic structure graph, and generates a structured data file based on the semantic structure graph.

[0101] In this embodiment, an automated parsing process for standard data files is constructed. First, the content of the file is converted into a unified format. Then, a heterogeneous graph is constructed for layout parsing. Next, hierarchical parsing and item recognition are performed to split the data file into multiple data units. For each data unit, semantic parsing is performed, and knowledge graphs are used for content calibration to ensure readability. Finally, structured extraction is performed to automatically generate structured data files. This achieves automatic conversion of standard data files across different domains, improving the parsing efficiency of standard data files and ensuring the accuracy of the parsing results. This solves the technical problem in related technologies where standard data files rely on manual parsing and calibration, resulting in low parsing efficiency and low accuracy of the parsing results.

[0102] Furthermore, the acquisition unit includes: a first scanning module for scanning a standard data file and identifying different types of data content in the standard data file, wherein the data content types include at least one of the following: image data, text data, and embedded tables; a first extraction module for extracting image data from the standard data file and converting the extracted image data into ordered page images in a unified format; a second extraction module for extracting text data from the standard data file and converting the extracted text data into content text in a unified format; and a third extraction module for extracting embedded tables from the standard data file and converting the extracted embedded tables into cell grids in a unified format.

[0103] Furthermore, the parsing unit includes: a first segmentation module, used to segment the preprocessed standard data file according to physical pages to obtain multiple independent processing units; a first construction module, used to identify feature elements within the independent processing units and construct an initial heterogeneous graph using the feature elements as nodes, wherein the feature elements include at least one of the following: text blocks, image blocks, cell grids, titles, headers, and footers; and a second construction module, used to obtain the association relationships between each feature element in the initial heterogeneous graph and construct the heterogeneous graph using the association relationships between feature elements as edges.

[0104] Furthermore, the parsing unit includes: a first analysis module for analyzing the hierarchical structure of the standard data file based on the header nodes in the heterogeneous graph; a first partitioning module for partitioning the standard data file based on the hierarchical structure to obtain multiple initial data units; a first identification module for identifying entries in each initial data unit; and a first segmentation module for segmenting the initial data unit based on the identified entries to obtain multiple data units.

[0105] Furthermore, the parsing device for standard data files also includes: a second identification module for identifying cross-page anchors in each data unit; and a first recovery module for recovering the separated data content based on the cross-page anchors to obtain new data units.

[0106] Furthermore, the calibration module includes: a third recognition module, used to perform content recognition on image blocks in each data unit using optical character recognition, to obtain image content recognition results; a fourth recognition module, used to perform content recognition on text blocks in each data unit using semantic recognition, to obtain text content recognition results; a fifth recognition module, used to perform content recognition on cell grids in each data unit using table recognition, to obtain grid content recognition results; and a first matching module, used to obtain a knowledge graph of the domain to which the standard data file belongs, and to match the image content recognition results, content recognition results, and grid content recognition results respectively, and to calibrate the content within the data unit based on the matching results.

[0107] Furthermore, the generation unit includes: a first extraction module, used to perform semantic recognition on each standardized data unit and extract entities from the standardized data unit based on the semantic recognition results, wherein the entity includes at least: a target field; a third construction module, used to construct a semantic structure graph with entities as nodes and the relationships between entities as edges; and a first transformation module, used to determine the target data format to be transformed, and to transform the semantic structure graph into a structured data file based on the target data format, and record the transformation rules.

[0108] The above-mentioned standard data file parsing device may also include a processor and a memory. The above-mentioned acquisition unit 31, parsing unit 32, calibration unit 33, generation unit 34, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0109] The processor described above contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and their parameters can be adjusted to parse standard data files.

[0110] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0111] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the parsing method of any of the above-mentioned standard data files.

[0112] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described standard data file parsing methods.

[0113] According to another aspect of the present invention, a computer program product is also provided, the computer program product including a computer program, wherein the computer program, when executed by a processor, implements the parsing method of any of the above-mentioned standard data files.

[0114] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: acquiring a standard data file and preprocessing the standard data file to convert the content of the standard data file into a unified format, wherein the standard data file represents a document formulated based on rules of its domain; constructing a heterogeneous graph for the preprocessed standard data file, and performing hierarchical parsing and entry recognition based on the heterogeneous graph to obtain multiple data units; performing content recognition on each data unit, and calibrating the recognition results based on the knowledge graph of the domain to which the standard data file belongs to obtain canonical data units; performing structured extraction on the canonical data units, constructing a semantic structure graph, and generating a structured data file based on the semantic structure graph.

[0115] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: scanning a standard data file, identifying different types of data content in the standard data file, wherein the types of data content include at least one of the following: image data, text data, and embedded tables; extracting image data from the standard data file and converting the extracted image data into ordered page images in a uniform format; extracting text data from the standard data file and converting the extracted text data into content text in a uniform format; and extracting embedded tables from the standard data file and converting the extracted embedded tables into cell grids in a uniform format.

[0116] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: dividing a preprocessed standard data file according to physical pages to obtain multiple independent processing units; identifying feature elements within the independent processing units and constructing an initial heterogeneous graph using the feature elements as nodes, wherein the feature elements include at least one of the following: text blocks, image blocks, cell grids, titles, headers, and footers; obtaining the association relationships between each feature element in the initial heterogeneous graph and constructing the heterogeneous graph using the association relationships between the feature elements as edges.

[0117] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: analyzing the hierarchical structure of a standard data file based on the header nodes in a heterogeneous graph; dividing the standard data file based on the hierarchical structure to obtain multiple initial data units; for each initial data unit, identifying entries in the initial data unit; and dividing the initial data unit based on the identified entries to obtain multiple data units.

[0118] This application also provides a computer program product that, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: for each data unit, identifying cross-page anchors in the data unit; and restoring the separated data content based on the cross-page anchors to obtain a new data unit.

[0119] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: using optical character recognition to perform content recognition on image blocks in each data unit to obtain image content recognition results; using semantic recognition to perform content recognition on text blocks in each data unit to obtain text content recognition results; using table recognition to perform content recognition on cell grids in each data unit to obtain grid content recognition results; obtaining a knowledge graph of the domain to which the standard data file belongs; matching the image content recognition results, content recognition results, and grid content recognition results respectively; and calibrating the content within the data unit based on the matching results.

[0120] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: performing semantic recognition for each canonical data unit, and extracting entities from the canonical data unit based on the semantic recognition results, wherein the entities include at least: target fields; constructing a semantic structure graph with entities as nodes and the relationships between entities as edges; determining the target data format to be converted, and converting the semantic structure graph into a structured data file based on the target data format, and recording the conversion rules.

[0121] Figure 4 This is a hardware structure block diagram of an electronic device (or mobile device) that performs a method for parsing standard data files according to an embodiment of the present invention. Figure 4 As shown, an electronic device may include one or more processors ( Figure 4 The processor, denoted by 402a, 402b, ..., 402n, can include, but is not limited to, a processing device such as a microprocessor (MCU) or a programmable logic device (FPGA), and a memory 404 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports in the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 4 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 4 The more or fewer components shown, or having the same Figure 4 The different configurations shown.

[0122] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0123] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0124] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0125] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0126] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0127] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0128] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for parsing a standard data file, characterized in that, include: Obtain a standard data file and preprocess the standard data file to convert the content of the standard data file into a unified format, wherein the standard data file represents a document formulated based on the rules of the relevant domain; A heterogeneous graph is constructed for the preprocessed standard data file, and hierarchical parsing and entry identification are performed based on the heterogeneous graph to obtain multiple data units; Content recognition is performed on each of the data units, and the recognition results are calibrated based on the knowledge graph of the domain to which the standard data file belongs, to obtain the standardized data units; The standardized data units are extracted in a structured manner to construct a semantic structure graph, and a structured data file is generated based on the semantic structure graph.

2. The method according to claim 1, characterized in that, The steps for converting the contents of the standard data file into a unified format include: The standard data file is scanned to identify different types of data content within the standard data file, wherein the types of data content include at least one of the following: image data, text data, and embedded tables; The image data is extracted from the standard data file and converted into an ordered page image in a unified format. Extract the text data from the standard data file and convert the extracted text data into content text in a unified format; The embedded table is extracted from the standard data file and converted into a uniformly formatted cell grid.

3. The method according to claim 1, characterized in that, The steps for constructing a heterogeneous graph for the preprocessed standard data file include: The preprocessed standard data file is divided into multiple independent processing units according to physical pages; Identify the feature elements within the independent processing unit, and construct an initial heterogeneous graph using the feature elements as nodes, wherein the feature elements include at least one of the following: text block, image block, cell grid, title, header, and footer; Obtain the association relationships between each feature element in the initial heterogeneous graph, and construct the heterogeneous graph using the association relationships between the feature elements as edges.

4. The method according to claim 1, characterized in that, The steps of performing hierarchical parsing and entry identification based on the heterogeneous graph to obtain multiple data units include: Analyze the hierarchical structure of the standard data file based on the header nodes in the heterogeneous graph; The standard data file is divided based on the hierarchical structure to obtain multiple initial data units; For each of the initial data units, identify the entries in the initial data unit; The initial data unit is segmented based on the identified entries to obtain multiple data units.

5. The method according to claim 1, characterized in that, After obtaining multiple data units, it also includes: For each of the data units, identify the cross-page anchor points within the data unit; Based on the cross-page anchor point, the separated data content is restored to obtain the new data unit.

6. The method according to claim 1, characterized in that, The steps of performing content recognition on each of the data units and calibrating the recognition results based on the knowledge graph of the domain to which the standard data file belongs include: Optical character recognition is used to perform content recognition on the image blocks in each of the data units to obtain the image content recognition result; Semantic recognition is used to identify the content of text blocks in each data unit to obtain text content recognition results; The content of the cell grid in each of the data units is identified by using table recognition to obtain the grid content recognition result; Obtain the knowledge graph of the domain to which the standard data file belongs, match the image content recognition result, the content recognition result, and the grid content recognition result respectively, and calibrate the content within the data unit based on the matching result.

7. The method according to claim 1, characterized in that, The steps of extracting the standardized data units in a structured manner, constructing a semantic structure graph, and generating a structured data file based on the semantic structure graph include: Semantic recognition is performed on each of the aforementioned standardized data units, and entities are extracted from the standardized data units based on the semantic recognition results, wherein the entities include at least: target fields; The semantic structure graph is constructed using the entities as nodes and the relationships between entities as edges. Determine the target data format to be converted, and based on the target data format, convert the semantic structure graph into the structured data file, and record the conversion rules.

8. A standard data file parsing device, characterized in that, include: An acquisition unit is used to acquire a standard data file and preprocess the standard data file to convert the content of the standard data file into a unified format, wherein the standard data file represents a document formulated based on the rules of its respective domain; The parsing unit is used to construct a heterogeneous graph for the preprocessed standard data file, and perform hierarchical parsing and entry identification based on the heterogeneous graph to obtain multiple data units; A calibration unit is used to perform content recognition on each of the data units and to calibrate the recognition results based on the knowledge graph of the domain to which the standard data file belongs, so as to obtain a standardized data unit. The generation unit is used to perform structured extraction of the standardized data units, construct a semantic structure graph, and generate a structured data file based on the semantic structure graph.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the parsing method for a standard data file as described in any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method for parsing a standard data file as described in any one of claims 1 to 7.

Images