Unstructured standard file conversion method and device, equipment, medium and product

Abstract

The invention discloses an unstructured standard file conversion method and device, equipment, a medium and a product, and relates to the technical field of file conversion, the method comprises the following steps: analyzing records of an original standard file output by an OCR tool line by line to obtain a first intermediate result file; performing in-page text cleaning and row reconstruction on the first intermediate result file to obtain a second intermediate result file; performing page type judgment and global section recognition on the second intermediate result file to obtain a third intermediate result file; performing cross-page splicing and chapter tree construction on the third intermediate result file to obtain a fourth intermediate result file; positioning a target chapter based on a structure anchor point of the third intermediate result file, and extracting term entries and technical contents in combination with the fourth intermediate result file to correspondingly obtain a fifth intermediate result file and a sixth intermediate result file; and mapping each intermediate result file and executing consistency verification, and obtaining a total element structured file after the consistency verification is passed. According to the method and the device, total-element structured processing can be performed on the file.

Description

Technical Field

[0001] This application relates to the field of document conversion technology, and in particular to a method, apparatus, device, medium and product for converting unstructured standard documents. Background Technology

[0002] With the widespread application of standardization work in fields such as public safety, energy, and emergency management, the number of standard documents continues to surge. Their content structure is complex and their format is highly standardized. They are mostly published in PDF or scanned document format, covering multiple core components such as cover, table of contents, preface, technical clauses, and appendices.

[0003] Existing standard digitization methods use OCR algorithms to directly process documents into PDFs or plain text, which only reaches the table of contents or chapter level and cannot be broken down into full-element data such as terminology entries and technical clauses. This makes it difficult to support in-depth applications such as knowledge base construction and compliance verification. Therefore, there is an urgent need to provide a method that can perform full-element structured processing on standard documents recognized by OCR algorithms. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the related technologies, the purpose of this application is to provide a method, apparatus, device, medium and product for converting unstructured standard documents, which can perform full-element structuring processing on PDF / scanned documents and provide data support for subsequent in-depth applications such as knowledge base construction and compliance verification.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for converting unstructured standard documents, comprising: identifying the standard document to be processed using an OCR tool to obtain an original standard document; the original standard document is in JSONL format; each record in the original standard document includes a page identifier; parsing the records of the original standard document line by line to extract the page identifier, aggregating and constructing a page-level data structure according to the page identifier to obtain a first intermediate result document; cleaning and reconstructing the text within the pages of the first intermediate result document to obtain a second intermediate result document; determining the page type and identifying global segments in the second intermediate result document to obtain a third intermediate result document; performing cross-page splicing and chapter tree construction on the third intermediate result document to obtain a fourth intermediate result document; locating the target chapter based on the structural anchor points of the third intermediate result document, and extracting terminology entries and technical content in conjunction with the fourth intermediate result document to obtain a fifth intermediate result document and a sixth intermediate result document; mapping the first, second, third, fourth, fifth, and sixth intermediate result documents and performing consistency checks, and obtaining a fully structured document after passing the checks.

[0006] Optionally, the step of parsing the records of the original standard file line by line to extract page identifiers, and aggregating and constructing a page-level data structure according to the page identifiers to obtain a first intermediate result file includes: performing JSON deserialization processing on each record of the original standard file, parsing and extracting the page identifier and text field in each record; during the extraction process, if the current record is missing text coordinate information, only the text field and page identifier of the current record are retained; if the current record is missing a page identifier, the page number is inferred according to the line order of the current record in the original standard file according to the continuously increasing rule, and used as the page identifier of the current record; aggregating all records corresponding to the same extracted page identifier, associating the aggregated records with the same page-level data structure, and supplementing the page meta-information in the same page-level data structure to obtain the page-level data structure; the page meta-information includes page width and height, number of text blocks, average confidence of text units, and total number of characters on the page; summarizing all the page-level data structures and sorting them according to the page identifier order to obtain the first intermediate result file.

[0007] Optionally, the step of cleaning and reconstructing the text within the first intermediate result file to obtain the second intermediate result file includes: extracting all page-level data structures from the first intermediate result file to obtain the text fields, text coordinate information, and page metadata contained in each page-level data structure; performing text normalization cleaning on the text fields in each page-level data structure; performing logical line reconstruction on the normalized and cleaned text fields based on the text coordinate information; marking the noise in the first intermediate result file based on the page metadata and text distribution characteristics; verifying the page-level data structure of each page that has completed text cleaning, logical line reconstruction, and removal of headers and footers; and obtaining the second intermediate result file when it is confirmed that the logical line order is consistent with the actual page layout, that there are no missing texts after cleaning, and that the noise marking is accurate.

[0008] Optionally, the step of performing page type determination and global segment identification on the second intermediate result file to obtain a third intermediate result file includes: generating a corresponding page feature vector for each page-level data structure; the page feature vector includes text features and layout features; determining the page type page by page based on the page feature vector and corresponding determination rules and marking the page type to obtain the corresponding page type determination result; dividing the global segment and clarifying the page range of each segment based on the type determination results of all pages and the page identifier order; marking the key segments and core positions of the global segment with structural anchor points and recording the page identifier and logical line position corresponding to the structural anchor points; summarizing the page type annotations, global segment ranges and structural anchor point positions of all pages to generate the third intermediate result file.

[0009] Optionally, the step of performing cross-page splicing and chapter tree construction on the third intermediate result file to obtain the fourth intermediate result file includes: locating the start and end page identifiers corresponding to the main text segments based on the page type labeling and global segment range, and filtering out all page-level data structures labeled as main text pages; arranging the normalized logical lines of the filtered main text pages in the continuous order of the page identifiers to form a preliminary main text sequence; verifying the main text sequence, and correcting the connection deviations after confirming that there are no missing texts, no duplicate splicing, and no semantic breaks, to form a complete and continuous main text sequence; constructing a chapter tree according to hierarchical rules based on the complete and continuous main text sequence and structural anchor points; summarizing the chapter trees of all pages, complete main text blocks, and the source page numbers corresponding to each text block, and sorting them according to the chapter hierarchy to generate the fourth intermediate result file.

[0010] Optionally, the step of locating the target chapter based on the structural anchor points of the third intermediate result file, and extracting terminology entries and technical content from the fourth intermediate result file to obtain the corresponding fifth and sixth intermediate result files, includes: extracting the structural anchor point positions, global section ranges, and page type annotations from the third intermediate result file to determine the terminology chapters; extracting the main text blocks and source page numbers corresponding to the terminology chapters from the fourth intermediate result file to obtain the fifth intermediate result file; and extracting the technical content from the fourth intermediate result file within the scope of technical chapters in the main text section and constructing structured data to generate the sixth intermediate result file.

[0011] Secondly, this application provides a device for converting unstructured standard documents, comprising: The acquisition module is used to identify the standard file to be processed based on an OCR tool to obtain the original standard file; the original standard file is in JSONL format; each record in the original standard file includes a page identifier; The parsing and extraction module is used to parse the records of the original standard file line by line to extract page identifiers, aggregate and construct page-level data structures according to the page identifiers, and obtain the first intermediate result file. The cleaning and reconstruction module is used to clean the text within the pages and reconstruct the lines of the first intermediate result file to obtain the second intermediate result file. The determination and identification module is used to determine the page type and global segment identification of the second intermediate result file to obtain the third intermediate result file; The splicing and building module is used to perform cross-page splicing and chapter tree construction on the third intermediate result file to obtain the fourth intermediate result file; The positioning and extraction module is used to locate the target chapter based on the structural anchor points of the third intermediate result file, and extract terminology entries and technical content in combination with the fourth intermediate result file to obtain the fifth and sixth intermediate result files. The mapping generation module is used to map the first intermediate result file, the second intermediate result file, the third intermediate result file, the fourth intermediate result file, the fifth intermediate result file, and the sixth intermediate result file and perform consistency verification. After the verification passes, a full-feature structured file is obtained.

[0012] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method for converting unstructured standard documents as described above.

[0013] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for converting unstructured standard documents as described above.

[0014] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method for converting unstructured standard files as described above.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, apparatus, device, medium, and product for converting unstructured standard documents. The method involves: identifying the standard document to be processed using an OCR tool to obtain the original standard document; extracting page identifiers by parsing the records of the original standard document line by line, and aggregating and constructing a page-level data structure according to the page identifiers to obtain a first intermediate result file; cleaning and reconstructing the text within the pages of the first intermediate result file to obtain a second intermediate result file; determining the page type and identifying global segments in the second intermediate result file to obtain a third intermediate result file; performing cross-page splicing and chapter tree construction on the third intermediate result file to obtain a fourth intermediate result file; locating the target chapter using the structural anchor points of the third intermediate result file, and extracting terminology and technical content from the fourth intermediate result file to obtain a fifth and sixth intermediate result file; and mapping the first, second, third, and fourth intermediate result files... The process involves generating three intermediate result files: the fifth intermediate result file, the sixth intermediate result file, and performing consistency checks. Upon successful verification, a fully structured file is obtained. On one hand, from parsing and aggregating the original JSONL format file, to sequentially completing text cleaning and reconstruction, page segment identification, chapter structure construction, core element extraction, and finally, full intermediate file mapping and verification, the data flow trajectory can be traced. Those skilled in the art can directly implement the process, significantly reducing the difficulty of technology implementation and replication, and solving the problems of disordered processes and chaotic data connections in traditional processing methods. On the other hand, it fully preserves all the essential information of the standard file, including page metadata, text content, hierarchical structure, terminology entries, and technical content, overcoming the shortcomings of incomplete structuring and extraction of only single content in traditional processing. The output fully structured file can be directly adapted to subsequent application scenarios such as digital management, intelligent retrieval, content reuse, and data mining of standard files, significantly improving the efficiency and value of digital utilization of unstructured standard files. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for converting unstructured standard documents according to an embodiment of this application; Figure 2 A schematic diagram of the functional modules of an unstructured standard document conversion device provided in an embodiment of this application; Figure 3This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] In one exemplary embodiment, such as Figure 1 As shown, a method for converting unstructured standard files is provided. This method is executed by a computer device. In this embodiment, the method includes the following steps S101 to S106. Wherein: Step S101: Based on the OCR tool, the standard file to be processed is identified to obtain the original standard file.

[0021] In the example embodiment, the standard file to be processed is a PDF or a scanned copy, which is required to be clear, without severe blurring, obstruction or damage, to ensure the recognition accuracy of the OCR tool; the original standard file is in JSONL format, and each record in the original standard file includes a page identifier.

[0022] It should be noted that the purpose of this application is to convert the standard documents to be processed into standard documents that conform to GB / T37967-2019 "Framework for Structured Markup of National Standards Based on XML", that is, to generate a full-element structured JSON file that conforms to the requirements of GB / T37967-2019 from PDF or scanned documents. It fully includes six core sections: basic information, auxiliary information, terminology elements, technical content elements, editing information, and implementation information, ensuring that the data is machine-readable, queryable, reusable, and can be accurately traced back to the original standard documents.

[0023] The OCR tool in this application supports outputting JSONL format, which includes at least page_id and text fields, and preferably supports bbox (coordinates), score (confidence level), and block_type (text block type) fields, such as TesseractOCR Advanced Edition and Baidu Smart Cloud OCR.

[0024] The unified encoding format of this application is UTF-8 to avoid garbled Chinese characters; a storage path for intermediate files (such as . / intermediate / ) is reserved to save the intermediate results after each step of processing for easy traceability; a final output path (such as . / output / ) is reserved to standardize the naming format; and the tool's operating environment is ensured to be stable with no permission or dependency issues.

[0025] Optionally, launch the OCR tool, import the PDF / scanned standard file, and configure the output parameters, such as selecting JSONL as the output format, checking "Preserve Text Block Coordinate Information," "Confidence Score," and "Text Block Type Classification," and setting the recognition language to Chinese. The OCR tool then recognizes the PDF / scanned document, generating the original standard file in JSONL format. Each line of JSON records corresponds to a text block, line of text, or page record in the file to be processed, and must contain at least two required fields: page_id (page identifier, incrementing from 1, uniquely corresponding to each page of the original file) and text (text content, the specific text recognized by the OCR). For each line of JSON records, supplement the bounding box coordinate information (e.g., x0, y0, x1, y1, corresponding to the horizontal and vertical coordinates of the top-left and bottom-right corners of the text block, with the bottom-left corner of the page as the origin), score (confidence score, ranging from 0 to 1, used for subsequent low-confidence text filtering), and block_type (text block type, such as "normal" for ordinary text blocks, "table" for table text blocks, and "figure_note" for figure description text blocks). Use a JSON parsing tool to verify the integrity of the JSON format line by line, filter out abnormal lines that cannot be parsed (mark them as abnormal records and save them separately to error_ocr.txt for subsequent manual review), and ensure that the page_id and text fields of all valid records are not empty.

[0026] Step S102: Parse the records of the original standard file line by line to extract page identifiers, aggregate and construct page-level data structures according to page identifiers, and obtain the first intermediate result file.

[0027] In the example implementation, JSON parsing and editing tools such as Visual Studio Code and JSONFormatter can be used to parse the original standard file.

[0028] Optionally, the specific steps include: performing JSON deserialization on each record in the original standard file, parsing and extracting the page identifier and text field from each record; during the extraction process, if the current record lacks text coordinate information, only the text field and page identifier of the current record are retained; if the current record lacks a page identifier, the page number is inferred according to the line order of the current record in the original standard file, following a continuously increasing rule, and used as the page identifier of the current record; all records corresponding to the same extracted page identifier are aggregated, and the aggregated records are associated with the same page-level data structure, and page metadata is added to the same page-level data structure to obtain a page-level data structure; the page metadata includes page width and height, number of text blocks, average confidence of text units, and total number of characters on the page; all page-level data structures are summarized and sorted according to the page identifier order to obtain the first intermediate result file.

[0029] Understandably, during the raw standard file reading stage, a text processing tool (such as Python) is used to write a parsing script to read the data line by line. JSON format validation is performed on each line, skipping marked abnormal lines and retaining valid records. Required fields (page_id, text) and optional fields (x0, y0, x1, y1, score, block_type) are extracted from the valid records. Missing fields are filled in: if there is no page_id, the page number is calculated according to the order of records in the JSONL file (e.g., page_id: 1 for the first valid record, incrementing sequentially); if there is no coordinate information (x0, y0, x1, y1), it is marked as "no coordinates," and only the text content is retained for subsequent processing; if there is no score, it is filled with 0.8 by default; if there is no block_type, it is marked as "normal" by default.

[0030] Next, grouping by page_id, all text block / text line records corresponding to the same page_id are merged to construct a page object. Each page object contains page metadata and a list of text blocks: the page metadata includes page width and height (calculated from the maximum x1 and y1 values ​​of all text blocks on the page), number of text blocks (total number of valid records on the page), average confidence score (average of all record scores), and total number of page characters (sum of character lengths of the text fields of all records); the text block list contains complete field information for all valid text blocks on the page. All page objects are integrated into a page object collection and saved as pages_raw.json (path: . / intermediate / ). This file contains the metadata, text block list, and mapping relationship between the original JSONL records and page objects for subsequent tracing and processing.

[0031] Step S103: Clean and reconstruct the text within the pages of the first intermediate result file to obtain the second intermediate result file.

[0032] Optionally, the process specifically includes: extracting all page-level data structures from the first intermediate result file to obtain the text fields, text coordinate information, and page metadata contained in each page-level data structure; performing text normalization cleaning on the text fields in each page-level data structure; performing logical line reconstruction on the normalized and cleaned text fields based on the text coordinate information; and marking the noise in the first intermediate result file based on the page metadata and text distribution characteristics. The process also includes verifying the page-level data structures for each page that have undergone text cleaning, logical line reconstruction, and removal of headers and footers. A second intermediate result file is obtained after confirming that the logical line order is consistent with the actual page layout, that no text is missing after cleaning, and that the noise marking is accurate.

[0033] Understandably, firstly, text normalization is performed, that is: using text processing tools (supporting regular expression matching, coordinate calculation, and text normalization, such as Python+Pandas, Java+Jackson) to read the page objects in pages_raw.json, and performing normalization on the text field of each text block, strictly preserving the numbering symbols of the "number-title-body" structure in the standard file (such as "5.1", "(a)", "-"). Specific operations include: unifying full-width and half-width characters (converting full-width numbers and punctuation to half-width), compressing consecutive whitespace characters (converting multiple spaces to single spaces), removing invisible characters (tabs, invalid newlines, etc.), repairing line breaks caused by hyphens (such as "tech-nology" being repaired to "technology"), and correcting abnormal encoding (such as correcting garbled characters "，" to ",").

[0034] Then, based on the coordinate information (x0, y0, x1, y1) of the text blocks, the text blocks on the same page are merged and reconstructed: the vertical center distance and horizontal spacing of adjacent text blocks are calculated. If the vertical center distance is less than a preset threshold (e.g., 5px) and the horizontal spacing is less than a preset threshold (e.g., 10px), they are determined to be the same line, and their text content is sorted and concatenated from left to right along the x-axis; if the conditions are not met, spaces or newlines are inserted to distinguish different logical lines; if the text block has no coordinate information, logical lines are concatenated only according to the semantic coherence of the text content.

[0035] For header and footer processing, header and footer identification is based on the coordinate information of text blocks: if the text block y0 < 10% of the page height (top area) or y1 > 90% of the page height (bottom area) and the text content is repeated across pages (such as "GB / TXXXX-2020" "Page X"), it is determined to be a header or footer, marked as "noise", removed from the main text, but retained in the intermediate file for subsequent tracing.

[0036] Finally, the cleaned and reconstructed collection of page objects is saved as pages_clean.json (path: . / intermediate / ), which contains a list of logical lines for each page, the mapping relationship between the original text blocks and the cleaned lines, and header and footer tag information.

[0037] Step S104: Perform page type determination and global segment identification on the second intermediate result file to obtain the third intermediate result file; Optionally, the process specifically includes: generating a corresponding page feature vector for each page-level data structure; the page feature vector includes text features and layout features; determining the page type page by page based on the page feature vector and corresponding judgment rules, and labeling the page type to obtain the corresponding page type judgment result; dividing the global segment based on the type judgment result of all pages and the page identifier order, and clarifying the page range of each segment; marking the key segments and core positions of the global segment with structural anchor points, and recording the page identifier and logical line position corresponding to the structural anchor points; summarizing the page type label, global segment range, and structural anchor point position of all pages to generate a third intermediate result file.

[0038] Understandably, firstly, a page feature vector is constructed, which involves reading the page objects in pages_clean.json and calculating text features and layout features for each page: text features include the frequency of occurrence of feature keywords (such as "Table of Contents", "Preface", "Appendix", "References") and the density of chapter numbering patterns (such as the proportion of text lines with numbering formats like "5." and "5.1"); layout features include line length distribution (average number of characters per line), proportion of centered lines (proportion of text lines with center alignment), proportion of short lines (proportion of text lines with less than 10 characters), and text block density (number of text blocks / page area).

[0039] Next, the page type is determined page by page, based on the page feature vector and the determination rules: Cover page (containing at least two of the following: "Standard Number", "Standard Name", and "ICS Classification Number", and page_id≤3); Table of Contents page (containing the keywords "Table of Contents" or "Table of Contents", with a "dotted line + page number" structure, and short lines >60%); Preface / Introduction page (containing the keywords "Preface" or "Introduction", without obvious chapter numbers, and the text is mainly explanatory); Main text page (with a chapter numbering pattern, and the text is mainly technical description and clause content); Appendix page (containing titles such as "Appendix A" and "Appendix B", with the numbering pattern "A.1" and "B.2"); References page (containing the keyword "References", and the text is mainly in the format of bibliographic citations).

[0040] Next, global sections are divided, that is, consecutive pages of the same type are merged into global sections, and the page range and structural anchor point positions of each global section are recorded: such as the cover section (page 1), the table of contents section (page 2-page 3), the main text section (page 4-page 20), the appendix section (page 21-page 23), and the reference section (page 24); the structural anchor point positions mark the starting page and key title positions of each core section (such as "Scope" corresponding to page 4, and "Terms and Definitions" corresponding to page 6).

[0041] Finally, the page type determination result, global segment range, and structural anchor point position are saved as page_types.json (path: . / intermediate / ) for subsequent chapter tree construction and element extraction.

[0042] Step S105: Perform cross-page splicing and chapter tree construction on the third intermediate result file to obtain the fourth intermediate result file; Optionally, the process specifically includes: locating the start and end page identifiers of the main text segments based on page type annotations and global segment ranges, and filtering out all page-level data structures marked as main text pages; arranging the normalized logical lines of the filtered main text pages in the continuous order of page identifiers to form a preliminary main text sequence; validating the main text sequence, correcting connection deviations after confirming no text omissions, no duplicate splicing, and no semantic breaks, to form a complete and continuous main text sequence; constructing a chapter tree according to hierarchical rules based on the complete and continuous main text sequence and structural anchors; summarizing the chapter trees of all pages, complete main text blocks, and the source page numbers corresponding to each text block, and sorting them according to chapter hierarchy to generate a fourth intermediate result file.

[0043] Understandably, firstly, the text content is filtered and sequentially concatenated, that is: read page_types.json and pages_clean.json, filter out the pages marked as "main text pages", extract the logical line text of these pages, and concatenate them into a continuous main text text sequence in ascending order of page_id, with each line of text labeled with its source page_id; at the same time, the text sequences of the table of contents, preface, appendix, and references sections are concatenated separately and not mixed with the main text.

[0044] Next, the chapter titles are identified, specifically based on the chapter numbering pattern and title text format: the chapter numbering pattern includes chapter-level numbers (such as "5" or "Chapter 5"), section-level numbers (such as "5.1" or "5.2"), and subsection-level numbers (such as "5.1.1" or "a"); the title text format includes lines of text shorter than the average length of the body text, located in the middle or centered area of ​​the page, and with obvious blank lines before and after; the text lines that meet the conditions are marked as chapter titles, and their numbers (such as sec_id), title text (such as sec_title), starting position, and source page_id are recorded to form a candidate set of chapter titles.

[0045] Next, a chapter tree is constructed, which involves establishing parent-child relationships for chapter titles based on the hierarchical relationship of chapter numbers. For example, "5" is the parent node, and "5.1" and "5.2" are child nodes, with "5.1.1" being a child node of "5.1". Logical lines between two adjacent chapter titles are then grouped into the chapter node corresponding to the latter title, forming a complete chapter tree structure. During this process, if any missing or skipped numbers occur, the numbers are supplemented according to the directory structure; if duplicate numbers occur, they are merged based on text similarity or sequentially labeled as "5.1-1" and "5.1-2".

[0046] Next, the paragraphs spanning multiple pages are joined together. Specifically, the continuity of the paragraphs is determined, and joining is performed if any two of the following conditions are met: First, the last paragraph on the previous page does not end with a complete sentence punctuation mark such as ".", "!", or "?"; the numbering or indentation of the preceding and following paragraphs is consistent. Second, the preceding and following paragraphs are semantically continuous, for example, by the presence of logical connectors such as "in addition" or "therefore," or by a referential relationship. After joining, the source page number range of the paragraph is marked (e.g., page 10-page 11).

[0047] Finally, the intermediate file output is: the chapter tree data after the cross-page paragraphs are concatenated, saved as sections.json (e.g., path: . / intermediate / ). This intermediate file can contain chapter numbers, titles, a list of body text blocks, the source page number range, and the chapter hierarchy.

[0048] Step S106: Locate the target chapter based on the structural anchor points of the third intermediate result file, and extract terminology entries and technical content from the fourth intermediate result file to obtain the fifth and sixth intermediate result files.

[0049] Optionally, the process includes: extracting the structural anchor points, global section range, and page type annotations from the third intermediate result file to determine the terminology chapters; extracting the main text blocks and source page numbers corresponding to the terminology chapters from the fourth intermediate result file to obtain the fifth intermediate result file; extracting the technical content from the fourth intermediate result file based on the technical chapters of the main text section and constructing structured data to generate the sixth intermediate result file.

[0050] Understandably, during the terminology element extraction phase, sections.json is read, and based on the structural anchors in page_types.json, the chapter node corresponding to "Terms and Definitions" is located, extracting all text blocks under that node. Each term entry is located according to the chapter numbering pattern (e.g., "3.1", "3.2"), and term-related fields are extracted, such as term number (t_id), Chinese name (t_cn), English equivalent (t_en, English in parentheses, left blank if none), definition text (t_def), notes (t_note, e.g., "Note 1", "Note 2"), source description (t_source, e.g., "[Source:…]"), and examples (t_exp). Multiple lines of text with the same term number are merged into a complete definition, and duplicate content is removed. Term entries with a score < 0.6 (low confidence) require manual review.

[0051] During the technical content element extraction stage, the sections.json file is read, and the chapters corresponding to the preface, introduction, scope, terms and definitions, and references are excluded. The remaining chapters are used as technical content chapters. The technical content is divided into a three-level structure: tech_item (corresponding to chapters or sections, such as "5 Technical Requirements"), tech_paragraph (corresponding to sections, such as "5.1 Design Requirements"), and tech_part (corresponding to clauses, paragraphs, or sections, such as "5.1.1 Material Selection"). The relevant fields of the technical clauses are extracted: technical clause number (tech_ptid), technical clause subheading (tech_ptname, a short title after the number, left blank if there is none), and technical clause body (tech_ptbody).

[0052] 6.3 Title Validation: The technical clause subheadings are validated. If any one of the following conditions is met, it is considered a misidentification: the text length of tech_ptname is greater than 50 characters; it begins with suggestive words such as "should", "should", or "may" and contains punctuation at the end of the sentence; it contains a colon ":" and the following content is a complete sentence; misidentified titles are merged into tech_ptbody, and tech_ptname is left empty.

[0053] Finally, the term extraction results are saved as term_struct.json, and the technical content extraction results are saved as technology_struct.json (path: . / intermediate / ). Both files contain information for each field, source chapter number, page number range, and low-confidence entry markers.

[0054] Step S107: Map the first intermediate result file, the second intermediate result file, the third intermediate result file, the fourth intermediate result file, the fifth intermediate result file, and the sixth intermediate result file and perform a consistency check. After the check passes, a full-feature structured file is obtained.

[0055] Understandably, in accordance with the GB / T37967-2019 standard, a top-level framework for a fully structured JSON is created, containing six core modules: basic_info (basic information), auxiliary_info (auxiliary information), term (term elements), technology (technical content elements), edit (editing information), and execute (implementation information). Each module contains sub-fields required by the standard. Therefore, it is necessary to extract corresponding data from each intermediate file and map it to the corresponding fields in the JSON top-level framework: basic_info is extracted from the cover page of page_types.json (standard name, ICS classification number, standard number, publication date, etc., with the date uniformly in "YYYY-MM-DD" format); auxiliary_info is extracted from each global section (table of contents, preface, appendix, etc.); term maps to the terminology entry list in term_struct.json; technology maps to the three-level technical structure in technology_struct.json; edit extracts the drafting unit and drafter from the preface, leaving it blank if none are found; execute reserves a feedback field with an initial value of an empty array.

[0056] Next, the GB / T37967-2019 compliance verification tool is used to perform multi-dimensional verification: format verification (standard number and date format meet requirements); integrity verification (core fields are not empty); hierarchy verification (chapter number hierarchy is consistent); uniqueness verification (term number and technical clause number are not duplicated); if the verification fails, an error report (error_report.txt) is output, indicating the location and cause of the problem; if the verification passes, a full-element structured JSON file is output, named in the format of "standard number_structured result.json" (e.g., "GB / T37967-2019_structured result.json"), and saved to the preset . / output / path, completing the core process output.

[0057] In other embodiments of this application, manual sampling and verification can also be employed. This involves extracting core fields (term definitions, key technical clauses, and basic information) from the structured results, with a sampling rate of no less than 10%. The focus is on verifying structured content corresponding to cross-page splicing and low-confidence text. The sampled structured content is then compared sentence-by-sentence with the corresponding text in the original standard document to check for omissions, errors, incorrect numbering, and field extraction deviations. Simultaneously, the source of the problem is traced through intermediate files, marking the processing stage to which the problem belongs. Using JSON parsing tools and GB / T37967-2019 compliance verification tools, the completeness of fields and the hierarchical standardization of the final JSON file are verified again to ensure that the file can be read and parsed normally by mainstream data processing tools.

[0058] Furthermore, in other embodiments, this application may also include result optimization and traceability loop, rule optimization, and archiving. Specifically, for problems discovered during sampling and verification, the process is traced back to the corresponding processing stage (e.g., terminology extraction errors are traced back to Stage 4, line reconstruction errors are traced back to Stage 1), relevant parameters or logic are corrected, and the corresponding steps are re-executed until the problem is resolved. Common problems are analyzed, and processing rules are optimized (e.g., adjusting chapter title recognition thresholds and cross-page splicing judgment conditions) to improve the processing accuracy of subsequent similar standard documents; anomaly log files are updated to improve the traceability system. The original standard documents, OCR original JSONL files, intermediate files for each Stage, final structured JSON files, error reports, and sampling records are archived according to specifications to form a complete processing archive, ensuring full traceability throughout the process.

[0059] Steps S101 to S107 are performed as follows: The standard document to be processed is identified using an OCR tool to obtain the original standard document; page identifiers are extracted by parsing the records of the original standard document line by line, and a page-level data structure is constructed based on these page identifiers to obtain the first intermediate result file; the text within the pages of the first intermediate result file is cleaned and reconstructed to obtain the second intermediate result file; page type determination and global segment identification are performed on the second intermediate result file to obtain the third intermediate result file; cross-page splicing and chapter tree construction are performed on the third intermediate result file to obtain the fourth intermediate result file; the target chapter is located using the structural anchor points of the third intermediate result file, and terminology entries and technical content are extracted from the fourth intermediate result file to obtain the fifth and sixth intermediate result files; the first, second, third, fourth, and fifth intermediate result files are mapped to... The sixth intermediate result file is then subjected to consistency verification. After the verification passes, a full-element structured file is obtained. On the one hand, from parsing and aggregating the original JSONL format file, to successively completing text cleaning and reconstruction, page segment identification, chapter structure construction, core element extraction, and then to full intermediate file mapping verification, the data flow trajectory can be traced. Those skilled in the art can directly implement the process, greatly reducing the difficulty of technology implementation and replication, and solving the problems of disordered processes and chaotic data connection in traditional processing methods. On the other hand, it fully preserves the full-element information of the standard file, including page meta-information, text content, hierarchical structure, terminology entries, and technical content, solving the shortcomings of incomplete structuring and extraction of only single content in traditional processing. The output full-element structured file can be directly adapted to subsequent application scenarios such as digital management of standard files, intelligent retrieval, content reuse, and data mining, greatly improving the efficiency and value of digital utilization of unstructured standard files.

[0060] Based on the same inventive concept, this application also provides an apparatus for converting unstructured standard documents to implement the aforementioned method for converting unstructured standard documents. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the unstructured standard document conversion apparatus provided below can be found in the limitations of the unstructured standard document conversion method described above, and will not be repeated here.

[0061] In one exemplary embodiment, such as Figure 2 As shown, an unstructured standard file conversion device 200 is provided, comprising: an acquisition module 201, a parsing and extraction module 202, a cleaning and reconstruction module 203, a judgment and recognition module 204, a splicing and construction module 205, a positioning and extraction module 206, and a mapping generation module 207, wherein: The acquisition module 201 is used to recognize the standard file to be processed based on the OCR tool to obtain the original standard file; the original standard file is in JSONL format; each record in the original standard file includes a page identifier; The parsing and extraction module 202 is used to parse the records of the original standard file line by line to extract page identifiers, aggregate and construct page-level data structures according to page identifiers, and obtain the first intermediate result file; The cleaning and reconstruction module 203 is used to clean the text within the pages and reconstruct the lines of the first intermediate result file to obtain the second intermediate result file. The determination and recognition module 204 is used to determine the page type and global segment recognition of the second intermediate result file to obtain the third intermediate result file; The splicing and building module 205 is used to splice the third intermediate result file across pages and build the chapter tree to obtain the fourth intermediate result file; The positioning and extraction module 206 is used to locate the target chapter based on the structural anchor points of the third intermediate result file, and extract terminology entries and technical content in combination with the fourth intermediate result file to obtain the fifth and sixth intermediate result files. The mapping generation module 207 is used to map the first intermediate result file, the second intermediate result file, the third intermediate result file, the fourth intermediate result file, the fifth intermediate result file, and the sixth intermediate result file and perform consistency verification. After the verification is passed, a full-feature structured file is obtained.

[0062] As an optional implementation, the parsing and extraction module 202 is specifically used to perform JSON deserialization processing on each record of the original standard file, parse and extract the page identifier and text field in each record; during the extraction process, if the current record is missing text coordinate information, only the text field and page identifier of the current record are retained; if the current record is missing a page identifier, the page number is inferred according to the line order of the current record in the original standard file according to the continuous increment rule, and used as the page identifier of the current record; all records corresponding to the same extracted page identifier are aggregated, the aggregated records are associated with the same page-level data structure, and page meta-information is supplemented in the same page-level data structure to obtain the page-level data structure; the page meta-information includes page width and height, number of text blocks, average confidence of text units, and total number of characters on the page; all page-level data structures are summarized and sorted according to the page identifier order to obtain the first intermediate result file.

[0063] As an optional implementation, the cleaning and reconstruction module 203 is specifically used to: extract all page-level data structures from the first intermediate result file to obtain the text fields, text coordinate information, and page metadata contained in each page-level data structure; perform text normalization cleaning on the text fields in each page-level data structure; perform logical line reconstruction on the normalized and cleaned text fields based on the text coordinate information; and mark the noise in the first intermediate result file based on the page metadata and text distribution characteristics; and verify the page-level data structure of each page that has completed text cleaning, logical line reconstruction, and removal of headers and footers. If the logical line order is confirmed to be consistent with the actual page layout, the cleaned text is complete, and the noise marking is accurate, a second intermediate result file is obtained.

[0064] As an optional implementation, the aforementioned determination and recognition module 204 is specifically used to: generate a corresponding page feature vector for each page-level data structure; the page feature vector includes text features and layout features; determine the page type page by page based on the page feature vector and corresponding determination rules, and mark the page type to obtain the corresponding page type determination result; divide the global segment and clarify the page range of each segment based on the type determination results of all pages and the page identification order; mark the key segments and core positions of the global segment with structural anchor points, and record the page identification and logical line position corresponding to the structural anchor points; summarize the page type markings, global segment ranges and structural anchor point positions of all pages, and generate a third intermediate result file.

[0065] As an optional implementation, the splicing and construction module 205 is specifically used to: locate the start page identifier and end page identifier corresponding to the main text segment based on the page type label and the global segment range; filter out all page-level data structures labeled as main text pages; arrange the normalized logical lines of the selected main text pages in the continuous order of the page identifiers to form a preliminary main text sequence; verify the main text sequence, and correct the connection deviations after confirming that there are no missing texts, no repeated splicing, and no semantic breaks, to form a complete and continuous main text sequence; construct a chapter tree according to the hierarchical rules based on the complete and continuous main text sequence and structural anchor points; summarize the chapter trees of all pages, complete main text blocks, and the source page numbers corresponding to each text block, and generate a fourth intermediate result file after sorting them according to the chapter hierarchy.

[0066] As an optional implementation, the above-mentioned positioning and extraction module 206 is specifically used to: extract the structural anchor point position, global section range and page type annotation from the third intermediate result file to determine the terminology chapter; extract the main text block and source page number corresponding to the terminology chapter from the fourth intermediate result file to obtain the fifth intermediate result file; extract the technical content of the fourth intermediate result file with the technical chapters of the main text section as the scope and construct structured data to generate the sixth intermediate result file.

[0067] This implementation method offers several advantages. First, it ensures traceability of data flow, enabling skilled personnel to directly implement the process from parsing and aggregating the original JSONL file to sequentially completing text cleaning and reconstruction, page segment identification, chapter structure construction, core element extraction, and full intermediate file mapping verification. This significantly reduces the difficulty of technology implementation and replication, solving the problems of disordered processes and chaotic data connections in traditional processing methods. Second, it fully preserves all elements of the standard file, including page metadata, text content, hierarchical structure, terminology entries, and technical content. This addresses the shortcomings of incomplete structuring and extraction of only single content in traditional processing. The output fully structured file can be directly adapted to subsequent application scenarios such as digital management, intelligent retrieval, content reuse, and data mining of standard files, greatly improving the efficiency and value of digital utilization of unstructured standard files.

[0068] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 3 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores conversion data for unstructured standard files. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a method for converting unstructured standard files.

[0069] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0070] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0071] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0072] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0074] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0075] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0076] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0077] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for converting unstructured standard documents, characterized in that, The conversion method for unstructured standard files includes: The standard file to be processed is identified using an OCR tool to obtain the original standard file; the original standard file is in JSONL format; each record in the original standard file includes a page identifier. The records of the original standard file are parsed line by line to extract page identifiers. Page-level data structures are then constructed based on the page identifiers to obtain the first intermediate result file. The text within the first intermediate result file is cleaned and the lines are reconstructed to obtain the second intermediate result file; The second intermediate result file is subjected to page type determination and global segment identification to obtain the third intermediate result file; The third intermediate result file is spliced ​​across pages and a chapter tree is constructed to obtain the fourth intermediate result file; Based on the structural anchor points of the third intermediate result file, the target chapter is located, and the terminology entries and technical content are extracted from the fourth intermediate result file to obtain the fifth and sixth intermediate result files. Map the first, second, third, fourth, fifth, and sixth intermediate result files and perform consistency checks. After the checks pass, a full-feature structured file is obtained.

2. The method for converting unstructured standard documents according to claim 1, characterized in that, The process of parsing the records of the original standard file line by line to extract page identifiers, and then aggregating and constructing a page-level data structure according to the page identifiers, yields a first intermediate result file, including: Perform JSON deserialization on each record of the original standard file, and parse and extract the page identifier and text field from each record; During the extraction process, if the current record is missing text coordinate information, only the text field and page identifier of the current record are retained; if the current record is missing a page identifier, the page number is inferred according to the line order of the current record in the original standard file and used as the page identifier of the current record. All records corresponding to the same page identifier extracted are aggregated, and the aggregated records are associated with the same page-level data structure. Page metadata is added to the same page-level data structure to obtain the page-level data structure. The page metadata includes page width and height, number of text blocks, average confidence of text units, and total number of characters on the page. The first intermediate result file is obtained by summarizing all the page-level data structures and sorting them according to the page identifier order.

3. The method for converting unstructured standard documents according to claim 2, characterized in that, The process of cleaning and reconstructing the text within the first intermediate result file to obtain a second intermediate result file includes: Extract all page-level data structures from the first intermediate result file to obtain the text fields, text coordinate information, and page metadata contained in each page-level data structure; Text normalization and cleaning are performed on the text fields in each page-level data structure; logical row reconstruction is performed on the normalized and cleaned text fields based on the text coordinate information; and noise in the first intermediate result file is marked based on the page metadata and text distribution characteristics. The page-level data structure of each page, after text cleaning, logical line reconstruction, and removal of headers and footers, is verified. The second intermediate result file is obtained after confirming that the logical line order is consistent with the actual page layout, that no text is missing after cleaning, and that noise is accurately marked.

4. The method for converting unstructured standard documents according to claim 3, characterized in that, The step of performing page type determination and global segment identification on the second intermediate result file to obtain the third intermediate result file includes: Generate a corresponding page feature vector for each page-level data structure; the page feature vector includes text features and layout features. Based on the page feature vector and the corresponding determination rules, the page type is determined page by page and the page type is labeled to obtain the corresponding page type determination result; Based on the type determination results of all pages and the page identification order, the global section is divided and the page scope of each section is defined; Structural anchor points are marked for key sections and core locations of the global section, and the page identifier and logical line position corresponding to the structural anchor points are recorded. The page type annotations, global section ranges, and structural anchor point positions of all pages are summarized to generate the third intermediate result file.

5. The method for converting unstructured standard documents according to claim 4, characterized in that, The process of performing cross-page concatenation and chapter tree construction on the third intermediate result file to obtain the fourth intermediate result file includes: Based on the page type label and the global segment range, locate the start page and end page labels corresponding to the main text segments, and filter out all page-level data structures labeled as main text pages; The normalized logical lines of the selected main text pages are arranged sequentially according to the continuous order of the page identifiers to form a preliminary main text sequence; The main text sequence is verified, and the connection deviation is corrected after confirming that there is no text omission, no repeated splicing, and no semantic breakage, so as to form a complete and continuous main text sequence. Based on the complete and continuous text sequence and structural anchor points, a chapter tree is constructed according to hierarchical rules; The fourth intermediate result file is generated by summarizing the chapter tree, complete text blocks, and source page numbers corresponding to each text block of all pages and sorting them in chapter hierarchy order.

6. The method for converting unstructured standard documents according to claim 5, characterized in that, The process of locating the target chapter based on the structural anchor points of the third intermediate result file, and extracting terminology entries and technical content from the fourth intermediate result file, yields the corresponding fifth and sixth intermediate result files, including: Extract the structural anchor point positions, global section ranges, and page type annotations from the third intermediate result file to determine the terminology and chapters; Extract the main text block and source page number corresponding to the terminology section from the fourth intermediate result file to obtain the fifth intermediate result file; The technical content of the fourth intermediate result file is extracted from the technical sections of the main text and structured data is constructed to generate the sixth intermediate result file.

7. A device for converting unstructured standard documents, characterized in that, The device for converting unstructured standard documents includes: The acquisition module is used to identify the standard file to be processed based on an OCR tool to obtain the original standard file; the original standard file is in JSONL format; each record in the original standard file includes a page identifier; The parsing and extraction module is used to parse the records of the original standard file line by line to extract page identifiers, aggregate and construct page-level data structures according to the page identifiers, and obtain the first intermediate result file. The cleaning and reconstruction module is used to clean the text within the pages and reconstruct the lines of the first intermediate result file to obtain the second intermediate result file. The determination and identification module is used to determine the page type and identify the global segment of the second intermediate result file to obtain the third intermediate result file. The splicing and building module is used to perform cross-page splicing and chapter tree construction on the third intermediate result file to obtain the fourth intermediate result file; The positioning and extraction module is used to locate the target chapter based on the structural anchor points of the third intermediate result file, and extract terminology entries and technical content in combination with the fourth intermediate result file to obtain the fifth and sixth intermediate result files. The mapping generation module is used to map the first intermediate result file, the second intermediate result file, the third intermediate result file, the fourth intermediate result file, the fifth intermediate result file, and the sixth intermediate result file and perform consistency verification. After the verification passes, a full-feature structured file is obtained.

8. A computer device, comprising: A memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the method for converting unstructured standard documents according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method for converting unstructured standard files as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method for converting unstructured standard files as described in any one of claims 1-6.

Images