Methods, devices, and form enhancement methods and systems for automatically converting cross-format documents to Markdown documents

Abstract

This invention discloses a method, apparatus, and form enhancement method and system for automatically converting cross-format documents to Markdown documents. The Markdown conversion method includes: first, acquiring a document image; then, mining hidden structural information from the document image, including target elements and their hierarchical relationships; next, performing semantic content analysis on the document image based on this structural information to obtain semantic content analysis results; then, extracting key information from the document image based on the analysis results; converting this key information into Markdown format and initially formatting it as a table or list to obtain a preliminary Markdown form; finally, optimizing the preliminary Markdown form to form a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown. This method can effectively identify and convert complex nested structures and irregularly laid-out forms in documents, ensuring the integrity and readability of form information when converting to Markdown format.

Description

Technical Field

[0001] This invention relates to a method, apparatus, and form enhancement method and system for automatically converting cross-format documents to Markdown documents. Background Technology

[0002] Markdown is a lightweight markup language that, with its concise syntax and good compatibility, has been widely used in technical documentation, blog publishing, and note-taking. Its advantages include ease of reading and writing, the ability to store data in plain text, and ease of version control and cross-platform editing. With the increasing popularity of Markdown, the demand for converting other document formats (such as Word documents and PDFs) to Markdown is growing, prompting the development of related technologies.

[0003] Against this backdrop, several existing solutions have been proposed to achieve document format conversion. Among them, patent application CN202311802017.4 proposes a conversion method based on image preservation. This method uses the Spire component to read Office documents and extract and save image information to ensure that the content and format of images are effectively preserved in Markdown documents. This method plays a positive role in image embedding and display. Furthermore, patent application CN201510310288.7 describes a document format conversion method that includes loading and converting a Word document into a web page document, and then converting the tags in the web page document into Markdown. During this process, style attributes are preserved to ensure the style effects of the Markdown document. This technology aims to avoid the loss of style effects during conversion, thereby enabling the re-editing of Markdown documents.

[0004] Despite some progress in existing technologies, none of them can effectively identify and convert forms with complex nested structures and irregular layouts in documents, making it impossible to ensure the integrity and readability of form information when converting documents to Markdown format. Summary of the Invention

[0005] The technical problem this invention aims to solve is to address the aforementioned shortcomings of existing technologies by proposing a method, apparatus, and form enhancement method and system for automatically converting cross-format documents to Markdown documents. This method can effectively identify and convert forms with complex nested structures and irregular layouts in documents, thereby ensuring the integrity and readability of form information when converting documents to Markdown format.

[0006] In a first aspect, the present invention provides a method for automatically converting cross-format documents to Markdown documents, the method comprising the following steps:

[0007] Step S1: Obtain the document image;

[0008] Step S2: Extract hidden structural information from the document image;

[0009] The structural information refers to the hierarchical relationship between target elements; the target elements are headings, and / or paragraphs, and / or lists, and / or tables.

[0010] Step S3: Based on the structural information, perform semantic content analysis on the document image to obtain the semantic content analysis results;

[0011] Step S4: Based on the semantic content analysis results, extract the key information of the document image;

[0012] Step S5: Convert the key information into Markdown format and perform preliminary formatting in the form of a table or list to obtain a preliminary Markdown form;

[0013] Step S6: Optimize the initial Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0014] Further, step S2 specifically includes the following steps:

[0015] Step S21: Use a convolutional neural network to extract features from the acquired document image to obtain a feature map;

[0016] Step S22: Input the feature map into a preset RPN to generate candidate regions;

[0017] Step S23: Use RoIPooling to map each candidate region to a feature representation of a fixed size to obtain the feature representation of the candidate region;

[0018] Step S24: Input the feature representation of the candidate region into the fully connected layer for processing to obtain the coordinate positions of the key elements; the key elements include titles, paragraphs, lists and tables;

[0019] Step S25: Based on the coordinate positions of the key elements, analyze the spatial layout and relative positional relationships of the key elements, and infer the hierarchical relationships between the key elements;

[0020] Step S26: Organize the coordinate positions of the key elements and the hierarchical relationships between the key elements to generate a tree-like or graphical representation that conforms to the logical structure, thereby obtaining the document image structure information.

[0021] Further, step S3 specifically includes:

[0022] By using recurrent neural networks and contextual coding, the structural information of document images is analyzed and identified, and paragraph types, form structures and code block content are extracted to obtain comprehensive semantic content analysis results.

[0023] The method utilizes recurrent neural networks and context encoding to analyze and identify the structural information of document images, extracting paragraph types, form structures, and code block content to obtain comprehensive semantic content analysis results. Specifically, it includes the following steps:

[0024] Step S31: Segment the document image structure information into paragraphs or sentences to form a text sequence;

[0025] Step S32: Input the text sequence into the Bi-LSTM model for context modeling, and dynamically focus on key indicators through an attention mechanism;

[0026] The key indicators are important elements or information in the document image;

[0027] Step S33: Based on the aforementioned key indicators, use MLP to classify each paragraph, identifying the paragraph type as plain text, form content, or code block; and,

[0028] Regular expressions are used to extract key fields from the form content and structure the form information to obtain the form structure; and,

[0029] Perform syntactic analysis on the code block to obtain its semantics and thus its content;

[0030] Step S34: Integrate the paragraph type, the form structure, and the code block content to form a semantic content integration result;

[0031] Step S35: Optimize and complete the semantic content integration results to obtain comprehensive semantic content analysis results.

[0032] Further, step S4 specifically includes:

[0033] Based on the semantic content analysis results, a graph-based sequence transformation model is used to extract key information from document images;

[0034] The extraction of key information from document images based on semantic content analysis results using a graph-based sequence transformation model includes the following steps:

[0035] Step S41: Transform the semantic content analysis results into a form topology structure;

[0036] In this form topology, each node represents an input field, and each edge represents a relationship between fields.

[0037] Step S42: Encode the form topology using a graph attention network to obtain the form topology encoding information;

[0038] Step S43: Based on the graph convolution and attention mechanism of the graph attention network, learn the interaction and dependency relationship between nodes in the table topology structure according to the table topology structure encoding information, and generate node representation vectors of context information;

[0039] Step S44: Based on the node representation vector, use a conditional random field to perform sequence labeling on the input field to identify key information of the document image;

[0040] The key information includes the input fields and layout in the document image.

[0041] Further, step S5 specifically includes the following steps:

[0042] Step S51: Annotate the key information to obtain the annotated field sequence;

[0043] Step S52: Input the annotated field sequence into the Transformer-based sequence transformation model, convert the field sequence into a structured Markdown format through a self-attention mechanism, and generate corresponding Markdown tags by filling in templates according to the semantic category of the fields and the form layout information.

[0044] Specifically, for table-type forms, Markdown's table syntax is used to generate a structured table representation; or, for list-type forms, Markdown's list syntax is used to generate structured list items.

[0045] Step S53: Generate a preliminary Markdown form based on the Markdown tags.

[0046] Further, step S6 specifically includes the following steps:

[0047] Step S61: Compare the layout structure differences between the document image and the preliminary Markdown form, and use scripts or plugins to validate and dynamically display the form structure of the preliminary Markdown form to obtain a first optimized Markdown form; and,

[0048] Based on the structural information, the title and subtitles of the initial Markdown form are generated accordingly, and the lists and code blocks in the initial Markdown form are syntactically formatted to obtain the second optimized Markdown form; and,

[0049] The list items in the initial Markdown form are analyzed to identify the indentation, numbering, or symbol features of the list items. Based on this, the unordered and ordered lists are converted into Markdown format to obtain the third optimized Markdown form.

[0050] Step S62: Integrate the first optimized Markdown form, the second optimized Markdown form, and the third optimized Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0051] Further, in step S61, the layout structure differences between the document image and the preliminary Markdown form are compared, and the form structure of the preliminary Markdown form is validated and dynamically displayed using a script or plugin tool to obtain the first optimized Markdown form. This specifically includes the following steps:

[0052] Step A1: Use the edit distance algorithm to compare the layout structure differences between the document image and the initial Markdown form to obtain the difference metric value;

[0053] Specifically, the edit distance algorithm is a quantification of the difference in the number of insertion, deletion, and replacement operations;

[0054] Step A2: Based on the difference metric, automatically optimize the layout structure of the initial Markdown form to obtain the first-process Markdown form; the automatic optimization includes automatically adjusting the alignment, indentation level, and blank line separation.

[0055] Step A3: Use a matching script tool or plugin to parse and optimize the Markdown form from the first process, generate a display template, and use a preset template engine for rendering and data binding to obtain the first optimized Markdown form;

[0056] Among them, the supporting script tools or plugins integrate verification and error correction functions, and automatically check the field type, value range and required fields through predefined rules, and provide error prompts and correction suggestions;

[0057] In step S61, based on the structural information, the title and subtitles of the preliminary Markdown form are generated accordingly, and the lists and code blocks in the preliminary Markdown form are syntactically formatted to obtain the second optimized Markdown form. Specifically, this includes the following steps:

[0058] Step B1: Based on the structural information, the document content is automatically annotated using a conditional random field sequence labeling model to obtain the annotation results;

[0059] The annotation results are used to identify different document elements and extract context and location information features;

[0060] Step B2: Based on the annotation results, automatically generate Markdown format headings and subheadings to obtain a preliminary Markdown form with hierarchical relationships;

[0061] Step B3: By performing multi-level list indentation and line breaks on the initial Markdown form with hierarchical relationships, and by formatting the code blocks in the initial Markdown form with hierarchical relationships, a second-process optimized Markdown form is obtained.

[0062] Step B4: Parse and render the second process optimized Markdown form into an HTML page or PDF file corresponding to the visual structure of the document image. By comparing and evaluating the accuracy and completeness of the Markdown text generation, continuously optimize the sequence labeling model and Markdown generation rules to form the second optimized Markdown form.

[0063] In step S61, the list items in the initial Markdown form are analyzed to identify their indentation, numbering, or symbolic features. Based on this, the unordered and ordered lists are converted into Markdown format to obtain the third optimized Markdown form. This process includes the following steps:

[0064] Step C1: Take the list items in the initial Markdown form as input, use the Conditional Random Field (CRF) model to perform sequence labeling to identify indentation, numbering, and symbol features, construct feature templates and extract relevant attributes to obtain new list item text;

[0065] The Conditional Random Field (CRF) model was trained using gradient descent and the L-BFGS algorithm.

[0066] Step C2: Use the trained Conditional Random Field (CRF) model to perform sequence labeling on the new list item text, identify the structural features of each list item, determine its list type, and obtain lists of different types;

[0067] Step C3: Based on the logical consistency of indentation and numbering for the different types of lists, generate a third-process optimized Markdown form;

[0068] Step C4: Based on the list item text in the document image, evaluate and optimize the list in the third optimized Markdown form to obtain the third optimized Markdown form.

[0069] Secondly, the present invention provides a form enhancement method based on automatic conversion of cross-format documents to Markdown, the method comprising the following steps:

[0070] Step K1: Obtain heterogeneous documents;

[0071] Step K2: Split the heterogeneous document into a text portion and a text-image portion;

[0072] Step K3: For the text portion, generate the first Markdown document corresponding to the text; and,

[0073] For the text image portion, the method for automatically converting cross-format documents to Markdown documents as described in the first aspect is used to generate a second Markdown document corresponding to the text image;

[0074] The structural and semantic coherence between all the first and second Markdown documents is analyzed, and the structural information between all the first and second Markdown documents is merged into a single Markdown document to complete the form enhancement based on automatic cross-format document to Markdown.

[0075] Thirdly, the present invention provides an apparatus for automatically converting cross-format documents to Markdown documents, the apparatus comprising:

[0076] The acquisition unit is used to acquire document images;

[0077] A mining unit, connected to the acquisition unit, is used to mine hidden structural information from the document image;

[0078] The structural information refers to the hierarchical relationship between target elements; the target elements are headings, and / or paragraphs, and / or lists, and / or tables.

[0079] An analysis unit, connected to the mining unit, is used to perform semantic content analysis on the document image based on the structural information to obtain semantic content analysis results;

[0080] An extraction unit, connected to the analysis unit, is used to extract key information from the document image based on the semantic content analysis results;

[0081] A processing unit, connected to the extraction unit, is used to convert the key information into Markdown format and perform preliminary formatting processing in the form of a table or list to obtain a preliminary Markdown form.

[0082] An optimization unit, connected to the processing unit, is used to optimize the initial Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0083] Fourthly, this invention provides a form enhancement system based on automatic cross-format document to Markdown conversion, the system comprising:

[0084] Acquisition device, used to acquire heterogeneous documents;

[0085] A splitting device, connected to the acquisition device, is used to split the heterogeneous document into a text portion and a text-image portion;

[0086] A generation device, connected to the splitting device, is used to generate a first Markdown document corresponding to the text portion;

[0087] The apparatus for automatically converting cross-format documents to Markdown documents as described in the third aspect is connected to the splitting device and is used to generate a second Markdown document corresponding to the image from the text image portion.

[0088] The merging device is connected to the generating device and the device for automatically converting cross-format documents to Markdown documents, respectively. It is used to merge all the first Markdown documents and all the second Markdown documents into a complete Markdown document based on the structural information and semantic coherence between the first Markdown document and the second Markdown document, so as to complete the form enhancement based on automatic conversion of cross-format documents to Markdown.

[0089] This invention can effectively identify and convert forms with complex nested structures and irregular layouts in documents, thereby ensuring the integrity and readability of form information when converting documents to Markdown format. Specific beneficial effects are as follows:

[0090] 1. Effective identification of complex structures: This invention can identify and convert complex nested structures and irregularly laid-out forms in documents, ensuring the integrity and readability of information while ensuring that key data is not lost, thereby improving the overall accuracy of conversion.

[0091] 2. Enhanced structural recognition capability: By analyzing the specific positions and hierarchical relationships of headings, paragraphs, lists, and tables, this invention can accurately identify the structure in a document, thereby ensuring that the actual layout of the document can be effectively captured during Markdown conversion.

[0092] 3. Semantic-based key information extraction: This invention uses a graph-based sequence transformation model to extract the input fields and layout information of the form at the semantic level, ensuring the integrity of key information. The extracted information is then converted into Markdown format for preliminary formatting, making the converted document easier to read and understand.

[0093] 4. Optimized Layout: After converting to Markdown format, this invention optimizes the form layout by comparing the structural differences between the original and converted forms. This optimization not only improves the document's aesthetics but also enhances readability.

[0094] 5. Dynamic Validation and Display: During the conversion process, this invention utilizes scripts or plugins to validate and dynamically display the form structure, ensuring the accuracy and consistency of the conversion results. This feature allows users to view the conversion effect in real time, enhancing the interactivity of the experience.

[0095] 6. Precise Syntax Formatting: When generating Markdown headings, subheadings, lists, and code blocks, this invention can accurately format the syntax according to the visual structure of the original document. This ensures that the converted Markdown document not only retains the logical fluency of the original document but also achieves an aesthetically pleasing and readable effect.

[0096] 7. Multi-Document Merging Capability: This invention analyzes the structural and semantic coherence between multiple documents, enabling seamless merging of content from documents with different formats into a single, coherent Markdown document. This process not only maintains the logical order of information but also ensures format consistency, significantly improving work efficiency. Simultaneously, it ensures a unified format for the merged output, maintaining the professionalism and standardization of the documents.

[0097] 8. Improved User Experience: This invention provides users with a convenient and efficient information processing experience, especially when processing complex documents and merging multiple documents, significantly reducing adjustment time costs and improving work efficiency and satisfaction; in addition, the converted documents are easier to understand, enhancing the user's reading experience. Attached Figure Description

[0098] Figure 1 This is a schematic diagram of the process for automatically converting cross-format documents to Markdown documents in an embodiment of the present invention;

[0099] Figure 2 This is a schematic diagram of a method for automatically converting cross-format documents to Markdown documents in an embodiment of the present invention;

[0100] Figure 3 This is a flowchart of the form enhancement method based on automatic conversion of cross-format documents to Markdown in an embodiment of the present invention;

[0101] Figure 4 This is a flowchart of the Markdown formatting process in an embodiment of the present invention;

[0102] Figure 5 This is a schematic diagram of an apparatus for automatically converting cross-format documents to Markdown documents according to an embodiment of the present invention;

[0103] In the attached figures, the reference numerals are: 10, acquisition unit; 20, mining unit; 30, analysis unit; 40, extraction unit; 50, processing unit; and 60, optimization unit. Detailed Implementation

[0104] To enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0105] It is understood that the specific embodiments and accompanying drawings described herein are merely for explaining the invention and are not intended to limit the invention.

[0106] It is understood that, without conflict, the various embodiments and features in the embodiments of the present invention can be combined with each other.

[0107] It is understood that, for ease of description, only the parts related to the present invention are shown in the accompanying drawings, while the parts unrelated to the present invention are not shown in the drawings.

[0108] It is understood that each unit or module involved in the embodiments of the present invention may correspond to only one entity structure, or may be composed of multiple entity structures, or multiple units or modules may be integrated into one entity structure.

[0109] It is understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of this invention may occur in a different order than that marked in the accompanying drawings.

[0110] It is understood that the flowcharts and block diagrams of this invention illustrate the possible architecture, functions, and operations of systems, apparatuses, devices, and methods according to various embodiments of this invention. Each block in the flowchart or block diagram may represent a unit, module, program segment, or code, containing executable instructions for implementing the specified function. Furthermore, each block or combination of blocks in the block diagram and flowchart can be implemented using a hardware-based system to achieve the specified function, or using a combination of hardware and computer instructions.

[0111] It is understood that the units and modules involved in the embodiments of the present invention can be implemented by software or by hardware. For example, the units and modules can be located in a processor.

[0112] Example 1:

[0113] like Figure 1 and Figure 2 As shown in the figure, this embodiment provides a method for automatically converting cross-format documents to Markdown documents, the method including the following steps:

[0114] Step S1: Obtain the document image.

[0115] Step S2: Extract hidden structural information from the document image. The structural information is the hierarchical relationship between target elements; the target elements are headings, and / or paragraphs, and / or lists, and / or tables.

[0116] Specifically, step S2 includes the following steps:

[0117] Step S21: Use a convolutional neural network to extract features from the acquired document image to obtain a feature map.

[0118] In step S21, the convolutional neural network uses an existing convolutional neural network, such as AlexNet, VGGNet, GoogLeNet (Inception), ResNet, and EfficientNet, which have been widely used and validated. These networks perform well in image processing and feature extraction tasks and can effectively capture important features in document images.

[0119] Step S22: Input the feature map into a preset RPN to generate candidate regions. RPN (Region Proposal Network) is a deep learning model for object detection.

[0120] Step S23: Use RoIPooling to map each candidate region to a fixed-size feature representation, obtaining the feature representation of the candidate region. RoIPooling (Region of Interest Pooling) is a technique used in computer vision, whose main purpose is to extract information about specific regions from feature maps to correspond to specific targets in the input image. In this way, it can effectively process and analyze target regions, thereby improving the accuracy of target detection and recognition.

[0121] Step S24: Input the feature representation of the candidate region into the fully connected layer for processing to obtain the coordinate positions of the key elements; the key elements include titles, paragraphs, lists and tables.

[0122] Step S25: Based on the coordinates of the key elements, analyze the spatial layout and relative positional relationships of the key elements, and infer the hierarchical relationships between the key elements.

[0123] Step S26: Organize the coordinate positions of the key elements and the hierarchical relationships between the key elements to generate a tree-like or graphical representation that conforms to the logical structure, thereby obtaining the document image structure information.

[0124] Step S3: Based on the structural information, perform semantic content analysis on the document image to obtain the semantic content analysis results.

[0125] Step S3 specifically involves:

[0126] By using recurrent neural networks and contextual coding, the structural information of document images is analyzed and identified, and paragraph types, form structures and code block content are extracted to obtain comprehensive semantic content analysis results.

[0127] The method utilizes recurrent neural networks and context encoding to analyze and identify the structural information of document images, extracting paragraph types, form structures, and code block content to obtain comprehensive semantic content analysis results. Specifically, it includes the following steps:

[0128] Step S31: Segment the document image structure information into paragraphs or sentences to form a text sequence;

[0129] Step S32: Input the text sequence into the Bi-LSTM model for context modeling, and dynamically focus on key indicators through an attention mechanism; Bi-LSTM (Bidirectional Long Short-Term Memory) is a model used in sequence data processing, especially natural language processing (NLP). Based on the LSTM (Long Short-Term Memory) network architecture, it can effectively capture contextual information in sequence data. The key indicators are important elements or information in the document image;

[0130] Step S33: Based on the aforementioned key indicators, use MLP to classify each paragraph, identifying the paragraph type as plain text, form content, or code block; and,

[0131] Regular expressions are used to extract key fields from the form content and structure the form information to obtain the form structure; and,

[0132] Perform syntactic analysis on the code block to obtain its semantics and thus its content;

[0133] Step S34: Integrate the paragraph type, the form structure, and the code block content to form a semantic content integration result;

[0134] Step S35: Optimize and complete the semantic content integration result to obtain a comprehensive semantic content analysis result. In practice, this will be combined with domain knowledge and external resources.

[0135] Step S4: Based on the semantic content analysis results, extract the key information of the document image.

[0136] Step S4 specifically includes:

[0137] Based on the semantic content analysis results, a graph-based sequence transformation model is used to extract key information from document images;

[0138] The extraction of key information from document images based on semantic content analysis results using a graph-based sequence transformation model includes the following steps:

[0139] Step S41: Transform the semantic content analysis results into a form topology structure;

[0140] In this form topology, each node represents an input field, and each edge represents a relationship between fields.

[0141] Step S42: Encode the form topology using a graph attention network to obtain the form topology encoding information;

[0142] Step S43: Based on the graph convolution and attention mechanism of the graph attention network, learn the interaction and dependency relationship between nodes in the table topology structure according to the table topology structure encoding information, and generate node representation vectors of context information;

[0143] Step S44: Based on the node representation vector, use a conditional random field to perform sequence labeling on the input field to identify key information of the document image;

[0144] The key information includes the input fields and layout in the document image.

[0145] Step S5: Convert the key information into Markdown format and perform preliminary formatting in the form of a table or list to obtain a preliminary Markdown form.

[0146] Step S5 specifically includes the following steps:

[0147] Step S51: Annotate the key information to obtain the annotated field sequence;

[0148] Step S52: Input the annotated field sequence into the Transformer-based sequence transformation model, convert the field sequence into a structured Markdown format through a self-attention mechanism, and generate corresponding Markdown tags by filling in templates according to the semantic category of the fields and the form layout information.

[0149] Specifically, for table-type forms, Markdown's table syntax is used to generate a structured table representation; or, for list-type forms, Markdown's list syntax is used to generate structured list items.

[0150] Step S53: Generate a preliminary Markdown form based on the Markdown tags.

[0151] Step S6: Optimize the initial Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0152] Step S6 specifically includes the following steps:

[0153] Step S61: Compare the layout structure differences between the document image and the preliminary Markdown form, and use scripts or plugins to validate and dynamically display the form structure of the preliminary Markdown form to obtain a first optimized Markdown form; and,

[0154] Based on the structural information, the title and subtitles of the initial Markdown form are generated accordingly, and the lists and code blocks in the initial Markdown form are syntactically formatted to obtain the second optimized Markdown form; and,

[0155] The list items in the initial Markdown form are analyzed to identify their indentation, numbering, or symbolic features. Based on this, unordered and ordered lists are converted into Markdown format, resulting in a third optimized Markdown form.

[0156] Specifically, after comparing the layout differences between the document image and the initial Markdown form, JavaScript and its libraries (such as jQuery or Vanilla JavaScript), front-end frameworks (such as React or Vue.js), and form validation libraries (such as Formic or Yup) can be used to validate and dynamically display the form structure. Specific optimization directions include ensuring layout consistency, implementing responsive design, performing real-time data validation, and dynamically showing or hiding form fields based on user input, thereby improving user experience and reducing erroneous submissions. Using these tools and optimization directions, a first optimized version of the Markdown form can be obtained.

[0157] Based on the structural information, clear titles and subheadings can be generated for the initial Markdown form, and the lists and code blocks can be syntactically formatted to obtain the second optimized Markdown form. This includes setting the main title to "Step Data Form," with subheadings such as "Basic Information," "Step Records," and "Code Examples" to better organize and display information. Simultaneously, steps and code examples are presented in an embedded and ordered list format, making them more readable and logical. The specific optimization focuses on improving the form's structure and readability. By appropriately using titles, subheadings, and formatted text, the content becomes more intuitive and organized, facilitating user understanding and completion, thereby enhancing the overall user experience.

[0158] Step S62: Integrate the first optimized Markdown form, the second optimized Markdown form, and the third optimized Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0159] This implementation first extracts the titles, subheadings, and key content from all versions to ensure complete and unique information. Then, it standardizes the format, improving readability through consistent syntax and structure (such as lists and code blocks). Next, it integrates dynamic interactive features and form validation logic from each version to enhance user experience. Finally, it polishes the language to make the form content concise and clear. The resulting Markdown form will have a clear structure, consistent format, complete information, good readability, and dynamic interactivity, effectively meeting the needs of user completion and data collection.

[0160] As a specific implementation method, in step S61, the layout structure differences between the document image and the preliminary Markdown form are compared, and the form structure of the preliminary Markdown form is validated and dynamically displayed using a script or plug-in tool to obtain a first optimized Markdown form. This specifically includes the following steps:

[0161] Step A1: Use the edit distance algorithm to compare the layout structure differences between the document image and the initial Markdown form to obtain the difference metric value;

[0162] Specifically, the edit distance algorithm is a quantification of the difference in the number of insertion, deletion, and replacement operations;

[0163] Step A2: Based on the difference metric, automatically optimize the layout structure of the initial Markdown form to obtain the first-process Markdown form; the automatic optimization includes automatically adjusting the alignment, indentation level, and blank line separation.

[0164] Step A3: Use a matching script tool or plugin to parse and optimize the Markdown form from the first process, generate a display template, and use a preset template engine for rendering and data binding to obtain the first optimized Markdown form;

[0165] Among them, the supporting script tools or plugins integrate verification and error correction functions, and automatically check the field type, value range and required fields through predefined rules, and provide error prompts and correction suggestions;

[0166] In step S61, based on the structural information, the title and subtitles of the preliminary Markdown form are generated accordingly, and the lists and code blocks in the preliminary Markdown form are syntactically formatted to obtain the second optimized Markdown form. Specifically, this includes the following steps:

[0167] Step B1: Based on the structural information, the document content is automatically annotated using a conditional random field sequence labeling model to obtain the annotation results;

[0168] The annotation results are used to identify different document elements and extract context and location information features;

[0169] Step B2: Based on the annotation results, automatically generate Markdown format headings and subheadings to obtain a preliminary Markdown form with hierarchical relationships;

[0170] Step B3: By performing multi-level list indentation and line breaks on the initial Markdown form with hierarchical relationships, and by formatting the code blocks in the initial Markdown form with hierarchical relationships, a second-process optimized Markdown form is obtained.

[0171] Step B4: Parse and render the second process optimized Markdown form into an HTML page or PDF file corresponding to the visual structure of the document image. By comparing and evaluating the accuracy and completeness of the Markdown text generation, continuously optimize the sequence labeling model and Markdown generation rules to form the second optimized Markdown form.

[0172] In step S61, the list items in the initial Markdown form are analyzed to identify their indentation, numbering, or symbolic features. Based on this, the unordered and ordered lists are converted into Markdown format to obtain the third optimized Markdown form. This process includes the following steps:

[0173] Step C1: Take the list items in the initial Markdown form as input, use the Conditional Random Field (CRF) model to perform sequence labeling to identify indentation, numbering, and symbol features, construct feature templates and extract relevant attributes to obtain new list item text;

[0174] The Conditional Random Field (CRF) model was trained using gradient descent and the L-BFGS algorithm.

[0175] Step C2: Use the trained Conditional Random Field (CRF) model to perform sequence labeling on the new list item text, identify the structural features of each list item, determine its list type, and obtain lists of different types;

[0176] Step C3: Based on the logical consistency of indentation and numbering for the different types of lists, generate a third-process optimized Markdown form;

[0177] Step C4: Based on the list item text in the document image, evaluate and optimize the list in the third optimized Markdown form to obtain the third optimized Markdown form.

[0178] The specific implementation process of this embodiment is as follows:

[0179] Step 1. Preprocessing steps: Document format identification and content type differentiation

[0180] (1) Identify the input document format by file extension, such as .pdf, .docx, .pptx, etc. Based on the identification results, select an appropriate parsing tool for processing: for PDF documents, use PDF parsing tools such as PyMuPDF, pdfplumber, etc.; for Word documents, use the python-docx tool; for PPT documents, use the python-pptx tool.

[0181] (2) For documents containing text content (such as Word and PDF), extract the text portion and save it as a string; for documents containing images (such as PDF and PPT), extract the embedded images and save them as separate files or objects; in Word documents, extract table and list structures, and in PDF documents, use a table recognition tool (such as pdfplumber) to extract table structures.

[0182] Step 2. Document Image Feature Extraction

[0183] (1) Feature Extraction. A convolutional neural network is used to extract features from the document image. By using convolutional kernels of different sizes and multiple convolutional layers, the positions of headings, paragraphs, list elements, and table elements are automatically obtained. For example, the first convolutional layer can use 64 3x3 kernels with a stride of 1 and padding of 1; the second convolutional layer can use 128 3x3 kernels with a stride of 1 and padding of 1; and the third convolutional layer can use 256 3x3 kernels with a stride of 1 and padding of 1. Max pooling can be used, with a kernel size of 2x2 and a stride of 2.

[0184] (2) Generating Candidate Regions. A convolutional neural network is used to extract feature maps, which are then used to generate candidate regions via a region proposal network (such as Faster R-CNN). Region interest pooling (RoIPooling) is then used to obtain fixed-size feature representations. The region proposal network generates multiple candidate regions of different scales and aspect ratios at each location by sliding a small network across the convolutional feature map. Bounding box regression and classification are used to optimize the location and category of the candidate regions. For example, three different scales (e.g., 128x128, 256x256, 512x512) and three different aspect ratios (e.g., 1:1, 1:2, 2:1) can be used on the convolutional feature map to generate a total of nine different candidate regions.

[0185] (3) Classification and Regression. The feature representations of the candidate regions are input into the fully connected layer for classification and location regression. The fully connected layer uses the ReLU activation function and the cross-entropy loss function for multi-class classification, and the SmoothL1 loss function for location regression to determine whether each candidate region contains the target element and obtain its precise location coordinates. The fully connected layer can use two hidden layers, each containing 1024 neurons, using the ReLU activation function. For classification tasks, the softmax function can be used to output the probability of each class; for location regression tasks, a linear function can be used to output the coordinate adjustment values ​​of the bounding box.

[0186] (4) Hierarchical Relationship Inference. Based on the obtained position coordinates of headings, paragraphs, lists, and tables, the hierarchical relationship of elements in the document structure is inferred and determined by analyzing the spatial layout and relative positional relationships between elements, such as geometric features like distance, overlap, and alignment. This constructs the document's hierarchical structure tree. A graph-based approach can be used, first representing each element in the document as a node in a graph, and then adding directed edges between nodes based on the spatial layout relationships (such as distance, overlap, and alignment) to construct a directed acyclic graph (DAG). By performing a topological sort on the DAG, the hierarchical relationship between elements can be obtained. If the distance between element A and element B is less than 10 pixels, and A is above B, then a directed edge from A to B is added.

[0187] (5) Organization and Encoding. The identified document structure information is organized and encoded. By recursively traversing the hierarchical structure tree of the document, the type, position, content and other information of each element are encoded into a data structure in a specific format, such as JSON or XML, to generate a tree-like or graphical representation that conforms to the document's logical structure for use in subsequent semantic understanding and information extraction tasks.

[0188] (6) Post-processing and optimization. The extracted document structure information is post-processed and optimized. This involves combining prior knowledge, such as document type, layout, and domain, with heuristic rules, such as the font size and position of headings, and paragraph indentation and spacing, to verify and correct the recognition results. This corrects potential errors and omissions, improving the accuracy and completeness of document structure recognition. For example, if the recognized heading is located at the bottom of the document, or if the paragraph width exceeds 80% of the document width, it can be considered a recognition error and corrected accordingly.

[0189] Step 3: Semantic content analysis of structural information

[0190] (1) Structural Information Input and Encoding. Based on the obtained document structural information, it is input into a recurrent neural network. The text sequence is encoded through a bidirectional long short-term memory network (Bi-LSTM) to capture contextual information and long-distance dependencies in the text, generating semantic representation vectors with context-aware capabilities. For example, when encoding text sequences using Bi-LSTM, the hidden layer dimension can be set to 256, the Adam optimizer can be used, the learning rate can be 0.001, the batch size can be 32, and the training can be performed for 50 epochs.

[0191] (2) Introducing an attention mechanism. During the encoding process, an attention mechanism, such as LuongAttention, is introduced. This mechanism dynamically generates attention weights by calculating the similarity between the query vector and the encoding vector. The encoding vector is then weighted and summed to dynamically focus on key information relevant to the current task, improving the accuracy of semantic analysis. LuongAttention calculates the similarity between the current decoder state and the encoder states at each time step, uses the softmax function to normalize the similarity into attention weights, and then weights and sums the encoder states to obtain the attention vector.

[0192] (3) Paragraph Attribute Classification. Based on the semantic representation vector of the text, a multilayer perceptron (MLP) is used to classify different paragraph attributes, determining whether each paragraph belongs to different types such as plain text, form content, or code block. The MLP can contain two hidden layers, each with 256 neurons, using the ReLU activation function. The probability distribution of each category is calculated through nonlinear transformation and the softmax function to obtain the classification label of the paragraph.

[0193] (4) Integration of layout and format features. When classifying paragraph attributes, the layout and format features of the paragraphs, such as paragraph indentation, line spacing, and font, are integrated to further improve the accuracy of classification. Layout and format features can be obtained by analyzing the document's style information and visual features.

[0194] (5) Form Content Recognition. For form content paragraphs, a rule-based approach is used for further recognition and extraction. Regular expressions and matching rules are defined to identify key fields and fields to be filled in, and to extract structured information. For example, the regular expression `name: \s*(\S+)` is used to match the name field, and `\d{4}-\d{2}-\d{2}` is used to match the date field. The matching results are then organized into JSON, CSV, or database record formats.

[0195] (6) Code block identification. By analyzing the syntactic features and keywords of code blocks, such as "def" and "class" in Python, and "public" and "static" in Java, the programming language and functional modules of the code can be identified, and the semantic information of the code can be extracted. For example, the start and end positions of the code block can be determined by the frequency and position of the keywords.

[0196] (7) Semantic structure fusion. The results of paragraph attribute classification are fused with the recognition results of forms and code blocks to form a complete semantic structure representation of the document, including the type of each paragraph, the structured information of forms, and the semantic information of code blocks.

[0197] (8) Optimization and Completion. During the generation of semantic structure representations, the recognition results are verified and corrected by combining resources such as domain knowledge bases, external dictionaries, and ontology, and missing semantic information is inferred and filled in. For example, external knowledge bases such as WordNet and WikiData are used to infer missing semantic information through synonym relationships, hyponyms, etc. If the recognized form is missing the "country" field but has a "city" field, the corresponding country information can be inferred based on the subordinate relationship between the city and the country.

[0198] Step 4: Extract key information and format Markdown

[0199] (1) Form diagram structure construction. For example Figure 4 As shown, the identified form content is first transformed into a graph structure, where each input field of the form is represented as a node in the graph, and the relationships between fields are represented as edges, thus constructing the form's topology. For example, for a form containing 10 fields, a 10x10 adjacency matrix can be constructed, where the elements of the matrix represent the relationships between the fields, with 1 indicating the existence of a relationship and 0 indicating the absence of a relationship.

[0200] (2) Graph Attention Network Encoding. A graph attention network is used to encode the scalar graph. Through graph convolution and attention mechanisms, the interactions and dependencies between nodes are learned, generating node representation vectors with contextual information. The graph attention network consists of 3 graph convolutional layers, each with 8 attention heads, and uses the ReLU activation function. The output dimension is 64, and L2 regularization and Dropout are used to prevent overfitting. The optimizer is Adam, and the learning rate is set to 0.001.

[0201] (3) Conditional Random Field Sequence Labeling. Based on the node representation vector, a conditional random field is used to label the input fields, identifying the semantic category of each field, such as name, date, and address. The conditional random field uses unigram and bigram features, decoded using the Viterbi algorithm, to obtain the optimal labeled sequence. Feature selection can include linguistic features such as part-of-speech tagging, named entity type, and dependency relations, as well as layout features such as the field's position and length in the form.

[0202] (4) Transformer-based sequence transformation. The labeled field sequence is input into a Transformer-based sequence transformation model. Through a self-attention mechanism and a feedforward neural network, the field sequence is transformed into a structured Markdown format. The Transformer model contains 6 encoder layers and 6 decoder layers, with 8 attention heads per layer. The feedforward network has a dimension of 2048 and uses the BeamSearch algorithm for decoding with a search width of 5.

[0203] (5) Markdown Format Generation. During Markdown format generation, based on the semantic categories of fields and the layout information of the form, template-based filling is used to generate corresponding Markdown tags and formats. For example, some common templates are predefined, such as "##Name: {name}", "-Date: {date}", etc., and then dynamically filled based on the field annotation results. For table-type forms, Markdown's table syntax is used, with field names as table headers and field values ​​as table cells, to generate a structured table representation; for list-type forms, Markdown's list syntax is used, with field names as list items and field values ​​as list content, to generate a structured list representation.

[0204] (6) Form Type Judgment and Processing. Form type is determined through heuristic rules based on the analysis of keywords and layout features in the form. For example, table-type forms usually contain consecutive vertical lines "|", while list-type forms usually contain consecutive "-" or "*". For mixed-type forms, a segmented processing strategy is adopted, extracting the table and list parts separately, generating Markdown content for each part separately, and then merging the generated Markdown content.

[0205] (7) Field value formatting. During the Markdown generation process, necessary formatting processing is performed on field values, such as date format conversion, number formatting, and text length limits, to ensure the readability and consistency of the generated Markdown content. For example, the template for a date field is "-date:{date:YYYY-MM-DD}", and the template for a number field is "-amount:{amount:,##0.00} yuan", etc.

[0206] Step 5: Optimize Markdown form layout

[0207] (1) Structural Difference Comparison. The generated Markdown formatted form is compared with the original form, and the Levenshtein distance algorithm is used to calculate the structural difference between the two. The Levenshtein distance algorithm measures the difference between two strings by calculating the minimum number of editing operations (insert, delete, replace) required to convert one string into another. For example, for a form containing 10 fields, it can be converted into a string of length 10, where each character represents the type of a field (e.g., "T" for text field, "N" for numeric field), and then the Levenshtein distance between the two strings is calculated using a dynamic programming algorithm to obtain the difference metric.

[0208] (2) Layout and Structure Optimization. Based on the difference metric, the layout and structure of the Markdown form are automatically optimized and adjusted to maintain consistency with the original form's layout as closely as possible. For example, for a table with 5 rows and 3 columns, the optimal width of each column can be calculated based on the length of the cell content. Then, the cell width of each row is adjusted sequentially to make it as close as possible to the optimal width until all rows meet the requirements. During the optimization process, a series of layout optimization rules are defined, such as adaptive table column width, indentation alignment of list items, and first-line indentation of paragraphs. Greedy algorithms or dynamic programming algorithms are used to structurally adjust and beautify the Markdown form.

[0209] (3) Develop a matching script tool using Python. This tool takes the optimized Markdown form as input, parses its structured information using regular expressions, and generates an HTML template corresponding to the original form. For example, when validating and correcting the form structure, built-in Python modules such as `re` (regular expressions) and `datetime` (date and time) can be used to check the type, format, and value range of each field in the form. For a date field, the regular expression `"^\d{4}-\d{2}-\d{2}$"` can be used for format checking, and the `datetime` module can be used to parse the date string and determine its validity.

[0210] (4) Template rendering and dynamic display. Through the Jinja2 template engine's rendering and Vue.js's data binding, the visual display and interactive operations of form content are achieved, such as table sorting, filtering, and pagination. For example, for a form containing user information, a table can be defined in the HTML template. The rows and columns of the table can be associated with the form data using Jinja2 syntax, and then the Vue.js v-for directive can be used to render the table, enabling dynamic display and updating of user information.

[0211] (5) Form Structure Validation and Error Correction. The script tool integrates form structure validation and error correction functions. Through predefined rules and constraints, such as field type (text, number, date, etc.), value range (enumerated values ​​or regular expressions), and required fields, it automatically checks and validates the form content and provides corresponding error messages and correction suggestions. For example, for a numeric field, the regular expression "^\d+(.\d+)?$" can be used for format checking, and the int or float function can be used to convert the string to a number to determine if it falls within the specified value range.

[0212] (6) User feedback and optimization. In practical applications, user feedback is collected through methods such as user satisfaction surveys and A / B testing. Based on the feedback results, the algorithm and strategy for optimizing the form structure are optimized to provide more intelligent and user-friendly services, reduce manual operations and repetitive work for users, and improve the efficiency and quality of form processing.

[0213] Step 6: Generating Markdown-formatted headings and subheadings

[0214] (1) Sequence Labeling and Document Element Recognition. Using the mined document structure and hierarchical information as input, sequence labeling models such as Conditional Random Fields (CRF) are employed to automatically label and segment document content, identifying different document elements such as titles, subheadings, paragraphs, lists, and code blocks. For example, when using a CRF model for sequence labeling, linguistic features such as part-of-speech tags, named entities, and dependency relations can be selected to construct feature templates. Unigram and bigram features of part-of-speech tags, unigram features of named entity tags, direction and distance features of dependency relations, and structural features such as the relative position and length range of text blocks can be used. Approximately 1000 manually labeled documents are used as training data, and 10-fold cross-validation and F1 scores are used to evaluate model performance.

[0215] (2) Generate Markdown format. Based on the sequence labeling results, automatically generate Markdown formatted headings and subheadings. For example, for headings, the appropriate number of "#" symbols can be added according to their level information, such as one "#" for a first-level heading, two "#" for a second-level heading, and so on, supporting up to six levels of headings. For lists, by analyzing the indentation, numbering, bullet points, and other information of the list items, the type and level of the list are determined, and the corresponding Markdown syntax format is generated. If a list item begins with a symbol such as "-", "+", or "*", an unordered list is generated; if it begins with a number or ".", an ordered list is generated. For code blocks, by analyzing the indentation and comment information of the code, the programming language of the code and the starting position of the code block are determined, and the corresponding Markdown syntax format is generated.

[0216] (3) Text content formatting. During the Markdown generation process, necessary escaping and formatting are performed on the text content. For example, special characters are escaped, links and images are processed using syntax, and emphasis and bolding are marked to ensure the correctness and readability of the generated Markdown text.

[0217] (4) Markdown parsing and HTML rendering. Existing Markdown parsers and rendering engines such as Python-Markdown and Pandoc are used to parse the generated Markdown text into HTML format. Then, CSS styles and JavaScript are used to format and render the HTML page. For example, the Python-Markdown library is used for parsing, and the Bootstrap CSS framework is used for formatting the HTML page. Media queries and responsive design are used to achieve the best display effect on different devices and screen sizes.

[0218] (5) Automated Testing and Evaluation. The generated Markdown text and HTML pages are parsed and evaluated. The accuracy, completeness, and readability of the generated Markdown text, as well as the correctness and consistency of the rendered HTML pages, are assessed through a combination of manual review and automated testing. For example, automated testing tools such as Selenium are used to perform functional and UI tests on the generated HTML pages, including link clickability, image loading speed, and page rendering time, and test reports are generated.

[0219] (6) Visual Structure Optimization. Consider using computer vision and image processing technologies, such as OpenCV, to analyze and extract features from the original document's images, identifying different layout elements such as text, images, and tables, and calculating their position, size, alignment, and other attributes. For example, use the OpenCV library to analyze the original document's images, perform edge detection, contour detection, and rectangle fitting using the Canny operator to extract layout elements such as text, images, and tables, calculate their coordinates, width, height, alignment, and other attributes, and then map these attributes to corresponding CSS styles, such as the margin and padding of text blocks, the width and height of images, and the border-collapse and width of table elements, to achieve the greatest possible restoration of the original document's visual effect.

[0220] Step 7: List Item Analysis and Transformation

[0221] (1) Sequence Labeling and Structural Feature Recognition. The generated list item text is used as input, and a Conditional Random Field (CRF) model is employed for sequence labeling and analysis to identify the structural features and format attributes of the list items. A feature template for the CRF model is defined, extracting features such as the indentation level, number or symbol type, text length, and contextual information of the list items to construct a feature function. For example, when defining the feature template for the CRF model, features such as the indentation level (integer value), the symbol type (e.g., "-", "+", "*", number + "."), the text length (integer value), and the contextual information (e.g., the symbol type and indentation level of the previous and next items) can be used.

[0222] (2) Training and evaluating the CRF model. A training dataset was created by manually labeling approximately 1000 list item samples. The L-BFGS algorithm was used to train the CRF model, with appropriate iteration counts and gradient thresholds set to learn the weight parameters of the feature functions. For example, 1000 list item samples were manually labeled, with 800 used as the training set and 200 as the test set. The L-BFGS algorithm was used to train the CRF model, with 100 iterations and a gradient threshold of 0.001. The model's performance was evaluated on the test set, yielding an accuracy of 0.95, a recall of 0.93, and an F1 score of 0.94.

[0223] (3) Handling out-of-vocabulary (OV) and rare words. During feature extraction, dictionary-based features or integrated external knowledge bases are used to enhance feature representation. For example, OV and rare words can be classified into the "UNK" class and dictionary-based features can be used, such as f(x,i) = 1 if x is in dictionary D else 0, where D can be an external knowledge base or domain dictionary.

[0224] (4) Sequence labeling and Markdown format generation. The trained CRF model is used to perform sequence labeling on the new list item text, predicting the structural features and format attributes of each list item. Based on the labeling results, the list type to which the list item belongs is determined. For example, lists starting with "-", "+", or "" and with an indentation level of 0 are unordered lists; lists starting with numbers or "." and with an indentation level of 0 are ordered lists; and lists with an indentation level greater than 0 are multi-level lists. Specifically, for list items with an indentation level of 0 and starting with "-", "+", or "", "-", "+", or "*" are generated; for list items with an indentation level of 0 and starting with numbers plus ".", "1.", "2.", etc., are generated; for list items with an indentation level of n (n>0), n spaces plus the corresponding list symbols are generated.

[0225] (5) Comparison and Evaluation. The generated Markdown formatted lists are compared with the original list item text to evaluate the accuracy and completeness of the conversion. Special attention is paid to list items with conversion errors, the causes are analyzed, and the CRF model and conversion rules are adjusted accordingly. For example, using random sampling, 100 converted lists were manually inspected, and 5 were found to have formatting errors, an error rate of 5%. Analysis of the causes of these errors revealed that they were mainly due to special characters or irregular indentation in the list items.

[0226] (6) Optimization and User Feedback. By analyzing the causes of errors, we further improve the feature templates and conversion rules to enhance the robustness of the conversion. At the same time, we collect user feedback, such as conversion speed and format aesthetics, and continuously iterate and optimize to improve the user experience.

[0227] Step 8: Merging Multiple Document Structures

[0228] (1) Input Format Conversion. The generated Markdown documents are converted to a sequence-to-sequence model input format. The structural information of each document is extracted and connected using special delimiters to form a long sequence. For example, special delimiters such as "" can be used. <doc> "and"< / doc> "To identify the start and end positions of each document, use..." <sec> "and"< / sec> "To identify the beginning and end of each chapter, use..." "and" This is used to identify the beginning and end of each paragraph, thereby encoding the document's structural information into the input sequence.

[0229] (2) Encoding and Attention Mechanisms. An encoder-decoder architecture is adopted, using a bidirectional recurrent neural network (Bi-RNN) to encode the input sequence, generating structural and semantic representation vectors for each document. During the encoding process, attention mechanisms such as LuongAttention or BahdanauAttention are used to calculate attention weights based on the encoded vectors of the current document and other documents, resulting in a weighted average context vector. For example, a two-layer LSTM with 256 hidden units can be used as the basic unit, and LuongAttention can be used to calculate attention weights at each time step.

[0230] (3) Decoding and Generation. The structural and semantic representation vectors of multiple documents are concatenated and used as input to the decoder. This concatenation is then processed by another recurrent neural network (RNN) to generate a unified Markdown document structure. During the decoding process, a gating mechanism controls the flow of information, selectively fusing structural information from different documents to maintain overall coherence and consistency. For example, when using an RNN for decoding, a two-layer LSTM with 512 hidden units can be used as the basic unit, and the beamsearch algorithm can be used for decoding, with the beamsize set to 5.

[0231] (4) Loss function design. During the generation process, by designing a suitable loss function, such as cross-entropy loss, the model parameters can be optimized to improve the quality and accuracy of the generation. For example, the cross-entropy loss and the ROUGE-L score can be weighted and averaged in an 8:2 ratio as the final optimization objective.

[0232] (5) Post-processing and optimization. The generated Markdown document undergoes post-processing and optimization. Topic models such as LDA or TextRank are used to extract topic words, and TextRank or term frequency statistics are used to extract keywords. The document is then reorganized and adjusted based on the topic words and keywords. For example, the TextRank algorithm can be used to extract the top-10 keywords, and paragraphs containing the same keywords can be merged into the same chapter.

[0233] (6) Semantic Enhancement. Utilize external knowledge bases and semantic resources, such as WordNet and ConceptNet, to link and disambiguate key concepts and entities in the document, thereby enhancing the semantic coherence of the document. For example, ConceptNet, with its broad coverage, can be selected as the primary knowledge source, and a graph attention network-based method can be used to learn the semantic relationships between words and concepts.

[0234] (7) Manual evaluation and optimization. The generated Markdown documents are manually evaluated and feedback is provided. Based on the evaluation results and user feedback, the sequence-to-sequence model and document generation strategy are continuously iterated and optimized. For example, 50 sample documents can be selected, and 3 experts can score the generated Markdown documents. The scoring dimensions include structural coherence, semantic coherence, readability, etc. A 5-point scale is used, and the average value is taken as the final score.

[0235] This embodiment can effectively identify and convert forms with complex nested structures and irregular layouts in documents, thereby ensuring the integrity and readability of form information when converting documents to Markdown format.

[0236] Example 2:

[0237] like Figure 3 As shown, this embodiment provides a form enhancement method based on automatic cross-format document conversion to Markdown. This method is primarily applied to scenarios such as document format conversion, article publishing, structured document generation, and team collaboration. It automatically converts documents of different formats (such as Word and PDF) to Markdown, improving document management efficiency and enhancing readability. The method includes the following steps:

[0238] Step K1: Obtain heterogeneous documents;

[0239] Step K2: Split the heterogeneous document into a text portion and a text-image portion;

[0240] Step K3: For the text portion, generate the first Markdown document corresponding to the text; and,

[0241] For the text image portion, the method for automatically converting cross-format documents to Markdown documents as described in Example 1 is used to generate a second Markdown document corresponding to the text image;

[0242] The structural and semantic coherence between all the first and second Markdown documents is analyzed, and the structural information between all the first and second Markdown documents is merged into a single Markdown document to complete the form enhancement based on automatic cross-format document to Markdown.

[0243] Example 3:

[0244] like Figure 5 As shown, this embodiment provides an apparatus for automatically converting cross-format documents to Markdown documents, the apparatus comprising:

[0245] Acquisition unit 10 is used to acquire document images;

[0246] The mining unit 20, connected to the acquisition unit 10, is used to mine hidden structural information from the document image;

[0247] The structural information refers to the hierarchical relationship between target elements; the target elements are headings, and / or paragraphs, and / or lists, and / or tables.

[0248] The analysis unit 30, connected to the mining unit 20, is used to perform semantic content analysis on the document image based on the structural information to obtain semantic content analysis results;

[0249] Extraction unit 40, connected to analysis unit 30, is used to extract key information of the document image based on semantic content analysis results;

[0250] Processing unit 50, connected to extraction unit 40, is used to convert the key information into Markdown format and perform preliminary formatting processing in the form of a table or list to obtain a preliminary Markdown form.

[0251] The optimization unit 60, connected to the processing unit 50, is used to optimize the preliminary Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0252] The apparatus in this embodiment is capable of performing the method in Embodiment 1.

[0253] Example 4:

[0254] This embodiment provides a form enhancement system based on automatic cross-format document conversion to Markdown, the system comprising:

[0255] Acquisition device, used to acquire heterogeneous documents;

[0256] A splitting device, connected to the acquisition device, is used to split the heterogeneous document into a text portion and a text-image portion;

[0257] A generation device, connected to the splitting device, is used to generate a first Markdown document corresponding to the text portion;

[0258] The apparatus for automatically converting cross-format documents to Markdown documents as described in Example 3 is connected to the splitting device and is used to generate a second Markdown document corresponding to the text image portion.

[0259] The merging device is connected to the generating device and the device for automatically converting cross-format documents to Markdown documents, respectively. It is used to merge all the first Markdown documents and all the second Markdown documents into a complete Markdown document based on the structural information and semantic coherence between the first Markdown document and the second Markdown document, so as to complete the form enhancement based on automatic conversion of cross-format documents to Markdown.

[0260] The system in this embodiment can execute the method in embodiment 2.

[0261] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for automatically converting cross-format documents to Markdown documents, characterized in that, The method includes the following steps: Step S1: Obtain the document image; Step S2: Extract hidden structural information from the document image; The structural information refers to the hierarchical relationship between target elements; the target elements are headings, and / or paragraphs, and / or lists, and / or tables. Step S3: Based on the structural information, perform semantic content analysis on the document image to obtain the semantic content analysis results; Step S4: Based on the semantic content analysis results, extract the key information of the document image; Step S5: Convert the key information into Markdown format and perform preliminary formatting in the form of a table or list to obtain a preliminary Markdown form; Step S5 specifically includes the following steps: Step S51: Annotate the key information to obtain the annotated field sequence; Step S52: Input the annotated field sequence into the Transformer-based sequence transformation model, convert the field sequence into a structured Markdown format through a self-attention mechanism, and generate corresponding Markdown tags by filling in templates according to the semantic category of the fields and the form layout information. Specifically, for table-type forms, Markdown's table syntax is used to generate a structured table representation; or, for list-type forms, Markdown's list syntax is used to generate structured list items. Step S53: Generate a preliminary Markdown form based on the Markdown tags; Step S6: Optimize the initial Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents; Step S6 specifically includes the following steps: Step S61: Compare the layout structure differences between the document image and the preliminary Markdown form, and use scripts or plugins to validate and dynamically display the form structure of the preliminary Markdown form to obtain a first optimized Markdown form; and, Based on the structural information, the title and subtitles of the initial Markdown form are generated accordingly, and the lists and code blocks in the initial Markdown form are syntactically formatted to obtain the second optimized Markdown form; and, The list items in the initial Markdown form are analyzed to identify the indentation, numbering, or symbol features of the list items. Based on this, the unordered and ordered lists are converted into Markdown format to obtain the third optimized Markdown form. Step S62: Integrate the first optimized Markdown form, the second optimized Markdown form, and the third optimized Markdown form to obtain the target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

2. The method for automatically converting cross-format documents to Markdown documents according to claim 1, characterized in that, Step S2 specifically includes the following steps: Step S21: Use a convolutional neural network to extract features from the acquired document image to obtain a feature map; Step S22: Input the feature map into a preset RPN to generate candidate regions; Step S23: Use RoIPooling to map each candidate region to a feature representation of a fixed size to obtain the feature representation of the candidate region; Step S24: Input the feature representation of the candidate region into the fully connected layer for processing to obtain the coordinate positions of the key elements; the key elements include titles, paragraphs, lists and tables; Step S25: Based on the coordinate positions of the key elements, analyze the spatial layout and relative positional relationships of the key elements, and infer the hierarchical relationships between the key elements; Step S26: Organize the coordinate positions of the key elements and the hierarchical relationships between the key elements to generate a tree-like or graphical representation that conforms to the logical structure, thereby obtaining the document image structure information.

3. The method for automatically converting cross-format documents to Markdown documents according to claim 1, characterized in that, Step S3 specifically includes: By using recurrent neural networks and contextual coding, the structural information of document images is analyzed and identified, and paragraph types, form structures and code block content are extracted to obtain comprehensive semantic content analysis results. The method utilizes recurrent neural networks and context encoding to analyze and identify the structural information of document images, extracting paragraph types, form structures, and code block content to obtain comprehensive semantic content analysis results. Specifically, it includes the following steps: Step S31: Segment the document image structure information into paragraphs or sentences to form a text sequence; Step S32: Input the text sequence into the Bi-LSTM model for context modeling, and dynamically focus on key indicators through an attention mechanism; The key indicators are important elements or information in the document image; Step S33: Based on the aforementioned key indicators, use MLP to classify each paragraph, identifying the paragraph type as plain text, form content, or code block; and, Regular expressions are used to extract key fields from the form content and structure the form information to obtain the form structure; and, Perform syntactic analysis on the code block to obtain its semantics and thus its content; Step S34: Integrate the paragraph type, the form structure, and the code block content to form a semantic content integration result; Step S35: Optimize and complete the semantic content integration results to obtain comprehensive semantic content analysis results.

4. The method for automatically converting cross-format documents to Markdown documents according to claim 1, characterized in that, Step S4 specifically includes: Based on the semantic content analysis results, a graph-based sequence transformation model is used to extract key information from document images; The extraction of key information from document images based on semantic content analysis results using a graph-based sequence transformation model includes the following steps: Step S41: Transform the semantic content analysis results into a form topology structure; In this form topology, each node represents an input field, and each edge represents a relationship between fields. Step S42: Encode the form topology using a graph attention network to obtain the form topology encoding information; Step S43: Based on the graph convolution and attention mechanism of the graph attention network, learn the interaction and dependency relationship between nodes in the table topology structure according to the table topology structure encoding information, and generate node representation vectors of context information; Step S44: Based on the node representation vector, use a conditional random field to perform sequence labeling on the input field to identify key information of the document image; The key information includes the input fields and layout in the document image.

5. The method for automatically converting cross-format documents to Markdown documents according to claim 1, characterized in that, In step S61, the layout structure differences between the document image and the preliminary Markdown form are compared, and the form structure of the preliminary Markdown form is validated and dynamically displayed using a script or plugin tool to obtain the first optimized Markdown form. Specifically, this includes the following steps: Step A1: Use the edit distance algorithm to compare the layout structure differences between the document image and the initial Markdown form to obtain the difference metric value; Specifically, the edit distance algorithm is a quantification of the difference in the number of insertion, deletion, and replacement operations; Step A2: Based on the difference metric, automatically optimize the layout structure of the initial Markdown form to obtain the first-process Markdown form; the automatic optimization includes automatically adjusting the alignment, indentation level, and blank line separation. Step A3: Use a matching script tool or plugin to parse and optimize the Markdown form from the first process, generate a display template, and use a preset template engine for rendering and data binding to obtain the first optimized Markdown form; The accompanying script tools or plugins integrate validation and error correction functions, and automatically check field types, value ranges, and required fields through predefined rules, providing error prompts and correction suggestions.

6. The method for automatically converting cross-format documents to Markdown documents according to claim 1, characterized in that, In step S61, based on the structural information, the title and subtitles of the preliminary Markdown form are generated accordingly, and the lists and code blocks in the preliminary Markdown form are syntactically formatted to obtain the second optimized Markdown form. Specifically, this includes the following steps: Step B1: Based on the structural information, the document content is automatically annotated using a conditional random field sequence labeling model to obtain the annotation results; The annotation results are used to identify different document elements and extract context and location information features; Step B2: Based on the annotation results, automatically generate Markdown format headings and subheadings to obtain a preliminary Markdown form with hierarchical relationships; Step B3: By performing multi-level list indentation and line breaks on the initial Markdown form with hierarchical relationships, and by formatting the code blocks in the initial Markdown form with hierarchical relationships, a second-process optimized Markdown form is obtained. Step B4: Parse and render the Markdown form optimized in the second process into an HTML page or PDF file corresponding to the visual structure of the document image. By comparing and evaluating the accuracy and completeness of the Markdown text generation, continuously optimize the sequence labeling model and Markdown generation rules to form the second optimized Markdown form.

7. The method for automatically converting cross-format documents to Markdown documents according to claim 1, characterized in that, In step S61, the list items in the initial Markdown form are analyzed to identify their indentation, numbering, or symbolic features. Based on this, the unordered and ordered lists are converted into Markdown format to obtain the third optimized Markdown form. This process includes the following steps: Step C1: Take the list items in the initial Markdown form as input, use the Conditional Random Field (CRF) model to perform sequence labeling to identify indentation, numbering, and symbol features, construct feature templates and extract relevant attributes to obtain new list item text; The Conditional Random Field (CRF) model was trained using gradient descent and the L-BFGS algorithm. Step C2: Use the trained Conditional Random Field (CRF) model to perform sequence labeling on the new list item text, identify the structural features of each list item, determine its list type, and obtain lists of different types; Step C3: Based on the logical consistency of indentation and numbering for the different types of lists, generate a third-process optimized Markdown form; Step C4: Based on the list item text in the document image, evaluate and optimize the list in the third optimized Markdown form to obtain the third optimized Markdown form.

8. A form enhancement method based on automatic conversion of cross-format documents to Markdown, characterized in that, The method includes the following steps: Step K1: Obtain heterogeneous documents; Step K2: Split the heterogeneous document into a text portion and a text-image portion; Step K3: For the text portion, generate the first Markdown document corresponding to the text; as well as, For the text image portion, the method for automatically converting cross-format documents to Markdown documents as described in any one of claims 1 to 7 is used to generate a second Markdown document corresponding to the text image; The structural and semantic coherence between all the first and second Markdown documents is analyzed, and the structural information between all the first and second Markdown documents is merged into a single Markdown document to complete the form enhancement based on automatic cross-format document to Markdown.

9. An apparatus for automatically converting cross-format documents to Markdown documents, characterized in that, The device includes: The acquisition unit is used to acquire document images; A mining unit, connected to the acquisition unit, is used to mine hidden structural information from the document image; The structural information refers to the hierarchical relationship between target elements; the target elements are headings, and / or paragraphs, and / or lists, and / or tables. An analysis unit, connected to the mining unit, is used to perform semantic content analysis on the document image based on the structural information to obtain semantic content analysis results; An extraction unit, connected to the analysis unit, is used to extract key information from the document image based on the semantic content analysis results; A processing unit, connected to the extraction unit, is used to convert the key information into Markdown format and perform preliminary formatting processing in the form of a table or list to obtain a preliminary Markdown form. The specific process of obtaining the initial Markdown form is as follows: The key information is annotated to obtain the annotated field sequence; The annotated field sequence is input into a Transformer-based sequence transformation model. The field sequence is transformed into a structured Markdown format through a self-attention mechanism. The corresponding Markdown tags are generated by filling in templates based on the semantic category of the fields and the form layout information. Specifically, for table-type forms, Markdown's table syntax is used to generate a structured table representation; or, for list-type forms, Markdown's list syntax is used to generate structured list items. Based on the Markdown tags, a preliminary Markdown form is generated; An optimization unit, connected to the processing unit, is used to optimize the initial Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents; The process of obtaining the target Markdown form and thus automatically converting cross-format documents to Markdown involves the following steps: By comparing the layout and structure differences between the document image and the initial Markdown form, and using scripts or plugins to validate and dynamically display the structure of the initial Markdown form, a first optimized Markdown form is obtained; and, Based on the structural information, the title and subtitles of the initial Markdown form are generated accordingly, and the lists and code blocks in the initial Markdown form are syntactically formatted to obtain the second optimized Markdown form; and, The list items in the initial Markdown form are analyzed to identify the indentation, numbering, or symbol features of the list items. Based on this, the unordered and ordered lists are converted into Markdown format to obtain the third optimized Markdown form. By integrating the first, second, and third optimized Markdown forms, the target Markdown form is obtained, thereby completing the automatic conversion of cross-format documents to Markdown documents.

10. A form enhancement system based on automatic cross-format document conversion to Markdown, characterized in that, The system includes: Acquisition device, used to acquire heterogeneous documents; A splitting device, connected to the acquisition device, is used to split the heterogeneous document into a text portion and a text-image portion; A generation device, connected to the splitting device, is used to generate a first Markdown document corresponding to the text portion; The apparatus for automatically converting cross-format documents to Markdown documents as described in claim 9, connected to the splitting device, is used to generate a second Markdown document corresponding to the text image portion; The merging device is connected to the generating device and the device for automatically converting cross-format documents to Markdown documents, respectively. It is used to merge all the first Markdown documents and all the second Markdown documents into a complete Markdown document based on the structural information and semantic coherence between the first Markdown document and the second Markdown document, so as to complete the form enhancement based on automatic conversion of cross-format documents to Markdown.

Images