Method and device for automatically converting cross-format document into Markdown document and form enhancement method and system

Abstract

The invention discloses a method and device for automatically converting a cross-format document into a Markdown document and a form enhancement method and system. The Markdown document conversion method comprises the following steps: firstly, obtaining a document image; mining hidden structure information from the document image, wherein the hidden structure information comprises target elements and hierarchical relationships among the target elements; performing semantic content analysis on the document image based on the structure information to obtain a semantic content analysis result; extracting key information in the document image according to an analysis result; the key information is converted into a Markdown format, preliminary formatting is carried out in a form or list mode, and a preliminary Markdown form is obtained; and finally, optimizing the preliminary Markdown form to form a target Markdown form, so that the cross-format document is automatically converted into the Markdown. According to the method, a complex nested structure and a form with irregular layout in the document can be effectively identified and converted, and the integrity and readability of form information when the form is converted into the Markdown format are ensured.

Description

Technical Field

[0001] The present invention relates to a method and device for automatically converting a cross-format document into a Markdown document, and a form enhancement method and system. Background Art

[0002] Markdown is a lightweight markup language that has been widely used in technical document writing, blog publishing, and note-taking for its concise syntax and good compatibility. Its advantages include being easy to read and write, and being able to be stored in plain text for easy version control and cross-platform editing. With the popularity of Markdown, the demand for converting other formats of documents (such as Word documents, PDF, etc.) to Markdown is growing, prompting the development of related technologies.

[0003] In this context, some existing solutions have been proposed to achieve document format conversion. Among them, patent application number CN202311802017.4 proposes a conversion method based on supporting the retention of images, using the Spire component to read Office documents, and extract and save the image information therein to ensure that the content and format of the images can be effectively retained in the Markdown document. This method has played a positive role in the embedding and display of images. In addition, patent application number CN201510310288.7 describes a method for document format conversion, which includes loading and converting a Word document into a web document, and then converting the tags in the web document into Markdown. In this process, the style effect of the Markdown format document is ensured by retaining the style attributes. This technology aims to avoid the loss of style effects during the conversion process, thereby realizing the re-editing of Markdown documents.

[0004] Although existing technologies have made certain progress, they are unable to effectively identify and convert forms with complex nested structures and irregular layouts in documents, making it impossible to ensure the integrity and readability of form information when converting documents into Markdown format. Summary of the invention

[0005] The technical problem to be solved by the present invention is to address the above-mentioned deficiencies in the prior art and propose a method, device, and form enhancement method and system for automatically converting cross-format documents to Markdown documents. The method can effectively identify and convert forms with complex nested structures and irregular layouts in documents, thereby ensuring the integrity and readability of form information when converting documents to Markdown format.

[0006] In a first aspect, the present invention provides a method for automatically converting a cross-format document into a Markdown document, the method comprising the following steps:

[0007] Step S1: Acquire a document image;

[0008] Step S2: mining hidden structural information from the document image;

[0009] Wherein, the structural information is the hierarchical relationship between target elements and target elements; the target elements are titles, and / or paragraphs, and / or lists, and / or tables;

[0010] Step S3: Based on the structural information, a semantic content analysis is performed on the document image to obtain a semantic content analysis result;

[0011] Step S4: extracting key information of the document image based on the semantic content analysis result;

[0012] Step S5: converting the key information into Markdown format, and performing preliminary formatting processing in the form of a table or list to obtain a preliminary Markdown form;

[0013] Step S6: Optimize the preliminary Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of the cross-format document to a Markdown document.

[0014] Furthermore, the step S2 specifically includes the following steps:

[0015] Step S21: using a convolutional neural network to extract features from the acquired document image to obtain a feature map;

[0016] Step S22: inputting the feature map into a preset RPN to generate a candidate region;

[0017] Step S23: Use RoIPooling to map each candidate region to a feature representation of a fixed size to obtain a feature representation of the candidate region;

[0018] Step S24: inputting the feature representation of the candidate region into the fully connected layer for processing to obtain the coordinate positions of key elements; the key elements include titles, paragraphs, lists and tables;

[0019] Step S25: Based on the coordinate positions of the key elements, analyze the spatial layout and relative positional relationship of the key elements, and infer the hierarchical relationship between the key elements;

[0020] Step S26: Organizing the coordinate positions of the key elements and the hierarchical relationships between the key elements to generate a tree or graphic representation that conforms to the logical structure, thereby obtaining document image structure information.

[0021] Furthermore, the step S3 is specifically:

[0022] Using recursive neural networks and context encoding, the structural information of document images is analyzed and recognized, and paragraph types, form structures, and code block contents are extracted to obtain comprehensive semantic content analysis results.

[0023] The method uses recursive neural networks and context encoding to analyze and identify the structural information of document images, extract paragraph types, form structures, and code block contents, so as to obtain comprehensive semantic content analysis results, which specifically includes the following steps:

[0024] Step S31: Segment the document image structure information into paragraphs or sentences to form a text sequence;

[0025] Step S32: input the text sequence into the Bi-LSTM model for context modeling, and dynamically focus on key indicators through the attention mechanism;

[0026] Wherein, the key indicators are important elements or information in the document image;

[0027] Step S33: Based on the key indicators, use MLP to classify each paragraph and identify the paragraph type as plain text, form content or code block; and,

[0028] Use regular expressions to extract key fields of the form content and structure the form information to obtain the form structure; and,

[0029] Perform syntax analysis on the code block to obtain the semantics of the code block and obtain the content of the code block;

[0030] Step S34: Integrate the paragraph type, the form structure and the code block content to form a semantic content integration result;

[0031] Step S35: Optimizing and completing the semantic content integration result, so as to obtain a comprehensive semantic content analysis result.

[0032] Furthermore, the step S4 is specifically:

[0033] Based on the results of semantic content analysis, a graph-based sequence transformation model is used to extract key information of document images;

[0034] The method of extracting key information of document images based on the semantic content analysis results by using a graph-based sequence transformation model specifically includes the following steps:

[0035] Step S41: converting the semantic content analysis result into a form topology structure;

[0036] Wherein, the nodes in the form topology structure are each input field, and the edges in the form topology structure are the relationships between the fields;

[0037] Step S42: using a graph attention network to encode the form topology structure to obtain encoding information of the form topology structure;

[0038] Step S43: Based on the graph convolution and attention mechanism of the graph attention network, the interaction and dependency between the nodes in the table topology structure are learned according to the table topology structure encoding information, and a node representation vector of the context information is generated;

[0039] Step S44: using a conditional random field to perform sequence labeling on the input field according to the node representation vector, and identifying key information of the document image;

[0040] The key information includes input fields and layout in the document image.

[0041] Furthermore, the step S5 specifically includes the following steps:

[0042] Step S51: marking the key information to obtain a marked field sequence;

[0043] Step S52: input the annotated field sequence into the Transformer-based sequence conversion model, convert the field sequence into a structured Markdown format through the self-attention mechanism, and use the template to fill in the field according to the semantic category and form layout information to generate the corresponding Markdown tag;

[0044] Among them, for a table type form, the Markdown table syntax is used to generate a structured table representation; or, for a list type form, the Markdown list syntax is used to generate a structured list item;

[0045] Step S53: Generate a preliminary Markdown form according to the Markdown tags.

[0046] Furthermore, the step S6 specifically includes the following steps:

[0047] Step S61: comparing the layout structure differences between the document image and the preliminary Markdown form, and implementing form structure verification and dynamic display of the preliminary Markdown form through a script or plug-in tool to obtain a first optimized Markdown form; and

[0048] Generate a title and a subtitle of the preliminary Markdown form according to the structural information, and perform syntax formatting on the lists and code blocks in the preliminary Markdown form to obtain a second optimized Markdown form; and

[0049] Analyze the list items in the preliminary Markdown form, identify the indentation, numbering or symbol features of the list items, and convert the unordered list and the ordered list into the Markdown format accordingly to obtain the third optimized Markdown form;

[0050] Step S62: Integrate the first optimized Markdown form, the second optimized Markdown form, and the third optimized Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0051] Furthermore, in step S61, the layout structure differences between the document image and the preliminary Markdown form are compared, and the form structure verification and dynamic display of the preliminary Markdown form are implemented through a script or plug-in tool to obtain a first optimized Markdown form, which specifically includes the following steps:

[0052] Step A1: using an edit distance algorithm to compare the layout structure differences between the document image and the preliminary Markdown form to obtain a difference measurement value;

[0053] The edit distance algorithm specifically quantifies the difference in the number of insertion, deletion and replacement operations;

[0054] Step A2: Automatically optimize the layout structure of the preliminary Markdown form according to the difference measurement value to obtain a first process Markdown form; the automatic optimization includes automatically adjusting the alignment, indentation level and blank line separation;

[0055] Step A3: A supporting script tool or plug-in parses and optimizes the Markdown form of the first process, generates a display template, and uses a preset template engine for rendering and data binding to obtain a first optimized Markdown form;

[0056] Among them, the verification and error correction functions are integrated in the supporting script tool or plug-in, and the field type, value range and required items are automatically checked through predefined rules, and error prompts and correction suggestions are provided;

[0057] In the step S61, according to the structural information, the title and subtitle of the preliminary Markdown form are generated accordingly, and the list and code block in the preliminary Markdown form are formatted grammatically to obtain a second optimized Markdown form, which specifically includes the following steps:

[0058] Step B1: Based on the structural information, the document content is automatically annotated using a conditional random field sequence annotation model to obtain an annotation result;

[0059] The annotation results are used to identify different document elements and extract context and location information features;

[0060] Step B2, automatically generating a title and a subtitle in Markdown format according to the annotation results, and obtaining a preliminary Markdown form with a hierarchical relationship;

[0061] Step B3: performing multi-level list indentation and line break processing on the preliminary Markdown form with a hierarchical relationship, and formatting the code blocks in the preliminary Markdown form with a hierarchical relationship, thereby obtaining a second process optimized Markdown form;

[0062] Step B4: parsing and rendering the Markdown form optimized by the second process into an HTML page or PDF file corresponding to the visual structure of the document image, evaluating the accuracy and completeness of the Markdown text generation by comparison, and continuously optimizing the sequence annotation model and the Markdown generation rules, thereby forming a second optimized Markdown form;

[0063] In the step S61, the list items in the preliminary Markdown form are analyzed to identify the indentation, numbering or symbol features of the list items, and the unordered list and the ordered list are converted into the Markdown format accordingly to obtain a third optimized Markdown form, which specifically includes the following steps:

[0064] Step C1: Take the list items in the preliminary Markdown form as input, use the conditional random field (CRF) model to perform sequence labeling to identify indentation, numbering and symbol features, build feature templates and extract relevant attributes to obtain new list item text;

[0065] Among them, the conditional random field CRF model is trained using the gradient descent method and the L-BFGS algorithm;

[0066] Step C2: using the trained conditional random field (CRF) model to perform sequence labeling on the new list item text, identifying the structural features of each list item, determining the list type to which it belongs, and obtaining lists of different types;

[0067] Step C3: generating a third process optimized Markdown form according to the different types of lists and based on the logical consistency of indentation and numbering;

[0068] Step C4: According to the list item text in the document image, the list in the Markdown form optimized by the third process is evaluated and optimized to obtain a third optimized Markdown form.

[0069] In a second aspect, the present invention provides a form enhancement method based on automatic conversion of cross-format documents to Markdown, the method comprising the following steps:

[0070] Step K1: Obtain heterogeneous documents;

[0071] Step K2: splitting the heterogeneous document into a text part and a text image part;

[0072] Step K3: for the text portion, generating a first Markdown document corresponding to the text; and,

[0073] For the text image portion, the method for automatically converting a cross-format document to a Markdown document described in the first aspect is used to generate a second Markdown document corresponding to the text image;

[0074] The structural and semantic coherence between all the first Markdown documents and the second Markdown documents is analyzed, and the structural information between all the first Markdown documents and the second Markdown documents is merged into one Markdown document to complete the form enhancement based on automatic conversion of cross-format documents to Markdown.

[0075] In a third aspect, the present invention provides a device for automatically converting a cross-format document to a Markdown document, the device comprising:

[0076] An acquisition unit, used for acquiring a document image;

[0077] A mining unit, connected to the acquisition unit, for mining hidden structural information from the document image;

[0078] Wherein, the structural information is the hierarchical relationship between target elements and target elements; the target elements are titles, and / or paragraphs, and / or lists, and / or tables;

[0079] An analysis unit, connected to the mining unit, configured to perform semantic content analysis on the document image based on the structural information to obtain a semantic content analysis result;

[0080] An extraction unit, connected to the analysis unit, for extracting key information of the document image based on the semantic content analysis result;

[0081] A processing unit, connected to the extraction unit, for converting the key information into a Markdown format and performing preliminary formatting processing in a table or list to obtain a preliminary Markdown form;

[0082] The optimization unit is connected to the processing unit and is used to optimize the preliminary Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0083] In a fourth aspect, the present invention provides a form enhancement system based on automatic conversion of cross-format documents to Markdown, the system comprising:

[0084] An acquisition device, used for acquiring heterogeneous documents;

[0085] A splitting device, connected to the acquisition device, for splitting the heterogeneous document into a text portion and a text image portion;

[0086] A generating device, connected to the splitting device, for generating a first Markdown document corresponding to the text portion from the text portion;

[0087] The device for automatically converting a cross-format document to a Markdown document according to the third aspect is connected to the splitting device and is used to generate a second Markdown document corresponding to the image from the text image part;

[0088] A merging device is respectively connected to the generating device and the device for automatically converting cross-format documents to Markdown documents, and is used to merge all of the first Markdown documents and all of the second Markdown documents into a complete Markdown document based on the structural information and semantic consistency between the first Markdown documents and the second Markdown documents, so as to complete the form enhancement based on the automatic conversion of cross-format documents to Markdown.

[0089] The present invention can effectively identify and convert forms with complex nested structures and irregular layouts in documents, thereby ensuring the integrity and readability of form information when converting documents to Markdown format. The specific beneficial effects are as follows:

[0090] 1. Effectively identify complex structures: The present invention can identify and convert complex nested structures and irregularly laid out forms in documents, ensuring the integrity and readability of information while ensuring that key data is not lost, thereby improving the accuracy of the overall conversion.

[0091] 2. Enhanced structure recognition capability: The present invention can accurately identify the structure in a document by analyzing the specific locations and hierarchical relationships of titles, paragraphs, lists, and table elements, thereby ensuring that the actual layout of the document can be effectively captured during the Markdown conversion process.

[0092] 3. Semantic-based key information extraction: The present invention uses a graph-based sequence conversion model to extract the input fields and layout information of the form at the semantic level to ensure the integrity of the key information, and converts the extracted information into Markdown format for preliminary formatting, thereby making the converted document easier to read and understand.

[0093] 4. Optimized layout processing: After conversion to Markdown format, the present invention optimizes the form layout by comparing the structural differences between the original form and the converted form. This optimization not only improves the aesthetics of the document layout, but also improves the readability of the information.

[0094] 5. Dynamic verification and display: During the conversion process, the present invention uses scripts or plug-in tools to implement verification and dynamic display of the form structure, which can ensure the accuracy and consistency of the conversion results. This feature allows users to view the conversion effect in real time, improving the interactivity of the experience.

[0095] 6. Accurate syntax formatting: When generating Markdown-formatted titles, subtitles, lists, and code blocks, the present invention can perform accurate syntax formatting according to the visual structure of the original document. This allows the converted Markdown document to not only retain the logical fluency of the original document, but also achieve an aesthetically pleasing and easy-to-read effect.

[0096] 7. Multi-document merging capability: This invention can seamlessly merge the contents of documents in different formats into a coherent Markdown document by analyzing the structural and semantic coherence between multiple documents. This process not only maintains the logical order of information, but also ensures the consistency of the format, greatly improving work efficiency. At the same time, it ensures that the format of the merged output of multiple documents is unified, maintaining the professionalism and standardization of the documents.

[0097] 8. Improve user experience: The present invention provides users with a convenient and efficient information processing experience, especially when processing complex documents and merging multiple documents, which significantly reduces the time cost of adjustment and improves work efficiency and satisfaction; in addition, the converted documents are easier to understand, which improves the user's reading experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Figure 1 This is a schematic diagram of the process of automatically converting a cross-format document to a Markdown document in an embodiment of the present invention;

[0099] Figure 2 A schematic diagram of a method for automatically converting a cross-format document to a Markdown document in an embodiment of the present invention;

[0100] Figure 3 It is a flow chart of a form enhancement method based on automatic conversion of cross-format documents to Markdown in an embodiment of the present invention;

[0101] Figure 4 A Markdown formatted flowchart in an embodiment of the present invention;

[0102] Figure 5 Schematic diagram of a device for automatically converting a cross-format document to a Markdown document in an embodiment of the present invention;

[0103] Among them, the figure numbers are: 10, acquisition unit, 20, mining unit, 30, analysis unit, 40, extraction unit, 50, processing unit, 60, optimization unit. DETAILED DESCRIPTION

[0104] In order to enable those skilled in the art to better understand the technical solution of the present invention, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0105] It should be understood that the specific embodiments and drawings described herein are only used to explain the present invention rather than to limit the present invention.

[0106] It can be understood that, in the absence of conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other.

[0107] It can be understood that, for the convenience of description, the drawings of the present invention only show the parts related to the present invention, while the parts irrelevant to the present invention are not shown in the drawings.

[0108] It can be understood that each unit and module involved in the embodiments of the present invention may correspond to only one physical structure, or may be composed of multiple physical structures, or multiple units and modules may be integrated into one physical structure.

[0109] It can be understood that, without conflict, the functions and steps marked in the flowcharts and block diagrams of the present invention may occur in an order different from that marked in the drawings.

[0110] It is understood that the flowcharts and block diagrams of the present invention illustrate the possible architectures, functions, and operations of the systems, devices, equipment, and methods according to the various embodiments of the present invention. Each box in the flowchart or block diagram may represent a unit, module, program segment, or code, which contains executable instructions for implementing the specified functions. Moreover, each box or combination of boxes in the block diagram and flowchart may be implemented by a hardware-based system that implements the specified functions, or may be implemented by a combination of hardware and computer instructions.

[0111] It can be understood that the units and modules involved in the embodiments of the present invention can be implemented by software or hardware. For example, the units and modules can be located in a processor.

[0112] Embodiment 1:

[0113] like Figure 1 and Figure 2 As shown, this embodiment provides a method for automatically converting a cross-format document to a Markdown document, the method comprising the following steps:

[0114] Step S1: Acquire a document image.

[0115] Step S2: mining hidden structural information from the document image. The structural information is the hierarchical relationship between target elements; the target elements are titles, and / or paragraphs, and / or lists, and / or tables.

[0116] Specifically, step S2 includes the following steps:

[0117] Step S21: Use a convolutional neural network to extract features from the acquired document image to obtain a feature map.

[0118] In step S21, the convolutional neural network uses an existing convolutional neural network, which may be AlexNet, VGGNet, GoogLeNet (Inception), ResNet, EfficientNet, etc., which have been widely used and verified. These networks perform well in image processing and feature extraction tasks and can effectively capture important features in document images.

[0119] Step S22: Input the feature map into a preset RPN to generate candidate regions. RPN (Region Proposal Network) is a deep learning model for target detection.

[0120] Step S23: Use RoIPooling to map each candidate region to a fixed-size feature representation to obtain a feature representation of the candidate region. RoIPooling (Region of Interest Pooling) is a technology used in computer vision, the main purpose of which is to extract information about a specific region from a feature map so that it corresponds to a specific target in the input image. In this way, it can effectively process and analyze the target region, thereby improving the accuracy of target detection and recognition.

[0121] Step S24: Input the feature representation of the candidate area into the fully connected layer for processing to obtain the coordinate positions of key elements; the key elements include titles, paragraphs, lists and tables.

[0122] Step S25: Based on the coordinate positions of the key elements, the spatial layout and relative positional relationship of the key elements are analyzed to infer the hierarchical relationship between the key elements.

[0123] Step S26: Organizing the coordinate positions of the key elements and the hierarchical relationships between the key elements to generate a tree or graphic representation that conforms to the logical structure, thereby obtaining document image structure information.

[0124] Step S3: Based on the structural information, a semantic content analysis is performed on the document image to obtain a semantic content analysis result.

[0125] The step S3 is specifically as follows:

[0126] Using recursive neural networks and context encoding, the structural information of document images is analyzed and recognized, and paragraph types, form structures, and code block contents are extracted to obtain comprehensive semantic content analysis results.

[0127] The method uses recursive neural networks and context encoding to analyze and identify the structural information of document images, extract paragraph types, form structures, and code block contents, so as to obtain comprehensive semantic content analysis results, which specifically includes the following steps:

[0128] Step S31: Segment the document image structure information into paragraphs or sentences to form a text sequence;

[0129] Step S32: Input the text sequence into the Bi-LSTM model for context modeling, and dynamically focus on key indicators through the attention mechanism; Bi-LSTM (Bidirectional Long Short-Term Memory) is a model used in sequence data processing, especially natural language processing (NLP). It is based on the LSTM (Long Short-Term Memory Network) architecture and can effectively capture context information in sequence data. The key indicators are important elements or information in the document image;

[0130] Step S33: Based on the key indicators, use MLP to classify each paragraph and identify the paragraph type as plain text, form content or code block; and,

[0131] Use regular expressions to extract key fields of the form content and structure the form information to obtain the form structure; and,

[0132] Perform syntax analysis on the code block to obtain the semantics of the code block and obtain the content of the code block;

[0133] Step S34: Integrate the paragraph type, the form structure and the code block content to form a semantic content integration result;

[0134] Step S35: Optimize and complete the semantic content integration result, so as to obtain a comprehensive semantic content analysis result. In the specific implementation, domain knowledge and external resources will be combined.

[0135] Step S4: extracting key information of the document image based on the semantic content analysis result.

[0136] The step S4 is specifically:

[0137] Based on the results of semantic content analysis, a graph-based sequence transformation model is used to extract key information of document images;

[0138] The method of extracting key information of document images based on the semantic content analysis results by using a graph-based sequence transformation model specifically includes the following steps:

[0139] Step S41: converting the semantic content analysis result into a form topology structure;

[0140] Wherein, the nodes in the form topology structure are each input field, and the edges in the form topology structure are the relationships between the fields;

[0141] Step S42: using a graph attention network to encode the form topology structure to obtain encoding information of the form topology structure;

[0142] Step S43: Based on the graph convolution and attention mechanism of the graph attention network, the interaction and dependency between the nodes in the table topology structure are learned according to the table topology structure encoding information, and a node representation vector of the context information is generated;

[0143] Step S44: using a conditional random field to perform sequence labeling on the input field according to the node representation vector, and identifying key information of the document image;

[0144] The key information includes input fields and layout in the document image.

[0145] Step S5: convert the key information into Markdown format, and perform preliminary formatting processing in the form of a table or list to obtain a preliminary Markdown form.

[0146] The step S5 specifically includes the following steps:

[0147] Step S51: marking the key information to obtain a marked field sequence;

[0148] Step S52: input the annotated field sequence into the Transformer-based sequence conversion model, convert the field sequence into a structured Markdown format through the self-attention mechanism, and use the template to fill in the field according to the semantic category and form layout information to generate the corresponding Markdown tag;

[0149] Among them, for a table type form, the Markdown table syntax is used to generate a structured table representation; or, for a list type form, the Markdown list syntax is used to generate a structured list item;

[0150] Step S53: Generate a preliminary Markdown form according to the Markdown tags.

[0151] Step S6: Optimize the preliminary Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of the cross-format document to a Markdown document.

[0152] The step S6 specifically includes the following steps:

[0153] Step S61: comparing the layout structure differences between the document image and the preliminary Markdown form, and implementing form structure verification and dynamic display of the preliminary Markdown form through a script or plug-in tool to obtain a first optimized Markdown form; and

[0154] Generate a title and a subtitle of the preliminary Markdown form according to the structural information, and perform syntax formatting on the lists and code blocks in the preliminary Markdown form to obtain a second optimized Markdown form; and

[0155] The list items in the preliminary Markdown form are analyzed to identify the indentation, numbering or symbol features of the list items, and the unordered list and the ordered list are converted into the Markdown format accordingly to obtain the third optimized Markdown form.

[0156] Specifically, after comparing the layout structure differences between the document image and the preliminary Markdown form, JavaScript and its libraries (such as jQuery or Vanilla JavaScript), front-end frameworks (such as React or Vue.js), and form validation libraries (such as Formik or Yup) can be used to implement form structure verification and dynamic display. Specific optimization directions include ensuring layout consistency, implementing responsive design, performing real-time data validation, and dynamically displaying or hiding form fields based on user input, thereby improving user experience and reducing erroneous submissions. Through these tools and optimization directions, the first version of the optimized Markdown form can be obtained.

[0157] According to the structural information, clear titles and subtitles can be generated for the preliminary Markdown form, and the lists and code blocks can be grammatically formatted to obtain the second optimized Markdown form. This includes setting the main title as "Step Data Form" with subtitles such as "Basic Information", "Step Record" and "Code Example" to better organize and display information. At the same time, the steps and code examples are presented in the form of embedded and ordered lists to make them more readable and logical. The specific optimization direction is to improve the degree of structuring and readability of the form. By reasonably using titles, subtitles and formatted text, the content is more intuitive and organized, which is convenient for users to understand and fill in, thereby improving the overall user experience.

[0158] Step S62: Integrate the first optimized Markdown form, the second optimized Markdown form, and the third optimized Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

[0159] This embodiment first extracts the titles, subtitles, and key content from all versions to ensure that the information is complete and non-repetitive; then, unifies the format to improve readability through consistent syntax and structure (such as lists and code blocks); then, integrates the dynamic interaction functions and form validation logic from each version to enhance the user experience; finally, performs language polishing to make the form content concise and clear. The resulting target Markdown form will have a clear structure, consistent format, complete information, good readability, and dynamic interaction features, which can effectively meet the needs of user filling and data collection.

[0160] As a specific implementation, in step S61, the layout structure differences between the document image and the preliminary Markdown form are compared, and the form structure verification and dynamic display of the preliminary Markdown form are implemented through a script or plug-in tool to obtain a first optimized Markdown form, which specifically includes the following steps:

[0161] Step A1: using an edit distance algorithm to compare the layout structure differences between the document image and the preliminary Markdown form to obtain a difference measurement value;

[0162] The edit distance algorithm specifically quantifies the difference in the number of insertion, deletion and replacement operations;

[0163] Step A2: Automatically optimize the layout structure of the preliminary Markdown form according to the difference measurement value to obtain a first process Markdown form; the automatic optimization includes automatically adjusting the alignment, indentation level and blank line separation;

[0164] Step A3: A supporting script tool or plug-in parses and optimizes the Markdown form of the first process, generates a display template, and uses a preset template engine for rendering and data binding to obtain a first optimized Markdown form;

[0165] Among them, the verification and error correction functions are integrated in the supporting script tool or plug-in, and the field type, value range and required items are automatically checked through predefined rules, and error prompts and correction suggestions are provided;

[0166] In the step S61, according to the structural information, the title and subtitle of the preliminary Markdown form are generated accordingly, and the list and code block in the preliminary Markdown form are formatted grammatically to obtain a second optimized Markdown form, which specifically includes the following steps:

[0167] Step B1: Based on the structural information, the document content is automatically annotated using a conditional random field sequence annotation model to obtain an annotation result;

[0168] The annotation results are used to identify different document elements and extract context and location information features;

[0169] Step B2, automatically generating a title and a subtitle in Markdown format according to the annotation results, and obtaining a preliminary Markdown form with a hierarchical relationship;

[0170] Step B3: performing multi-level list indentation and line break processing on the preliminary Markdown form with a hierarchical relationship, and formatting the code blocks in the preliminary Markdown form with a hierarchical relationship, thereby obtaining a second process optimized Markdown form;

[0171] Step B4: parsing and rendering the Markdown form optimized by the second process into an HTML page or PDF file corresponding to the visual structure of the document image, evaluating the accuracy and completeness of the Markdown text generation by comparison, and continuously optimizing the sequence annotation model and the Markdown generation rules, thereby forming a second optimized Markdown form;

[0172] In the step S61, the list items in the preliminary Markdown form are analyzed to identify the indentation, numbering or symbol features of the list items, and the unordered list and the ordered list are converted into the Markdown format accordingly to obtain a third optimized Markdown form, which specifically includes the following steps:

[0173] Step C1: Take the list items in the preliminary Markdown form as input, use the conditional random field (CRF) model to perform sequence labeling to identify indentation, numbering and symbol features, build feature templates and extract relevant attributes to obtain new list item text;

[0174] Among them, the conditional random field CRF model is trained using the gradient descent method and the L-BFGS algorithm;

[0175] Step C2: using the trained conditional random field (CRF) model to perform sequence labeling on the new list item text, identifying the structural features of each list item, determining the list type to which it belongs, and obtaining lists of different types;

[0176] Step C3: generating a third process optimized Markdown form according to the different types of lists and based on the logical consistency of indentation and numbering;

[0177] Step C4: According to the list item text in the document image, the list in the Markdown form optimized by the third process is evaluated and optimized to obtain a third optimized Markdown form.

[0178] The specific implementation process of this embodiment is as follows:

[0179] Step 1. Preprocessing step: document format determination and content type distinction

[0180] (1) Identify the input document format by file extension, such as .pdf, .docx, .pptx, etc. Select the appropriate parsing tool for processing based on the recognition result: for PDF documents, use PDF parsing tools such as PyMuPDF, pdfplumber, etc.; for Word documents, use the python-docx tool; for PPT documents, use the python-pptx tool.

[0181] (2) For documents containing text content (such as Word and PDF), extract the text part and save it as a string; for documents containing images (such as PDF and PPT), extract the embedded images and save them as separate files or objects; in Word documents, extract the table and list structure; in PDF documents, use a table recognition tool (such as PDFPlumber) to extract the table structure.

[0182] Step 2. Document image feature extraction

[0183] (1) Feature extraction. A convolutional neural network is used to extract features from document images. Through convolution kernels of different sizes and multiple convolution layers, the positions of titles, paragraphs, lists, and table elements are automatically obtained. For example, the first convolution layer can use 64 3x3 convolution kernels with a step size of 1 and a padding of 1; the second convolution layer can use 128 3x3 convolution kernels with a step size of 1 and a padding of 1; the third convolution layer can use 256 3x3 convolution kernels with a step size of 1 and a padding of 1. The pooling layer can use maximum pooling with a pooling kernel size of 2x2 and a step size of 2.

[0184] (2) Generate candidate regions. Use a convolutional neural network to extract feature maps, then generate candidate regions through a region proposal network (such as Faster R-CNN), and use region of interest pooling (RoI Pooling) to obtain a fixed-size feature representation. The region proposal network generates multiple candidate regions of different scales and aspect ratios at each location by sliding a small network on the convolutional feature map, and uses bounding box regression and classification to optimize the location and category of the candidate region. For example, three anchor boxes of different scales (such as 128x128, 256x256, 512x512) and three different aspect ratios (such as 1:1, 1:2, 2:1) can be used on the convolutional feature map to generate a total of 9 different candidate regions.

[0185] (3) Classification and regression. The feature representation of the candidate region is input into the fully connected layer for classification and position regression. The fully connected layer uses the ReLU activation function, uses the cross entropy loss function for multi-category classification, and uses the SmoothL1 loss function for position regression to determine whether each candidate region contains the target element and obtain its precise position coordinates. In the fully connected layer, two hidden layers can be used, each hidden layer contains 1024 neurons, and uses the ReLU activation function. For classification tasks, the softmax function can be used to output the probability of each category; for position regression tasks, a linear function can be used to output the coordinate adjustment value of the bounding box.

[0186] (4) Inference of hierarchical relationships. Based on the acquired position coordinates of the title, paragraph, list, and table elements, by analyzing the spatial layout and relative positional relationship between the elements, such as the distance, overlap, alignment, and other geometric features between the elements, their hierarchical relationship in the document structure is inferred and determined, and a hierarchical structure tree of the document is constructed. A graph-based method can be used to first represent each element in the document as a node in the graph, and then add directed edges between the nodes based on the spatial layout relationship between the elements (such as distance, overlap, alignment, etc.) to construct a directed acyclic graph (DAG). By topologically sorting the DAG, the hierarchical relationship between the elements can be obtained. If the distance between element A and element B is less than 10 pixels, and A is above B, then add a directed edge from A to B.

[0187] (5) Organization and encoding. The identified document structure information is organized and encoded. By recursively traversing the document's hierarchical structure tree, the type, location, content, and other information of each element is encoded into a data structure in a specific format, such as JSON or XML, to generate a tree representation or graphical representation that conforms to the document's logical structure for subsequent semantic understanding and information extraction tasks.

[0188] (6) Post-processing and optimization. The extracted document structure information is post-processed and optimized. Combined with prior knowledge, such as the document type, format, and field, as well as heuristic rules, such as the font size and position of the title, the indentation and spacing of the paragraph, the recognition results are verified and corrected to correct possible errors and omissions, thereby improving the accuracy and completeness of document structure recognition. For example, if the recognized title is at the bottom of the document, or the width of the paragraph exceeds 80% of the document width, it can be determined as a recognition error and corrected accordingly.

[0189] Step 3: Semantic content analysis of structural information

[0190] (1) Structural information input and encoding. Based on the obtained document structure information, it is input into the recurrent neural network, and the text sequence is encoded through the bidirectional long short-term memory network (Bi-LSTM) to capture the contextual information and long-distance dependencies in the text, and generate a semantic representation vector with context-awareness. For example, when using Bi-LSTM to encode the text sequence, the dimension of the hidden layer can be set to 256, the Adam optimizer is used, the learning rate is 0.001, the batch size is 32, and the training is 50 epochs.

[0191] (2) Introducing the attention mechanism. In the encoding process, an attention mechanism, such as LuongAttention, is introduced. The attention weight is dynamically generated by calculating the similarity between the query vector and the encoding vector, and the encoding vector is weighted and summed. The key information related to the current task is dynamically focused on to improve the accuracy of semantic analysis. LuongAttention calculates the similarity between the current decoder state and the encoder state at each time step, uses the softmax function to normalize the similarity into the attention weight, and then performs a weighted summation of the encoder state to obtain the attention vector.

[0192] (3) Paragraph attribute classification. Based on the semantic representation vector of the text, a multi-layer perceptron (MLP) is used to classify different paragraph attributes to determine whether each paragraph belongs to different types such as plain text, form content, or code block. The multi-layer perceptron can contain two hidden layers, each with 256 neurons, and use the ReLU activation function. The probability distribution of each category is calculated through nonlinear transformation and softmax function to obtain the classification label of the paragraph.

[0193] (4) Fusion of layout and format features. When classifying paragraph attributes, the layout features and format features of the paragraph, such as paragraph indentation, line spacing, font, etc., are integrated to further improve the accuracy of classification. The layout features and format features can be obtained by analyzing the style information and visual features of the document.

[0194] (5) Form content identification. For form content paragraphs, a rule-based approach is used for further identification and extraction. Define regular expressions and matching rules, identify key fields and entry items, and extract structured information. For example, use the regular expression name: \s*(\S+) to match the name field, \d{4}-\d{2}-\d{2} to match the date field, and organize the matching results into JSON, CSV, or database record formats.

[0195] (6) Code block identification. By analyzing the syntax features and keywords of the code block, such as "def" and "class" in Python, "public" and "static" in Java, etc., the programming language and functional module of the code are identified and the semantic information of the code is extracted. For example, the start and end positions of the code block can be determined by the frequency and position of the keywords.

[0196] (7) Semantic structure fusion. The results of paragraph attribute classification are integrated with the recognition results of forms and code blocks to form a complete document semantic structure representation, including the types of each paragraph, the structured information of the form, the semantic information of the code block, etc.

[0197] (8) Optimization and completion. In the process of generating semantic structure representation, domain knowledge bases, external dictionaries, ontologies and other resources are combined to verify and correct the recognition results, and missing semantic information is inferred and filled. For example, external knowledge bases such as WordNet and WikiData are used to infer missing semantic information through synonymy, hyponymy and other relationships of words. If the "country" field is missing in the recognized form, but the "city" field exists, the corresponding country information can be inferred based on the subordinate relationship between the city and the country.

[0198] Step 4: Extract key information and format in Markdown

[0199] (1) Form graph structure construction. Figure 4 As shown in the figure, the recognized form content is first converted into a graph structure, where each input field of the form is represented as a node in the graph, and the relationship between the fields is represented as an edge, so as to construct the topological structure of the form. For example, for a form containing 10 fields, a 10x10 adjacency matrix can be constructed, and the elements of the matrix represent the relationship between the fields, 1 indicates the existence of a relationship, and 0 indicates the absence of a relationship.

[0200] (2) Graph Attention Network Encoding. A graph attention network is used to encode the form graph. Through graph convolution and attention mechanism, the interactions and dependencies between nodes are learned to generate node representation vectors with contextual information. The graph attention network contains 3 layers of graph convolution layers, each with 8 attention heads and a ReLU activation function. The output dimension is 64, L2 regularization and Dropout are used to prevent overfitting, Adam is used as the optimizer, and the learning rate is set to 0.001.

[0201] (3) Conditional random field sequence labeling. Based on the node representation vector, the conditional random field is used to perform sequence labeling on the input fields to identify the semantic category of each field, such as name, date, address, etc. The conditional random field uses unigram features and bigram features, and is decoded through the Viterbi algorithm to obtain the optimal labeling sequence. Feature selection can include linguistic features such as the part of speech, named entity type, and dependency relationship of the field, as well as layout features such as the position and length of the field in the form.

[0202] (4) Transformer-based sequence conversion. The annotated field sequence is input into the Transformer-based sequence conversion model, and the field sequence is converted into a structured Markdown format through the self-attention mechanism and feedforward neural network. The Transformer model contains 6 encoder layers and 6 decoder layers. The number of attention heads in each layer is 8. The dimension of the feedforward network is 2048. The BeamSearch algorithm is used for decoding, and the search width is 5.

[0203] (5) Markdown format generation. In the process of generating the Markdown format, the corresponding Markdown tags and formats are generated by using template filling according to the semantic category of the field and the layout information of the form. For example, some common templates are predefined, such as "##Name: {name}", "-Date: {date}", etc., and then dynamically filled according to the annotation results of the field. For table-type forms, the table syntax of Markdown is used, and the field name is used as the table header and the field value is used as the table unit to generate a structured table representation; for list-type forms, the list syntax of Markdown is used, and the field name is used as the list item and the field value is used as the list content to generate a structured list representation.

[0204] (6) Form type determination and processing. The form type is determined by analyzing the keywords and layout features in the form using heuristic rules. For example, table-type forms usually contain continuous vertical bars "|", while list-type forms usually contain continuous "-" or "*". For mixed-type forms, a block processing strategy is adopted to extract the table part and the list part separately, generate Markdown format for each, and then merge the generated Markdown content.

[0205] (7) Field value formatting. In the process of generating the Markdown format, the field value is formatted as necessary, such as date format conversion, number formatting, text length limit, etc., to ensure the readability and consistency of the generated Markdown content. For example, the template for the date field is "-Date: {date: YYYY-MM-DD}", and the template for the number field is "-Amount: {amount:,##0.00} yuan", etc.

[0206] Step 5: Markdown form layout optimization

[0207] (1) Structural difference comparison. The generated Markdown format form is compared with the original form, and the Levenshtein distance algorithm is used to calculate the structural difference between the two. The Levenshtein distance algorithm measures the difference between two strings by calculating the minimum number of editing operations (insertion, deletion, replacement) required to convert one string into another. For example, for a form containing 10 fields, it can be converted into a string of length 10, where each character represents the type of a field (such as "T" for text field and "N" for numeric field), and then the dynamic programming algorithm is used to calculate the Levenshtein distance between the two strings to obtain the difference metric.

[0208] (2) Layout structure optimization. Based on the difference metric, the layout structure of the Markdown format form is automatically optimized and adjusted to keep it as consistent as possible with the layout of the original form. For example, for a table with 5 rows and 3 columns, the optimal width of each column can be calculated based on the length of the cell content, and then the cell width of each row can be adjusted in turn to make it as close to the optimal width as possible until all rows meet the requirements. During the optimization process, a series of layout optimization rules are defined, such as table column width adaptation, list item indent alignment, paragraph first line indent, etc., and a greedy algorithm or dynamic programming algorithm is used to structurally adjust and beautify the Markdown format form.

[0209] (3) Use Python to develop a supporting script tool, take the optimized Markdown format form as input, parse the structured information in it through regular expressions, and generate an HTML display template corresponding to the original form. For example, when performing form structure verification and error correction, you can use Python's built-in modules such as re (regular expression) and datetime (date time) to check the type, format, value range, etc. of each field in the form. For a date field, you can use the regular expression "^\d{4}-\d{2}-\d{2}$" to check the format, and use the datetime module to parse the date string to determine whether it is legal.

[0210] (4) Template rendering and dynamic display. Through the rendering of the Jinja2 template engine and the data binding of Vue.js, the visual display and interactive operations of the form content are realized, such as table sorting, filtering, paging and other functions. For example, for a form containing user information, you can define a table in the HTML template, associate the rows and columns of the table with the form data through Jinja2 syntax, and then use the v-for instruction of Vue.js to render the table to realize the dynamic display and update of user information.

[0211] (5) Form structure verification and error correction. In the script tool, the form structure verification and error correction function is integrated to automatically check and verify the form content through predefined rules and constraints, such as field type (text, number, date, etc.), value range (enumeration value or regular expression), required items, etc., and give corresponding error prompts and correction suggestions. For example, for a numeric field, you can use the regular expression "^\d+(.\d+)?$" to check the format, use the int or float function to convert the string into a number, and determine whether it is within the specified value range.

[0212] (6) User feedback and optimization. In actual applications, user feedback is collected through methods such as user satisfaction surveys and A / B testing. The form structure optimization algorithm and strategy are optimized based on the feedback results to provide more intelligent and humanized services, reduce user manual operations and repetitive work, and improve the efficiency and quality of form processing.

[0213] Step 6: Generate title and subtitle in Markdown format

[0214] (1) Sequence labeling and document element identification. The mined document structure and hierarchical information are used as input, and sequence labeling models such as conditional random fields (CRF) are used to automatically label and divide the document content, and identify different document elements such as titles, subtitles, paragraphs, lists, and code blocks. For example, when using the CRF model for sequence labeling, you can select linguistic features such as part of speech, named entities, and dependencies to build feature templates. You can use the unigram and bigram features of part of speech tags, the unigram features of named entity tags, the direction and distance features of dependencies, and the relative position and length range of text blocks. Structural features. About 1,000 documents are manually annotated as training data, and the model performance is evaluated using 10-fold cross validation and F1 value.

[0215] (2) Generate Markdown format. Automatically generate titles and subtitles in Markdown format based on the results of sequence annotation. For example, for titles, you can add a corresponding number of "#" symbols based on their hierarchical information, such as adding one "#" to a first-level title, two "#" to a second-level title, and so on. Up to six levels of titles are supported. For lists, by analyzing the indentation, numbering, bullet points and other information of the list items, the type and hierarchy of the list are determined, and the corresponding Markdown syntax format is generated. If a list item starts with a symbol such as "-", "+", "*", an unordered list is generated; if it starts with a number or ".", an ordered list is generated. For code blocks, by analyzing the indentation and comment information of the code, the programming language of the code and the starting position of the code block are determined, and the corresponding Markdown syntax format is generated.

[0216] (3) Text content formatting. In the process of generating the Markdown format, the text content is escaped and formatted as necessary. For example, special characters are escaped, links and images are syntax processed, and emphasis and bold are marked to ensure the correctness and readability of the generated Markdown text.

[0217] (4) Markdown parsing and HTML rendering. Use existing Markdown parsers and rendering engines such as Python-Markdown and Pandoc to parse the generated Markdown text into HTML format, and then use CSS styles and JavaScript scripts to layout and render the HTML page. For example, when parsing Markdown and rendering HTML, use the Python-Markdown library for parsing, and then use the CSS framework Bootstrap to layout the HTML page. Through media queries and responsive design, the best display effect on different devices and screen sizes can be achieved.

[0218] (5) Automated testing and evaluation. Parse and evaluate the generated Markdown text and HTML pages. Evaluate the accuracy, completeness, and readability of the generated Markdown text, as well as the correctness and consistency of the HTML page rendering, through manual review and automated testing. For example, use automated testing tools such as Selenium to perform functional and UI testing on the generated HTML pages, such as link clickability, image loading speed, page rendering time, etc., and generate test reports.

[0219] (6) Visual structure optimization. Consider using computer vision and image processing technologies, such as OpenCV, to analyze and extract features from the original document image, identify different layout elements such as text, pictures, and tables, and calculate their position, size, alignment, and other properties. For example, use the OpenCV library to analyze the original document image, perform edge detection, contour detection, and rectangle fitting using the Canny operator to extract layout elements such as text, pictures, and tables, calculate their coordinates, width, height, alignment, and other properties, and then map these properties to corresponding CSS styles, such as margin and padding for text blocks, width and height for pictures, border-collapse for tables, width for td, etc., to achieve the greatest possible restoration of the visual effects of the original document.

[0220] Step 7: List Item Analysis and Conversion

[0221] (1) Sequence labeling and structural feature recognition. The generated list item text is used as input, and the conditional random field (CRF) model is used to perform sequence labeling and analysis on it to identify the structural features and format attributes of the list item. Define the feature template of the CRF model, extract features such as the indentation level, number or symbol type, text length, context information, etc. of the list item, and construct a feature function. For example, when defining the feature template of the CRF model, you can use the indentation level of the list item (integer value), the symbol type of the list item (such as "-", "+", "*", number + ".", etc.), the text length of the list item (integer value), the context information of the list item (such as the symbol type and indentation level of the previous and next items), etc.

[0222] (2) Training and evaluating the CRF model. By manually annotating about 1,000 list item samples to form a training data set, the L-BFGS algorithm is used to train the CRF model, and appropriate hyperparameters such as the number of iterations and gradient threshold are set to learn the weight parameters of the feature function. For example, by manually annotating 1,000 list item samples, 800 of them are used as training sets and 200 as test sets. The CRF model is trained using the L-BFGS algorithm, with the number of iterations set to 100 and the gradient threshold set to 0.001. The performance of the model is evaluated on the test set, with an accuracy of 0.95, a recall of 0.93, and an F1 value of 0.94.

[0223] (3) Handling unregistered and rare words. When extracting features, use dictionary-based features or integrate external knowledge bases to enhance feature representation. For example, unregistered and rare words can be classified as "UNK" and dictionary-based features can be used, such as f(x,i) = 1 if x is in dictionary D else 0, where D can be an external knowledge base or domain dictionary.

[0224] (4) Sequence labeling and Markdown format generation. Use the trained CRF model to perform sequence labeling on the new list item text and predict the structural features and format attributes of each list item. According to the labeling results, determine the list type to which the list item belongs. For example, a list that starts with "-", "+", "" and has an indentation level of 0 is an unordered list, a list that starts with a number and "." and has an indentation level of 0 is an ordered list, and a list that has an indentation level greater than 0 is a multi-level list. Specifically, for list items with an indentation level of 0 and starting with "-", "+", "", generate "-", "+", "*"; for list items with an indentation level of 0 and starting with a number + ".", generate "1.", "2.", etc.; for list items with an indentation level of n (n>0), generate n spaces plus the corresponding list symbol.

[0225] (5) Comparison and evaluation. Compare the generated Markdown format list with the original list item text to evaluate the accuracy and completeness of the conversion, focus on the list items with conversion errors, analyze the reasons, and adjust the CRF model and conversion rules accordingly. For example, using random sampling to manually check 100 converted lists, 5 were found to have format errors, with an error rate of 5%. By analyzing the causes of the errors, it was found that they were mainly caused by special characters or irregular indentations in the list items.

[0226] (6) Optimization and user feedback. By analyzing the causes of errors, we further improve the feature templates and conversion rules to improve the robustness of the conversion. At the same time, we collect user feedback, such as conversion speed, format aesthetics, etc., and continuously iterate and optimize to improve user experience.

[0227] Step 8: Merge multiple document structures

[0228] (1) Input format conversion. Convert the generated multiple Markdown documents into the input format of the sequence-to-sequence model, extract the structural information of each document, and connect them with special delimiters to form a long sequence. For example, you can use special delimiters such as " <doc> "and"< / doc> " to identify the start and end of each document, use " <sec> "and"< / sec> " to mark the start and end of each chapter, use " "and" " to identify the start and end positions of each paragraph, thereby encoding the structural information of the document into the input sequence.

[0229] (2) Encoding and attention mechanism. The encoder-decoder architecture is used to encode the input sequence through a bidirectional recurrent neural network (Bi-RNN) to generate the structure and semantic representation vector of each document. During the encoding process, an attention mechanism such as LuongAttention or BahdanauAttention is used to calculate the attention weight based on the encoding vector of the current document and the encoding vectors of other documents to obtain a weighted average context vector. For example, a two-layer LSTM with 256 hidden units can be used as the basic unit, and LuongAttention is used to calculate the attention weight at each time step.

[0230] (3) Decoding and generation. The structural and semantic representation vectors of multiple documents are concatenated as the input of the decoder, and decoded by another recurrent neural network (RNN) to generate a unified Markdown document structure. During the decoding process, the flow of information is controlled by a gating mechanism, and the structural information of different documents is selectively integrated to maintain the overall coherence and consistency. For example, when using RNN for decoding, a two-layer LSTM with 512 hidden units can be used as the basic unit, and the beamsearch algorithm can be used for decoding, and the beamsize can be set to 5.

[0231] (4) Loss function design. During the generation process, by designing a suitable loss function, such as cross entropy loss, the model parameters are optimized to improve the quality and accuracy of generation. For example, the cross entropy loss and ROUGE-L score can be weighted averaged in a ratio of 8:2 as the final optimization target.

[0232] (5) Post-processing and optimization. Post-process and optimize the generated Markdown document. Use topic models such as LDA or TextRank to extract the subject words of the document. Use TextRank or word frequency statistics to extract the keywords of the document. Reorganize and adjust the document based on the subject words and keywords. For example, you can use the TextRank algorithm to extract the top 10 keywords, and then merge the paragraphs containing the same keywords into the same chapter.

[0233] (6) Semantic enhancement. Use external knowledge bases and semantic resources, such as WordNet and ConceptNet, to link and disambiguate key concepts and entities in documents and enhance the semantic coherence of documents. For example, you can choose ConceptNet, which has a wide coverage, as the main knowledge source and use a graph attention network-based method to learn the semantic associations between words and concepts.

[0234] (7) Manual evaluation and optimization. Perform manual evaluation and feedback on the generated Markdown documents. Based on the evaluation results and user feedback, continuously iterate and optimize the sequence-to-sequence model and document generation strategy. For example, 50 sample documents can be selected and three experts can be asked to score the generated Markdown documents. The scoring dimensions include structural coherence, semantic coherence, readability, etc., using a 5-point scale, and the average value is taken as the final score.

[0235] This embodiment can effectively identify and convert forms with complex nested structures and irregular layouts in documents, thereby ensuring the integrity and readability of form information when converting documents into Markdown format.

[0236] Embodiment 2:

[0237] like Figure 3 As shown, this embodiment provides a form enhancement method based on automatic conversion of cross-format documents to Markdown. The form enhancement method based on automatic conversion of cross-format documents to Markdown provided in this embodiment is mainly used in scenarios such as document format conversion, article publishing, structured document generation, and team collaboration. By automatically converting documents of different formats (such as Word, PDF, etc.) to Markdown, document management efficiency is improved and readability is enhanced. The method comprises the following steps:

[0238] Step K1: Obtain heterogeneous documents;

[0239] Step K2: splitting the heterogeneous document into a text part and a text image part;

[0240] Step K3: for the text portion, generating a first Markdown document corresponding to the text; and,

[0241] For the text image portion, the method for automatically converting a cross-format document to a Markdown document described in Example 1 is used to generate a second Markdown document corresponding to the text image;

[0242] The structural and semantic coherence between all the first Markdown documents and the second Markdown documents is analyzed, and the structural information between all the first Markdown documents and the second Markdown documents is merged into one Markdown document to complete the form enhancement based on automatic conversion of cross-format documents to Markdown.

[0243] Embodiment 3:

[0244] like Figure 5 As shown, this embodiment provides a device for automatically converting a cross-format document to a Markdown document, the device comprising:

[0245] An acquisition unit 10, used for acquiring a document image;

[0246] A mining unit 20, connected to the acquisition unit 10, for mining hidden structural information from the document image;

[0247] Wherein, the structural information is the hierarchical relationship between target elements and target elements; the target elements are titles, and / or paragraphs, and / or lists, and / or tables;

[0248] The analysis unit 30 is connected to the mining unit 20 and is used to perform semantic content analysis on the document image based on the structural information to obtain a semantic content analysis result;

[0249] An extraction unit 40, connected to the analysis unit 30, for extracting key information of the document image based on the semantic content analysis result;

[0250] The processing unit 50 is connected to the extraction unit 40 and is used to convert the key information into a Markdown format and perform preliminary formatting processing in a table or list to obtain a preliminary Markdown form;

[0251] The optimization unit 60 is connected to the processing unit 50 and is used to optimize the preliminary Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of a cross-format document to a Markdown document.

[0252] The device in this embodiment can execute the method in embodiment 1.

[0253] Embodiment 4:

[0254] This embodiment provides a form enhancement system based on automatic conversion of cross-format documents to Markdown, the system comprising:

[0255] An acquisition device, used for acquiring heterogeneous documents;

[0256] A splitting device, connected to the acquisition device, for splitting the heterogeneous document into a text portion and a text image portion;

[0257] A generating device, connected to the splitting device, for generating a first Markdown document corresponding to the text portion from the text portion;

[0258] The device for automatically converting a cross-format document to a Markdown document described in Example 3 is connected to the splitting device and is used to generate a second Markdown document corresponding to the image from the text image part;

[0259] A merging device is respectively connected to the generating device and the device for automatically converting cross-format documents to Markdown documents, and is used to merge all of the first Markdown documents and all of the second Markdown documents into a complete Markdown document based on the structural information and semantic consistency between the first Markdown documents and the second Markdown documents, so as to complete the form enhancement based on the automatic conversion of cross-format documents to Markdown.

[0260] The system in this embodiment can execute the method in Embodiment 2.

[0261] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for automatically converting a cross-format document to a Markdown document, characterized in that: The method comprises the following steps: Step S1: Acquire a document image; Step S2: mining hidden structural information from the document image; Wherein, the structural information is the hierarchical relationship between target elements and target elements; the target elements are titles, and / or paragraphs, and / or lists, and / or tables; Step S3: Based on the structural information, a semantic content analysis is performed on the document image to obtain a semantic content analysis result; Step S4: extracting key information of the document image based on the semantic content analysis result; Step S5: converting the key information into Markdown format, and performing preliminary formatting processing in the form of a table or list to obtain a preliminary Markdown form; Step S6: Optimize the preliminary Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of the cross-format document to a Markdown document.

2. The method for automatically converting a cross-format document to a Markdown document according to claim 1, characterized in that: The step S2 specifically includes the following steps: Step S21: using a convolutional neural network to extract features from the acquired document image to obtain a feature map; Step S22: inputting the feature map into a preset RPN to generate a candidate region; Step S23: Use RoIPooling to map each candidate region to a feature representation of a fixed size to obtain a feature representation of the candidate region; Step S24: inputting the feature representation of the candidate region into the fully connected layer for processing to obtain the coordinate positions of key elements; the key elements include titles, paragraphs, lists and tables; Step S25: Based on the coordinate positions of the key elements, analyze the spatial layout and relative positional relationship of the key elements, and infer the hierarchical relationship between the key elements; Step S26: Organizing the coordinate positions of the key elements and the hierarchical relationships between the key elements to generate a tree or graphic representation that conforms to the logical structure, thereby obtaining document image structure information.

3. The method for automatically converting a cross-format document to a Markdown document according to claim 1, characterized in that: The step S3 is specifically: Using recursive neural networks and context encoding, the structural information of document images is analyzed and recognized, and paragraph types, form structures, and code block contents are extracted to obtain comprehensive semantic content analysis results. The method uses recursive neural networks and context encoding to analyze and identify the structural information of document images, extract paragraph types, form structures, and code block contents, so as to obtain comprehensive semantic content analysis results, which specifically includes the following steps: Step S31: Segment the document image structure information into paragraphs or sentences to form a text sequence; Step S32: input the text sequence into the Bi-LSTM model for context modeling, and dynamically focus on key indicators through the attention mechanism; Wherein, the key indicators are important elements or information in the document image; Step S33: Based on the key indicators, use MLP to classify each paragraph and identify the paragraph type as plain text, form content or code block; and, Use regular expressions to extract key fields of the form content and structure the form information to obtain the form structure; and, Perform syntax analysis on the code block to obtain the semantics of the code block and obtain the content of the code block; Step S34: Integrate the paragraph type, the form structure and the code block content to form a semantic content integration result; Step S35: Optimizing and completing the semantic content integration result, so as to obtain a comprehensive semantic content analysis result.

4. The method for automatically converting a cross-format document to a Markdown document according to claim 1, characterized in that: The step S4 is specifically: Based on the results of semantic content analysis, a graph-based sequence transformation model is used to extract key information of document images; The method of extracting key information of document images based on the semantic content analysis results by using a graph-based sequence transformation model specifically includes the following steps: Step S41: converting the semantic content analysis result into a form topology structure; Wherein, the nodes in the form topology structure are each input field, and the edges in the form topology structure are the relationships between fields; Step S42: using a graph attention network to encode the form topology structure to obtain encoding information of the form topology structure; Step S43: Based on the graph convolution and attention mechanism of the graph attention network, the interaction and dependency between the nodes in the table topology structure are learned according to the table topology structure encoding information, and a node representation vector of the context information is generated; Step S44: using a conditional random field to perform sequence labeling on the input field according to the node representation vector, and identifying key information of the document image; The key information includes input fields and layout in the document image.

5. The method for automatically converting a cross-format document to a Markdown document according to claim 1, characterized in that: The step S5 specifically includes the following steps: Step S51: marking the key information to obtain a marked field sequence; Step S52: input the annotated field sequence into the Transformer-based sequence conversion model, convert the field sequence into a structured Markdown format through the self-attention mechanism, and use the template to fill in the field according to the semantic category and form layout information to generate the corresponding Markdown tag; Among them, for a table type form, the Markdown table syntax is used to generate a structured table representation; or, for a list type form, the Markdown list syntax is used to generate a structured list item; Step S53: Generate a preliminary Markdown form according to the Markdown tags.

6. The method for automatically converting a cross-format document to a Markdown document according to any one of claims 1 to 5, characterized in that: The step S6 specifically includes the following steps: Step S61: comparing the layout structure differences between the document image and the preliminary Markdown form, and implementing form structure verification and dynamic display of the preliminary Markdown form through a script or plug-in tool to obtain a first optimized Markdown form; and Generate a title and a subtitle of the preliminary Markdown form according to the structural information, and perform syntax formatting on the lists and code blocks in the preliminary Markdown form to obtain a second optimized Markdown form; and Analyze the list items in the preliminary Markdown form, identify the indentation, numbering or symbol features of the list items, and convert the unordered list and the ordered list into the Markdown format accordingly to obtain the third optimized Markdown form; Step S62: Integrate the first optimized Markdown form, the second optimized Markdown form, and the third optimized Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

7. The method for automatically converting a cross-format document to a Markdown document according to claim 6, characterized in that: In the step S61, the layout structure differences between the document image and the preliminary Markdown form are compared, and the form structure verification and dynamic display of the preliminary Markdown form are implemented through a script or plug-in tool to obtain a first optimized Markdown form, which specifically includes the following steps: Step A1: using an edit distance algorithm to compare the layout structure differences between the document image and the preliminary Markdown form to obtain a difference measurement value; The edit distance algorithm specifically quantifies the difference in the number of insertion, deletion and replacement operations; Step A2: Automatically optimize the layout structure of the preliminary Markdown form according to the difference measurement value to obtain a first process Markdown form; the automatic optimization includes automatically adjusting the alignment, indentation level and blank line separation; Step A3: A supporting script tool or plug-in parses and optimizes the Markdown form of the first process, generates a display template, and uses a preset template engine for rendering and data binding to obtain a first optimized Markdown form; Among them, the verification and error correction functions are integrated in the supporting script tool or plug-in, and the field type, value range and required items are automatically checked through predefined rules, and error prompts and correction suggestions are provided; In the step S61, according to the structural information, the title and subtitle of the preliminary Markdown form are generated accordingly, and the list and code block in the preliminary Markdown form are formatted grammatically to obtain a second optimized Markdown form, which specifically includes the following steps: Step B1: Based on the structural information, the document content is automatically annotated using a conditional random field sequence annotation model to obtain an annotation result; The annotation results are used to identify different document elements and extract context and location information features; Step B2, automatically generating a title and a subtitle in Markdown format according to the annotation results, and obtaining a preliminary Markdown form with a hierarchical relationship; Step B3: performing multi-level list indentation and line break processing on the preliminary Markdown form with a hierarchical relationship, and formatting the code blocks in the preliminary Markdown form with a hierarchical relationship, thereby obtaining a second process optimized Markdown form; Step B4: parsing and rendering the Markdown form optimized by the second process into an HTML page or PDF file corresponding to the visual structure of the document image, evaluating the accuracy and completeness of the Markdown text generation by comparison, and continuously optimizing the sequence annotation model and the Markdown generation rules, thereby forming a second optimized Markdown form; In the step S61, the list items in the preliminary Markdown form are analyzed to identify the indentation, numbering or symbol features of the list items, and the unordered list and the ordered list are converted into the Markdown format accordingly to obtain a third optimized Markdown form, which specifically includes the following steps: Step C1: Take the list items in the preliminary Markdown form as input, use the conditional random field (CRF) model to perform sequence labeling to identify indentation, numbering and symbol features, build feature templates and extract relevant attributes to obtain new list item text; Among them, the conditional random field CRF model is trained using the gradient descent method and the L-BFGS algorithm; Step C2: using the trained conditional random field (CRF) model to perform sequence labeling on the new list item text, identifying the structural features of each list item, determining the list type to which it belongs, and obtaining lists of different types; Step C3: generating a third process optimized Markdown form according to the different types of lists and based on the logical consistency of indentation and numbering; Step C4: According to the list item text in the document image, the list in the Markdown form optimized by the third process is evaluated and optimized to obtain a third optimized Markdown form.

8. A form enhancement method based on automatic conversion of cross-format documents to Markdown, characterized in that: The method comprises the following steps: Step K1: Obtain heterogeneous documents; Step K2: splitting the heterogeneous document into a text part and a text image part; Step K3: for the text portion, generate a first Markdown document corresponding to the text; as well as, For the text image portion, the method for automatically converting a cross-format document to a Markdown document according to any one of claims 1 to 8 is used to generate a second Markdown document corresponding to the text image; The structural and semantic coherence between all the first Markdown documents and the second Markdown documents is analyzed, and the structural information between all the first Markdown documents and the second Markdown documents is merged into one Markdown document to complete the form enhancement based on automatic conversion of cross-format documents to Markdown.

9. A device for automatically converting a cross-format document to a Markdown document, characterized in that: The device comprises: An acquisition unit, used for acquiring a document image; A mining unit, connected to the acquisition unit, for mining hidden structural information from the document image; Wherein, the structural information is the hierarchical relationship between target elements and target elements; the target elements are titles, and / or paragraphs, and / or lists, and / or tables; An analysis unit, connected to the mining unit, configured to perform semantic content analysis on the document image based on the structural information to obtain a semantic content analysis result; An extraction unit, connected to the analysis unit, for extracting key information of the document image based on the semantic content analysis result; A processing unit, connected to the extraction unit, for converting the key information into a Markdown format and performing preliminary formatting processing in a table or list to obtain a preliminary Markdown form; The optimization unit is connected to the processing unit and is used to optimize the preliminary Markdown form to obtain a target Markdown form, thereby completing the automatic conversion of cross-format documents to Markdown documents.

10. A form enhancement system based on automatic conversion of cross-format documents to Markdown, characterized in that: The system comprises: An acquisition device, used for acquiring heterogeneous documents; A splitting device, connected to the acquisition device, for splitting the heterogeneous document into a text portion and a text image portion; A generating device, connected to the splitting device, for generating a first Markdown document corresponding to the text portion from the text portion; The device for automatically converting a cross-format document to a Markdown document as described in claim 9 is connected to the splitting device and is used to generate a second Markdown document corresponding to the image from the text image part; A merging device is respectively connected to the generating device and the device for automatically converting cross-format documents to Markdown documents, and is used to merge all of the first Markdown documents and all of the second Markdown documents into a complete Markdown document based on the structural information and semantic consistency between the first Markdown documents and the second Markdown documents, so as to complete the form enhancement based on the automatic conversion of cross-format documents to Markdown.

Images