Data processing method and device suitable for retrieval enhancement generation
By extracting and optimizing text structure using a large language model and mapping it to Markdown format, the problem of knowledge dependency in generative language models and structure loss in document transcription is solved, achieving efficient structured processing and accurate generation of long texts.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ANT BLOCKCHAIN TECHNOLOGY (SHANGHAI) CO LTD
- Filing Date
- 2026-01-06
- Publication Date
- 2026-05-12
AI Technical Summary
Existing generative language models suffer from the problem of relying on training data for knowledge in text generation, making it impossible to obtain the latest information in real time. The generated content lacks accuracy and may produce illusions. Furthermore, existing document transcription techniques are inefficient when dealing with complex layouts and long texts, resulting in the loss or error of text structure.
By extracting text structure using a large language model, optimizing hierarchical relationships, mapping to Markdown format, and performing structured slicing, the problems of lost and erroneous text structure are solved. This approach is suitable for long text processing and reduces the risk of illusions.
It optimizes document structure and improves accuracy, is applicable to various structured slicing methods, reduces processing costs and time, and improves the accuracy and efficiency of generated content.
Smart Images
Figure CN122019748A_ABST
Abstract
Description
Technical Field
[0001] This specification relates to one or more embodiments in the field of computer technology, and more particularly to data processing methods and apparatus suitable for retrieval enhancement generation. Background Technology
[0002] With the rapid development of Natural Language Processing (NLP) technology, generative language models (such as GPT and LLaMA) have demonstrated powerful capabilities in text generation, question answering systems, and dialogue interaction. However, these models also have some inherent limitations. For example, their knowledge relies entirely on training data and cannot acquire the latest information in real time; the generated content may lack accuracy, or even exhibit "illusion" phenomena (i.e., the generated content appears reasonable but does not conform to reality). To address these issues, researchers have proposed the RAG (Retrieval-Augmented Generation) architecture.
[0003] The core idea of RAG (Resource Aggregator) is to combine an external knowledge base with a generative model. A retrieval module retrieves relevant text or fragments from the external knowledge base, and the generative module then enhances the final output based on this expanded information. This approach not only improves the accuracy and timeliness of the generated content but also enhances the model's performance in specific domain tasks. For example, in vertical fields such as medicine, law, or scientific research, RAG can generate more credible answers by citing authoritative sources.
[0004] When building a RAG system, the quality of knowledge provision is crucial. In practice, raw input files may come from various sources, such as PDF (Portable Document Format) documents, scanned copies, images, Word documents, or other unstructured formats. However, textual structure information is essential for improving the performance of a RAG system, as it provides semantic and logical contextual clues. Optimizing the structured information of the text (such as headings, paragraphs, and sections) can be used to segment the text into smaller semantic units (chunks), facilitating index building. Good text structure can significantly improve the performance of the retrieval module. Various types of files often contain complex layouts, embedded images, or tables; using them directly can lead to difficulties in information extraction, low retrieval efficiency, and even erroneous parsing. If various models (such as OCR models, ASR speech recognition models, and page layout detection models) are used for document transcription, problems such as lost, incomplete, or incorrect text structure may occur due to model limitations or the lack of structure in the original document. Summary of the Invention
[0005] This specification describes one or more embodiments of a data processing method and apparatus suitable for retrieval enhancement generation, which solves one or more problems mentioned in the background art.
[0006] According to a first aspect, a data processing method suitable for retrieval enhancement generation is provided, used to convert text used for retrieval enhancement into Markdown format text. The method includes: obtaining a first text to be processed; extracting the text structure of the first text using a large structure extraction model to obtain initial structure information, the initial structure information including a title and hierarchical identifiers in Markdown format; aligning the initial structure information with hierarchical relationships using an alignment model to obtain target structure information corrected for the initial structure information; and mapping the target structure information onto the first text to obtain a first structured text in Markdown format, used for text slicing according to a predetermined method.
[0007] In one embodiment, the large structure extraction model corresponds to a length window, which describes the maximum length of text processed by the large structure extraction model in a single operation. When the number of bytes in the first text is greater than the length window, the step of extracting the text structure of the first text using the large structure extraction model to obtain initial structure information includes: truncating the first text into multiple segments according to the length window, wherein the number of bytes in a single segment is not greater than the length window; extracting the structure information of each sub-document corresponding to each segment using the large structure extraction model; and merging the structure information of each sub-document to obtain the initial structure information of the first text.
[0008] In one embodiment, the hierarchical alignment includes aligning titles that meet predetermined rules to the same level, the predetermined rules including at least one of the following: semantic similarity greater than a predetermined similarity threshold, containing the same keywords, or containing the same sentence structure.
[0009] In one embodiment, the structural extraction big model and the alignment big model are the same big language model.
[0010] In one embodiment, the target structure information includes multiple structures, each structure including a single title and a corresponding structure hierarchy identifier. Mapping the target structure information onto the first text to obtain the first structured text in Markdown format includes: segmenting the first text into sentences; sequentially matching each title in the multiple structures with each sentence to obtain each target sentence that matches each title; and replacing each matching target sentence with the corresponding structure to obtain the first structured text.
[0011] In a further embodiment, the plurality of structures includes a first structure corresponding to a first title. The step of sequentially matching each sentence with each title in the plurality of structures to obtain each target sentence that matches each title includes: if the first structure is the first structure, starting from the first sentence, sequentially matching each sentence with the first title until a matching first target sentence is obtained; if the first structure is not the first structure, starting from the sentence following the target sentence that matches the previous structure, sequentially matching each sentence with the first title until a matching first target sentence is obtained.
[0012] In a further embodiment, the matching method between a single title and each sentence is as follows: the similarity between the title string corresponding to the single title and each sentence string is calculated by at least one of the following: edit distance ratio, longest common substring, and predetermined model score; the sentence with the highest similarity is determined as the single target sentence that matches the single title.
[0013] In a further embodiment, when a single title matches a single sentence, the longest common substring in the single sentence that matches the single title is obtained as the target sentence, and the subsequent characters of the longest common substring are used as the next sentence for matching the next title.
[0014] In one embodiment, the predetermined method is a structured slicing based on at least one of the following logical structure slicing rules: title, chapter, table, list.
[0015] According to a second aspect, a data processing apparatus suitable for generating retrieval enhancements is provided, for converting text used as retrieval enhancements into Markdown format text, the apparatus comprising:
[0016] The acquisition unit is configured to acquire the first text to be processed.
[0017] The extraction unit is configured to extract the text structure of the first text using a large structure extraction model to obtain initial structure information, which includes heading levels in Markdown format.
[0018] The alignment unit is configured to perform hierarchical relationship alignment on the initial structural information by aligning the large alignment model to obtain target structural information corrected for the initial structural information.
[0019] The mapping unit is configured to map the target structure information onto the first text to obtain a first structured text in Markdown format, which is used for text slicing in a predetermined manner.
[0020] According to a third aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method of the first aspect.
[0021] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method of the first aspect.
[0022] The methods and apparatus provided in the embodiments of this specification allow for the generation of data suitable for retrieval enhancement through structured processing in Markdown format. For a specific text, such as the first text, the text structure is first extracted to obtain initial structural information in Markdown format. Then, the initial structural information is aligned hierarchically to obtain corrected target structural information. Finally, the target structural information is mapped to the first text to obtain structured text. This structured text can be sliced according to one or more corresponding structured slicing methods as a data reserve for retrieval enhancement generation. This method of data processing, through three stages—coarse structure extraction, structure optimization, and structure restoration—optimizes the speed and effectiveness of text structuring, solves problems such as lost, incomplete, and incorrect text structure during document transcription, and is applicable to long texts. Furthermore, in the long text processing, the structure optimization stage deals with the full-text structural information, not the full-text content, which reduces the risk of illusion. Attached Figure Description
[0023] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 The diagram illustrates an application scenario based on search enhancement.
[0025] Figure 2 This is a schematic diagram of the data processing architecture generated by retrieval enhancement in conventional technologies;
[0026] Figure 3 A flowchart illustrating a specific example of expanding a performance evaluation dataset for a large language model is provided.
[0027] Figure 4 This diagram illustrates how target structural information is mapped to the original text in a specific example.
[0028] Figure 5 This illustrates a structured process for long texts according to embodiments of this specification;
[0029] Figure 6 A screenshot showing an example of the final generated Markdown formatted structured document;
[0030] Figure 7 This is a schematic block diagram of a data processing apparatus suitable for retrieval enhancement generation according to one embodiment of this specification. Detailed Implementation
[0031] First, let's explain some of the terms that may be used.
[0032] A language model is a mathematical model that uses probability distributions to describe the probability of a word string or string.
[0033] Large Language Models (LLMs) are complex neural network models trained using deep learning techniques that are capable of understanding and generating human natural language. These models typically have hundreds of millions to trillions of parameters, enabling them to excel in a wide range of natural language processing tasks, including but not limited to text generation, translation, question answering, summarizing, and dialogue. LLMs are generally used in two modes: direct inference and fine-tuning. In direct inference mode, users can design prompts to guide the LLM to generate specific outputs. Prompts can be task descriptions or instructions in text form, used to stimulate the LLM's semantic understanding and generation capabilities. In fine-tuning mode, the LLM is further trained on small datasets within a specific domain to optimize its performance on specific tasks.
[0034] Retrival Augmented Generation (RAG), also known as retrieval-enhanced generation, is a system architecture for natural language processing that combines the advantages of retrieval-based and generative models. RAG retrieves relevant knowledge fragments from a knowledge base using a retrieval-based approach, and then utilizes a generative model for further processing and generation. This approach, combining two models, improves the model's performance on tasks such as question answering, text generation, and semantic search, addressing the timeliness and illusion problems of LLM knowledge.
[0035] A prompt is information used to guide the model in generating natural language text. A prompt can be one or more of the following: a complete sentence, a question, reference text, task description, examples, etc. It provides the model with a starting point and direction for text generation. The model understands the context based on the prompt and generates context-appropriate text content accordingly. The quality and accuracy of the prompt have a significant impact on the quality of the text generated by the model, helping it to better understand the user's intent and generate more reasonable and fluent responses.
[0036] Markdown is a plain text document format that uses Markdown syntax to define the structure and format of text. As a lightweight markup language, it was created by John Gruber and Aaron Swartz in 2004 to allow authors to write content in a simple, readable, and easy-to-write plain text format, which can then be easily converted to HTML (Hypertext Markup Language) or other formats. Markdown is characterized by its simplicity, readability, portability, and convertibility. Its simplicity refers to the use of intuitive symbols to represent formatting, such as # for headings and * for list items; its readability refers to its ability to clearly display document structure and hierarchy; its portability means that as a plain text file, it can be opened and edited in any text editor, without relying on specific software or operating systems; and its convertibility means that it can be easily converted to various formats such as HTML, PDF, and Word to meet different publishing needs. The syntax of Markdown documents includes: using "#" to represent headings, "##" to represent second-level headings, "*" or "-" to represent list items, "**text**" or "--text--" to represent bold text, "*text*" or "-text-" to represent italics, and so on.
[0037] The solution provided in this specification will now be described with reference to the accompanying drawings.
[0038] Figure 1 This illustrates a specific application scenario for the generative model. The specific application scenario is a question-and-answer platform. Its architecture can include a client and a server. The client can be a user terminal or a question-and-answer platform application running on the user terminal. The client can obtain user questions through interaction between the user and the user terminal. User questions can be various questions, instructions, or requests raised by the user seeking answers, such as... Figure 1The example uses the phrase "I want to lose weight, please help me create a weight loss plan." The client sends this question to the server via the network. The server, using a computing platform equipped with a generative model, retrieves target text from candidate texts based on the user's question and performs information augmentation. Target text can include web pages and various pre-collected information (such as documents, textbooks, papers, images, etc.). The generative model then uses the augmented information and the user's question to generate the corresponding answer. The answer is then fed back to the client via the server.
[0039] Understandable. Figure 1 A simple example is given. In real-world scenarios, there can be multiple clients, or the computing platform can be set up on the user terminal without a server. Specific scenarios are not limited to question-and-answer, but can also include various scenarios for completing instructions and tasks.
[0040] Figure 2 This paper illustrates a concrete example of the RAG (Rich Language Acquisition) technology chain for a large language model, including step 1 (knowledge base construction), step 2 (document retrieval), and step 3 (answer synthesis). Document retrieval refers to knowledge retrieval enhancement tailored to specific user questions. Figure 2 In the specific example shown, the retrieval process can be implemented through vector matching, but in practice it can also take other forms, such as keyword matching.
[0041] like Figure 2 As shown, in step 1, various file types can be received as input, such as Word documents, PDF documents, PowerPoint presentations (PPT), HTML (Hypertext Markup Language), etc. These texts typically contain multiple information formats, including text, images, tables, and charts, and have complex layouts; direct processing can lead to information loss or incorrect parsing. To extract high-quality knowledge from these texts, it is necessary to combine technologies such as layout analysis, OCR (Optical Character Recognition), reading order detection models, and table recognition to convert heterogeneous files from multiple sources into a unified file format, such as Markdown (files written in plain text format) or Text. After converting various file types into structured text, document chunking can be performed. The sliced documents, after vectorization processing (such as semantic embedding), can be stored in a vector database (also known as a knowledge base). A single data entry in the vector database can include a single vector and correspond to a single document slice.
[0042] In step 2, the actual user question (or request) can be vectorized. The resulting user question vector is then retrieved from a vector database to obtain several document chunks (TopK Chunks) for knowledge enhancement. The retrieval component searches the knowledge base (i.e., the vector database) for the most relevant fragments or text based on the user question. This process may involve natural language matching techniques, such as text matching and semantic similarity calculation, to ensure that the found document chunks are more relevant to the user's needs.
[0043] Then, in step 3, the retrieved relevant information is passed to the generative model. By using the user question and several knowledge-enhanced document slices as input to the generative model, an answer is generated for the user question. For example, in the case of a Large Language Model (LLM), the user question and several knowledge-enhanced document slices are used as prompts for the LLM to generate the answer. In this stage, the generative model relies not only on internally trained data (such as the corpus used during pre-training) but also on enhanced information retrieved from the knowledge base to generate more accurate, detailed, and targeted answers or content. This approach allows the generated answer to better reflect the latest, specific, or domain-specific knowledge. After the information fusion and generation stages, the system outputs the final result to the user. This may be a direct answer to the question, a suggestion, an explanation, or other form of content designed to meet the user's specific needs.
[0044] In particular, the discussion in dashed box 201 pertains to the document processing part of the knowledge base construction process, aiming to correct problems such as missing text structure and inaccurate text structure that may occur during text transcription.
[0045] In conventional techniques, document processing methods can be document slicing or document transcription. Document slicing methods include: fixed-length slicing, which divides the document into segments of a fixed length (e.g., a fixed number of characters, words, or tokens); paragraph slicing, which slices the document according to its paragraph structure, with each paragraph as an independent segment; sentence slicing, which divides the document into sentences, with each sentence as a segment; structured slicing, which slices the document according to its logical structure (e.g., headings, chapters, tables, lists); semantic-based slicing, which uses natural language processing techniques (e.g., sentence embedding, topic modeling) to analyze the document content and slices based on semantic similarity; and so on.
[0046] Among the various approaches: Fixed-length slicing is simple to implement, requires no additional analysis, and is suitable for plain text documents with low structure; however, it may compromise semantic integrity (e.g., splitting a complete sentence into two parts) and performs poorly for tasks sensitive to paragraph boundaries. Paragraph slicing preserves paragraph-level semantic integrity and is suitable for highly structured documents (e.g., academic papers, technical manuals), but excessively long paragraphs may still lead to contextual constraints on the generative model. Sentence slicing maintains sentence-level semantic integrity and is suitable for tasks requiring fine-grained analysis; however, the number of segments may be excessive, increasing the complexity of indexing and retrieval. Structured slicing preserves the semantic and logical structure of the document to the greatest extent possible and is suitable for handling complex documents (e.g., books, reports, technical documents), but requires deep parsing of the document, resulting in high implementation complexity, and is typically used for Markdown documents. Semantic-based slicing can dynamically adjust the slice size to ensure semantic consistency of each segment. It is suitable for handling unstructured or semantically complex documents, but requires additional NLP tools and computational resources.
[0047] Common document transcription methods include: Pure multimodal large model approach, which splits the document into pages and converts each page into an image, then directly inputs the image content into a multimodal large model, such as claude-4 or gemini-2.5 pro, along with a prompt, and uses the multimodal large model's output as the parsing result; document divide-and-conquer approach, which uses different parsing schemes for different types of files based on their file protocol or encoding method. For example, there are corresponding tools for parsing Office-related files (e.g., doc, docx, ppt), html, and pdf, such as liboffice; and layout analysis + multimodal large model + sequence model approach, where files are uniformly converted to image format, then the layout analysis model identifies the layout areas, the identified areas are uniformly parsed by the multimodal large model, and finally the layout is restored using a sequence detection model.
[0048] These solutions often involve other tools or models, rely on their performance, and increase resource usage costs during processing. Especially for long texts, document transcription solutions combined with large models may also suffer from several drawbacks, including: unsuitability for long texts (large models typically have limited context windows, making it impossible to fit extremely long texts entirely into them), low efficiency (the time consumed by large models is directly proportional to the number of output tokens, with long texts showing significantly longer processing times), the need for high computing power and the cost of using large models (some high-performance large models charge based on the number of tokens), the illusion of large language models, and the addition of content not present in the original text, among other issues.
[0049] In view of this, this specification provides a data processing scheme suitable for retrieval enhancement generation. First, various documents are structured into Markdown format, making them applicable to various structured slicing methods and long texts. Specifically, a large language model is first used to extract structural information from the text, obtaining the initial structural information of the entire text. Then, the large language model optimizes the document structure information of the entire text, finely distinguishing levels to obtain corrected document structure information, denoted as the target structure information. The corrected document structure information is mapped to the corresponding positions in the initial text, resulting in text with a fine structure, which is then used for document slicing using various conventional techniques. Because the large language model only operates on the text structure and does not participate in the output of the full text content when optimizing the document structure, the illusion risk of the large language model is reduced, and the number of output tokens is significantly reduced, thus reducing costs and processing time.
[0050] Furthermore, for longer texts, the original text can be initially sliced using a combination of length and rule-based slicing methods. Structural information is extracted from each slice, and the extracted structural information is then merged to obtain the initial structural information of the entire text. This information is then refined using a large language model. In this way, the text length can be extended to arbitrary lengths, solving the problem that conventional slicing methods are difficult to apply to long texts.
[0051] The technical concept of this specification is described in detail below with reference to the accompanying drawings.
[0052] Figure 3 This specification illustrates a data processing flow suitable for retrieval enhancement generation, provided by one embodiment. The execution entity of this flow can be a computer, device, or server with a certain computing power; more specifically, for example… Figure 1 This process can perform Markdown-formatted structured processing on text of any format, and the processed text can be used for document slicing using various text structure-based slicing methods.
[0053] like Figure 3 As shown, the data processing flow for retrieval enhancement generation provided in this specification may include: Step 301, obtaining the first text to be processed; Step 302, extracting the text structure of the first text using a large structure extraction model to obtain initial structure information, which includes heading levels in Markdown format; Step 303, aligning the hierarchical relationships of the initial structure information using an alignment model to obtain target structure information corrected for the initial structure information; Step 304, mapping the target structure information to the first text to obtain the first structured text in Markdown format, which is used for text slicing according to a predetermined method.
[0054] First, in step 301, the first text to be processed is obtained.
[0055] It is understandable that the term "text" generally refers to the written form of language, including sentences, paragraphs, or chapters—written expressions with complete semantic meaning. In a broad sense, text can be any fixed discourse formed by writing; in a narrow sense, text can be a self-sufficient system composed of language and writing.
[0056] Here, the text to be processed is the text to be structured, which can be in various file formats such as Word documents, PDF documents, PowerPoint presentations, HTML documents, etc. The text to be processed can come from a pre-collected knowledge base. The first text can be any one of the texts to be processed.
[0057] When the document contains images or other information, according to one optional embodiment, only the text content in the file can be obtained as the corresponding text, such as the first text. According to another optional embodiment, the images and other information can be recognized as text information through OCR, large language models, etc., and added to the original text of the document as the text, such as the first text. In short, the first text can be determined based on the original single document.
[0058] Next, in step 302, the text structure of the first text is extracted using a large language model to obtain initial structural information.
[0059] Text structure often refers to the overall framework and organizational form of a text. Understanding text structure helps in better analyzing and comprehending the text content. The overall framework may include, for example, a title, introduction, body, and content modules. The organizational form describes the order and logical relationships between the various parts of the text, such as sequential relationships, parallel relationships, and inclusion relationships.
[0060] For the first text, its structure extraction can be achieved using a large language model. As described earlier regarding the use of large language models, this can be a pre-trained model specifically designed for structure extraction, or a general-purpose large language model. When using a general-purpose large language model, the first text, the structure extraction task and its rules, and a small number (e.g., 5) of samples can be provided as prompts. The large language model then utilizes its powerful natural language processing and logical reasoning capabilities to extract the structure from the first text. Optionally, the prompts can also include information such as the format requirements for the returned data. For ease of description, the large language model performing the structure extraction can be referred to as the structure extraction large model.
[0061] As a concrete example, suppose the original text (the ellipsis indicates omitted text content) is as follows:
[0062] "Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors of *** Insurance Co., Ltd."
[0063] General Principles
[0064] ...
[0065] Article 1
[0066] ...
[0067] Article 2
[0068] ...
[0069] Article 3
[0070] ...
[0071] Article 4
[0072] ...
[0073] Article 5
[0074] ...
[0075] Insurance liability
[0076] ……
[0077] When using a large language model for structure extraction, the prompts input to the large language model could be, for example:
[0078] "# Role
[0079] You are a professional Markdown heading extraction expert, skilled at extracting potential headings from unstructured text and outputting the heading content in Markdown Header format.
[0080] # rule
[0081] 0. **Knowledge**: Generally, punctuation marks are not present in headers.
[0082] 1. **Identify Headings:** Analyze the text content, identify possible headings, and display them using Markdown's Header format. Headings may not be at the beginning of the text.
[0083] 2. **Header hierarchy consistency:** When generating headers, refer to the previous header information to ensure that the output hierarchy is consistent, but this does not include the content in `history_headers`.
[0084] # Sample
[0085] ## example1 (Sample 1)
[0086] human: {"content": ""}
[0087] ai: {"result": ""} (Artificial Intelligence: {"result": "…………"})
[0088] …………
[0089] # Return format requirements
[0090] Please provide your output in Chinese according to the following structure, and please use \n for line breaks.
[0091] ```
[0092] {
[0093] "result": "This is where you put the converted Markdown title"
[0094] }
[0095] ```
[0096] user_prompt: |- (User prompt)
[0097] {{
[0098] "content": "{content} (current text content)"
[0099] }}”.
[0100] Based on the above prompts, the resulting text structure is as follows:
[0101] # *** Insurance Co., Ltd.
[0102] ## Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors
[0103] ## General Principles
[0104] ### First Article
[0105] ####First Model
[0106] ### Second Article
[0107] ...
[0108] As is understandable, in Markdown format, # represents a first-level heading, ## represents a second-level heading, ### represents a third-level heading, and so on.
[0109] Typically, large language models have a length window when processing document data; for example, a large language model might have a length window of 128K (128 kilobytes). This length window corresponds to the maximum processing length of a single text by the large language model. If the length of the first text exceeds the length window of the structure extraction model, it can be considered a long text. In this case, the structure extraction model struggles to process it directly. To adapt to the processing rules of the large language model, the first text can be initially segmented according to the length window of the structure extraction model, resulting in multiple segments. This initial segmentation can be done using one of the following methods: paragraph slicing, sentence slicing, or fixed-length slicing (where the fixed length is no greater than the length window of the structure extraction model; in this case, semantic boundaries such as paragraphs or sentences can be used as boundaries to avoid cutting off headings). Then, for each segment, the structure can be extracted using the large language model, for example, denoted as the structure information of each sub-document.
[0110] Using the previous text as an example, after initial slicing, we get the following segments:
[0111] Segment 1
[0112] "Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors of *** Insurance Co., Ltd."
[0113] General Principles
[0114] ...
[0115] Article 1
[0116] ...";
[0117] Segment 2
[0118] Article 2
[0119] ...
[0120] Article 3
[0121] ...
[0122] Article 4
[0123] ...
[0124] Article 5
[0125] ...";
[0126] Segment 3
[0127] Insurance liability
[0128] ……
[0129] Extract the document structure for each segment separately to obtain the sub-document structure information for each segment, for example:
[0130] Subdocument structure information for fragment 1:
[0131] # *** Insurance Co., Ltd.
[0132] # Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors
[0133] ## General Principles
[0134] ### Article 1;
[0135] Subdocument structure information for fragment 2:
[0136] Article 2
[0137] # Article 3
[0138] # Article 4
[0139] # Article 5;
[0140] Subdocument structure information for fragment 3:
[0141] "# Insurance Liability".
[0142] The document structure extraction methods for each segment are described above and will not be repeated here. In an optional embodiment, when extracting the sub-document structure information of later segments, the structure information of earlier segments can be used as reference information, i.e., as prompting information for the overall structure extraction model. For example, in the prompting information of the previous example, before or after ""content": "{content}" (where the current text content is the current segment) in the "user_prompt" section, the prompt "history_headers":"{history_headers}" (historically mined titles, such as the titles of earlier segments, which may include title hierarchy information) can be added.
[0143] In an optional embodiment, the extracted document structure information can be formatted. If formats other than Markdown are included, and an error is found or illegal information is present, the document structure information can be re-extracted, or illegal information can be deleted, retaining only the valid information (information conforming to Markdown format) as the extracted document structure information. For example, if illegal information is deleted and valid information is retained, the large language model is configured to output only responses conforming to a preset JSON Schema, which specifies that the 'result' field is a non-empty string and contains only valid Markdown Header lines.
[0144] It's understandable that the structural information only includes the document structure, not the actual text content. Then, by merging the sub-document structural information of each segment, we can obtain the initial structural information of the first text. For example:
[0145] # *** Insurance Co., Ltd.
[0146] # Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors
[0147] ## General Principles
[0148] ### First Article
[0149] # Second Article
[0150] # Article 3
[0151] # Article 4
[0152] # Article 5
[0153] #Insurance liability.
[0154] It's understandable that in Markdown format, the number of # symbols represents the heading level, serving as a hierarchy identifier. The initial structural information is a rough structure and may contain some hierarchical errors, especially when the first text is long. Since each segment is obtained from different calls to the larger structural model, and contextual hierarchy information may be lost during the extraction process, the hierarchy may be disordered. For example, in the previous text, segment 1 contains the second-level heading "General Principles" and its preceding first-level heading, so the first item is identified as a third-level heading. However, in segment 2, since only headings like "Item *" exist, all are identified as first-level headings. Clearly, "Item *" in segment 1 and "Item *" in segment 2 should belong to the same level of heading.
[0155] In order to obtain a more accurate and reasonable title, in step 303, the hierarchical relationship of the initial structural information is aligned by aligning the large model to obtain the target structural information after correction of the initial structural information.
[0156] In layman's terms, hierarchical alignment means representing parallel headings as belonging to the same level. For example, in the previous example, the headings "Article 1" and "Article 2" are at the same level. Headings at the same level are usually parallel and often share certain similarities, such as: similar meanings (e.g., Article 1, Article 2); similar sentence structures (e.g., improve accuracy, reduce hallucinations); overlapping words (e.g., "Clause ****", "Clause **********"); and so on.
[0157] In some alternative implementations, hierarchical alignment can be determined based on whether it conforms to predetermined rules. These predetermined rules may include at least one of the following: semantic similarity, identical sentence structure, or the presence of the same keywords.
[0158] In one embodiment, a corresponding module can determine whether a title conforms to predetermined rules. For example, semantic similarity can be determined by a semantic similarity calculation module. If the vector similarity of the semantic embedding vectors is greater than a predetermined threshold, semantic similarity is determined. Sentence structure similarity can be determined by sentence structure detection; for example, if both titles are verb + object (e.g., improving accuracy, reducing hallucinations), they are considered similar. If two titles have the same keywords, they can be included in the same position (e.g., at the end), and so on. In this case, if two titles at different levels meet the predetermined rules, they can be aligned to the same level; if two titles at the same level do not meet the predetermined rules, at least one of the titles can be aligned to a different level.
[0159] In another embodiment, a large language model can be used to determine whether a title conforms to a predetermined rule. In this case, the predetermined rule can be used as a prompt for the large language model. For example, the prompt might include phrases like "Ensure that semantically similar headers are categorized into the same Markdown level." Optionally, to prevent the large language model from becoming too divergent, constraints can be specified in the prompt, such as "Strictly review and correct according to the header information provided by the user; no header content may be added or deleted." In this case, if two titles at different levels are determined by the large language model to conform to the predetermined rule, the two titles can be corrected to the same level. The specific level to which they are corrected is determined by the large language model based on its internal logical reasoning.
[0160] As a concrete example, the prompts for title hierarchy alignment using a large language model are as follows:
[0161] "You are a professional Markdown hierarchy review expert, responsible for reviewing and correcting the reasonableness of the Header hierarchy in Markdown documents."
[0162] # Rules and Regulations
[0163] 1. **Semantic Hierarchy Matching**: Based on the semantic content of the Header, ensure that Headers with similar semantics are classified into the same Markdown level.
[0164] 2. **Content Completeness:** Header information must be strictly reviewed and corrected based on user-provided header information. No header content may be added or deleted.
[0165] # Response Format (Output Format)
[0166] Please provide your output in Chinese according to the following structure, and please use \n for line breaks.
[0167] ```
[0168] {
[0169] "result": "This is where you put the converted Markdown content"
[0170] }
[0171] ```
[0172] user_prompt: |-
[0173] {{
[0174] "content": {content (initial structural information of the first text)}
[0175] }}”.
[0176] It's important to note that the prompt information provides the structural information of the first text, not its content. The large language model here, denoted as the alignment model, can be the same as the structure extraction model, or it can be different. If it's the same large language model, different prompts, such as different task objectives or different texts, can be used to invoke the large language model to complete the relevant task. This large language model can be a general-purpose model not trained in a specific vertical domain, or it can be a vertical domain model trained using data from a specific domain, such as the insurance field; there are no restrictions here.
[0177] In other alternative implementations, a large language model can be directly used as the alignment model, combined with a small number (e.g., 3-5) of examples. The alignment model then uses its extensive corpus accumulation, logical analysis, and learning capabilities to obtain the alignment results. In some embodiments, to comprehensively explore situations where headings may be at the same level, different types of examples can be used as prompts. For example, each example may contain headings at the same level with the same semantics, headings at the same level with similar sentence structures, or headings containing the same keywords.
[0178] In other implementations, header level alignment in the initial structural information can be achieved in other ways, which will not be elaborated here. The structural information after header level alignment can be called the target structural information. In the previous example, the target structural information after header level alignment is as follows:
[0179] # *** Insurance Co., Ltd.
[0180] # Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors
[0181] ## General Principles
[0182] ### First Article
[0183] ### Second Article
[0184] ### Article 3
[0185] ### Article 4
[0186] ### Article 5
[0187] ## Insurance liability.
[0188] As can be seen, the original first-level headings such as the second and third items, after hierarchical alignment, belong to the same third-level headings as the first item. In the target structure information, a single heading can be recorded as a single structure, and a single structure corresponds to a single heading and the hierarchical identifier of that single heading (such as #). In optional embodiments, when using a large structure extraction model to extract the text structure of the first text, or when aligning the hierarchical relationship of the initial structure information through an alignment model, constraint rules, such as number system skipping, are added to logically constrain the large language model.
[0189] Further, in step 304, the target structure information is mapped onto the first text to obtain the first structured text in Markdown format, which is used for text slicing in a predetermined manner.
[0190] The purpose of obtaining structural information is to structure the text, allowing the target structural information to be mapped back to the first text, forming structured text within a Markdown structure. One way to map the target structural information back to the first text is to replace the original title with the headings (such as the first item) and hierarchical identifiers (such as "###") representing the structure in the target structural information, or to insert them into the heading positions of the first text. This gives the first text a Markdown structure, making it Markdown formatted text.
[0191] In one embodiment, the location of the target structural information mapped to the first text can be determined by matching each sentence in the first text with the titles of each structure in the target structural information. Specifically, on the one hand, the text content of the first text can be segmented into sentences according to predetermined characters, such as periods ".", semicolons ";", and newline characters "\". On the other hand, the target structural information is segmented, with each line representing a structure, corresponding to a title and its hierarchical identifier. This structure can also be called a structural string, which can be segmented by newline characters. For a single structural string, its Markdown title prefixes such as "#" and "###" can be removed to obtain the corresponding single title. The similarity of this single title with each sentence in the first text is calculated sequentially, and the sentence with the highest similarity is used for structural replacement.
[0192] Various reasonable methods can be used to calculate the specific similarity.
[0193] In one embodiment, for the first title, the similarity scores between it and each clause of the first text are determined sequentially. The sentence with the highest similarity score (denoted as the first target sentence) is selected, and the first structure string (including the title and its hierarchical identifier) is used to replace the first target sentence or inserted before / after it. Then, for the second title, the similarity scores between it and each sentence after the first target sentence are calculated sequentially. The sentence with the highest similarity score (denoted as the second target sentence) is selected, and the second structure string (with the title prefix) is used to replace the second target sentence or inserted before / after it. This process is repeated until all structure strings are mapped back to the first text.
[0194] Figure 4 This illustrates an example of how similarity calculation maps structural information of a target text to the first text. For example... Figure 4 As shown, on the one hand, the original content of the first text is segmented into m sentences, denoted as sentence 1, sentence 2, ..., sentence m. On the other hand, the target structural information is segmented (since the target structural information describes a single structure in a single line, it can be segmented by line), resulting in n structural strings (containing titles and their hierarchical information), denoted as structure 1, structure 2, ..., structure n. First, structure 1 is compared with the m sentences for similarity. If sentence 2 is found to have the highest similarity with structure 1, then sentence 2 can be used as the target sentence corresponding to structure 2, and sentence 2 can be replaced with structure 2. Then, starting from sentence 3, the similarity between structure 2 and each (e.g., m-2) sentence is checked until the corresponding target sentence is detected, and then replaced.
[0195] Similarity calculation methods include, for example, edit distance percentage, longest common substring, and scoring using a pre-defined model (such as the Rerank model). Edit distance is the minimum number of edit operations required to transform one string into the other, quantifying the degree of difference between the two strings. Edit distance percentage measures the degree of difference between the two strings. As a concrete example, for two strings S1 and S2, the similarity determined by the edit distance percentage is E(S1, S2) / max(S1, S2), where the edit distance E(S1, S2) is the number of characters that need to be edited to transform S1 into S2. Editing operations include insertion, modification, and deletion. For example, insertion and deletion increase the number of edited characters by 1, modification increases it by 2, and so on. The longest common substring problem finds the longest consecutive identical substring between two strings. Assuming two strings of length m and n, and the length of the longest common substring is L, the similarity determined by the longest common substring is L / max(m, n). The rerank model, or reordering model, ranks sentences most relevant to a structural string to determine the matching sentences. The rerank model can be pre-trained in a supervised manner using multiple training samples. A single training sample can include two strings and their similarity labels, or a target string and multiple candidate strings, along with similarity ranking labels between them.
[0196] Various similarity calculation methods can be selected, either by choosing one method or by combining them according to predetermined rules. For example, the predetermined rules are: priority is given to using the edit distance ratio to determine the similarity between the structural string and the sentence; when the title text length is less than or equal to 4 characters (e.g., 2 Chinese characters), the longest common substring length ratio is used as an auxiliary criterion to determine the similarity between the structural string and the sentence; if the system integrates a Rerank model as an auxiliary similarity judgment method, the Rerank model's judgment score weight is not less than 0.7, otherwise the edit distance ratio is used as the sole similarity determination method.
[0197] When replacing sentences with structure strings, the longest common substring of the original sentence can be replaced with the structure string. The remaining part of the original sentence forms a new sentence, which is then inserted at the beginning of the next sentence sequence to be matched. For example, suppose a structure string is "##General Rules". After similarity calculation, it matches a sentence in the first text, "General Rules First Article aaabbbccc". In this case, the longest common substring "General Rules" in the first text is replaced with "##General Rules", and the subsequent sentence "First Article aaabbbccc" will be used for similarity detection with the next structure string.
[0198] Optionally, the similarity detection process described above can be computed in a distributed manner to improve detection efficiency. With the advancement of computer technology, the cost of similarity detection may be far lower than the cost of processing the entire first text using a large language model.
[0199] In another embodiment, for a single structure string, detection can stop when a sentence with a similarity exceeding a predetermined threshold (e.g., 0.95) is detected, and the corresponding sentence is replaced as the target sentence. Then, the similarity between subsequent sentences and the next structure string is detected to avoid missing structural information when two structural information pieces are close together, and so on. In this way, it is not necessary to detect the similarity between all structure strings and every sentence in the first text, further reducing the amount of data processing.
[0200] In other embodiments, similarity can be detected in other ways, which will not be elaborated here. The following is a simple example of the structured text from the previous example:
[0201] # *** Insurance Co., Ltd.
[0202] # Medical Insurance Terms and Conditions for Special Drugs for Malignant Tumors
[0203] ## General Principles
[0204] ...
[0205] ### First Article
[0206] ...
[0207] ### Second Article
[0208] ...
[0209] ### Article 3
[0210] ...
[0211] ### Article 4
[0212] ...
[0213] ### Article 5
[0214] ...
[0215] ## Insurance Liability
[0216] ……
[0217] Here, due to space limitations, a simplified example is given, where “…” represents other text content in the first text.
[0218] In order to clarify the technical concept of this specification as a whole, Figure 5A more detailed flowchart of the implementation process for structuring long texts is provided. Figure 5 The text shown is long text, meaning text that exceeds the processing length window of the large language model. This text can be in various formats. Figure 5 As shown, the original text is first processed into preliminary document slicing, with the length of each segment not exceeding the length window of the large language model, resulting in text segment 1, text segment 2, ..., text segment n. Then, the document structure extraction large model (... Figure 5 The structure of each segment is extracted (denoted as LLM, as mentioned earlier), resulting in the structure of segment 1, judgment 2, and so on, up to segment n. The structural information of each segment is then merged to obtain a rough full-text structure, i.e., the initial structural information. Next, a document structure optimization model is used to align the structural levels, resulting in a refined full-text structure, i.e., the target structural information. Finally, the original text content and the refined full-text structure are combined to reconstruct the text structure, yielding the structured text content.
[0219] Figure 6 The screenshot shows a portion of a structured document. Sensitive information related to specific insurance companies is obscured by gray blocks. Figure 6 As shown, after structuring, each heading is displayed in Markdown document syntax format, and merged with statements that do not match other structure strings to obtain text content with precise structure. Specifically, in... Figure 6 The document includes a first-level heading structure (marked with "#"), a second-level heading structure (marked with "##"), and in practice, even lower-level headings can be added, which are not shown here. This results in a document that corresponds to... Figure 1 In step 1, the structured text is sliced according to various slicing methods, such as structured slicing with logical structure slicing rules including at least one of the following: title, chapter, table, list, etc. In the case of enhancing the generation model by vectorizing the slices and adding them to the vector database, the retrieval is performed by vectors.
[0220] To recap the above process, in order to generate data suitable for retrieval enhancement, the data can be structured using Markdown format. For a specific text, such as the first text, the text structure is first extracted to obtain initial structural information in Markdown format. Then, the initial structural information is aligned hierarchically to obtain the corrected target structural information. Finally, the target structural information is mapped to the first text to obtain structured text. This structured text can be processed according to one or more corresponding slices as a data reserve for retrieval enhancement. This method of data processing, through the decoupling of the three stages of coarse structure extraction, structure optimization, and structure restoration, can optimize the speed and effectiveness of text structuring, ensuring accuracy while reducing cost and latency. Simultaneously, it can solve the problems of text structure loss (preventing cross-segment hierarchical breaks caused by slicing), incomplete text structure, and text structure errors that exist during document transcription. The technical solution provided in this specification is applicable to text of any length, especially long text. Furthermore, in the long text processing, the structure optimization stage deals with the full-text structural information, not the full-text content, which can reduce the risk of illusion.
[0221] According to another embodiment, a data processing apparatus suitable for generating search enhancements is also provided, which can be installed in a computer, device, or server with a certain computing power. This apparatus can be used to convert text used for search enhancements into Markdown format text.
[0222] Figure 7 A data processing apparatus 700 suitable for retrieval enhancement generation according to one embodiment is shown. For example... Figure 7 As shown, the data processing apparatus 700 suitable for retrieval enhancement generation may include: an acquisition unit 701 configured to acquire a first text to be processed; an extraction unit 702 configured to extract the text structure of the first text using a large structure extraction model to obtain initial structure information, wherein the initial structure information includes heading levels in Markdown format; an alignment unit 703 configured to align the initial structure information hierarchically using an alignment model to obtain target structure information corrected for the initial structure information; and a mapping unit 704 configured to map the target structure information onto the first text to obtain a first structured text in Markdown format, used for text slicing according to a predetermined method.
[0223] It is worth noting that, Figure 7 The device 700 shown is Figure 3 The method embodiment shown corresponds to this, therefore, Figure 3 The corresponding descriptions in the method embodiments can also be applied to... Figure 7 The device 700 shown will not be described in detail here.
[0224] According to another embodiment, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform a combination Figure 3 The methods described above.
[0225] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, it implements a combination... Figure 3 The methods described above.
[0226] Those skilled in the art will recognize that the functions described in the embodiments of this specification in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.
[0227] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the technical concept in this specification. It should be understood that the above description is only a specific embodiment of the technical concept in this specification and is not intended to limit the scope of protection of the technical concept in this specification. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solutions of the embodiments in this specification should be included within the scope of protection of the technical concept in this specification.
Claims
1. A data processing method suitable for retrieval enhancement generation, used to convert text used as retrieval enhancement into Markdown format text, the method comprising: Get the first text to be processed; The text structure of the first text is extracted using a large structure extraction model to obtain initial structure information, which includes the title and the hierarchical identifier in Markdown format. By aligning the hierarchical relationships of the initial structural information using the large alignment model, the target structural information after correction based on the initial structural information is obtained. The target structure information is mapped onto the first text to obtain a first structured text in Markdown format, which is used for text slicing according to a predetermined method.
2. The method as described in claim 1, wherein, The structure extraction model has a length window, which describes the maximum length of the text processed by the structure extraction model in a single operation. When the number of bytes in the first text is greater than the length window, the step of extracting the text structure of the first text using a large structure extraction model to obtain initial structure information includes: The first text is truncated into multiple segments according to the length window, and the number of bytes in a single segment is no greater than the length window. The structural extraction model described above is used to extract the structural information of each sub-document corresponding to each segment; The initial structure information of the first text is obtained by merging the structure information of each sub-document.
3. The method as described in claim 1, wherein, The hierarchical alignment includes aligning titles that meet predetermined rules to the same level. The predetermined rules include at least one of the following: semantic similarity greater than a predetermined similarity threshold, containing the same keywords, or containing the same sentence structure.
4. The method of claim 1, wherein, The structure extraction model and the alignment model are the same large language model.
5. The method of claim 1, wherein, The target structure information includes multiple structures, each structure including a single title and a corresponding structure hierarchy identifier. Mapping the target structure information onto the first text to obtain the first structured text in Markdown format includes: The first text is segmented into sentences to obtain individual sentences; Each title in the multiple structures is used sequentially to match each sentence to obtain each target sentence that matches each title. Then, each target sentence that matches each title is replaced with the corresponding structure to obtain the first structured text.
6. The method of claim 5, wherein, The plurality of structures includes a first structure, which corresponds to a first title. The step of sequentially matching each title in the plurality of structures with each sentence to obtain target sentences that match each title includes: If the first structure is the first structure, starting from the first sentence, each sentence is matched with the first title in turn until the first target sentence that matches is obtained; If the first structure is not the first structure, starting from the sentence following the target sentence that matches the previous structure, each sentence is matched with the first title sequentially until the first target sentence that matches is obtained.
7. The method of claim 5 or 6, wherein, The matching method between a single title and each sentence is as follows: The similarity between the title string and each sentence string corresponding to the single title is calculated using at least one of the following: edit distance ratio, longest common substring, and predefined model score; The sentence with the highest similarity is identified as the single target sentence that matches the single title.
8. The method of claim 7, wherein, In the case where a single title matches a single sentence, the longest common substring in the single sentence that matches the single title is obtained as the target sentence, and the subsequent characters of the longest common substring are used as the next sentence to match the next title.
9. The method of claim 1, wherein, The predetermined method is a structured slicing based on at least one of the following logical structure slicing rules: title, chapter, table, list.
10. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-9.