Document segmentation method and device, electronic equipment and readable storage medium

Abstract

The invention discloses a document segmentation method and device, electronic equipment and a readable storage medium, and the method comprises the steps that a semantic structure unit in a to-be-segmented document is recognized, and the semantic structure unit comprises at least one of a causal relationship chain, a process step chain and an entity-action-result mode; the to-be-segmented document is segmented on the basis of the semantic structure unit, at least one first candidate block is obtained, and the at least one first candidate block comprises the first candidate block corresponding to the semantic structure unit; for each first candidate block, under the condition that the length of the first candidate block is greater than a first threshold value, segmenting the first candidate block according to first semantics to obtain at least one target block; for each target block, the target block is associated with target metadata corresponding to the target block, and the target metadata comprises the document position where the target block is located and the semantic structure unit to which the target block belongs.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document segmentation method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Retrieval-Augmented Generation (RAG) systems are systems that combine information retrieval and text generation technologies. In RAG systems, the quality of text block segmentation directly affects the accuracy of knowledge retrieval and the reliability of generated content.

[0003] In related technologies, RAG systems typically employ a fixed-size chunking strategy to segment documents. This involves mechanically dividing the document into chunks based on a fixed number of characters (e.g., 500 characters) or tokens (e.g., 256 tokens), and selectively setting overlapping regions (e.g., 50-character overlap) to reduce the loss of boundary information. However, fixed-size chunking can compromise semantic integrity, potentially splitting the document at the sentence, paragraph, or even word level. This fragmentation of semantic units affects the Large Language Model's (LLM) understanding of the context, thereby impacting the subsequent inference and generation performance of the RAG system. Summary of the Invention

[0004] This application discloses a document segmentation method, apparatus, electronic device, and readable storage medium, which can improve the reasoning and generation performance of the RAG system.

[0005] To solve the above problems, this application adopts the following technical solution: In a first aspect, embodiments of this application disclose a document segmentation method, comprising: identifying semantic structure units in a document to be segmented, wherein the semantic structure unit includes at least one of a causal relationship chain, a process step chain, and an entity-action-result pattern; segmenting the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment, wherein the at least one first candidate segment includes a first candidate segment corresponding to the semantic structure unit; for each first candidate segment, if the length of the first candidate segment is greater than a first threshold, segmenting the first candidate segment according to a first semantic to obtain at least one target segment; for each target segment, associating the target segment with target metadata corresponding to the target segment, wherein the target metadata includes the document location of the target segment and the semantic structure unit to which the target segment belongs.

[0006] Secondly, embodiments of this application disclose a document segmentation apparatus, comprising: an identification module, configured to identify semantic structure units in a document to be segmented, wherein the semantic structure unit includes at least one of a causal relationship chain, a process step chain, and an entity-action-result pattern; an obtaining module, configured to segment the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment, wherein the at least one first candidate segment includes a first candidate segment corresponding to the semantic structure unit; the obtaining module is further configured to, for each first candidate segment, if the length of the first candidate segment is greater than a first threshold, segment the first candidate segment according to a first semantic to obtain at least one target segment; and an association module, configured to, for each target segment, associate the target segment with target metadata corresponding to the target segment, wherein the target metadata includes the document location of the target segment and the semantic structure unit to which the target segment belongs.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of the method described in the first aspect.

[0010] The technical solution adopted in this application can achieve the following beneficial effects: This application provides a document segmentation method. By identifying semantic structural units in the document to be segmented, the semantic structural units include at least one of causal relationship chains, process step chains, and entity-action-result patterns. Based on the semantic structural units in the document to be segmented, the document is segmented to obtain at least one first candidate segment. The at least one first candidate segment includes a first candidate segment corresponding to a semantic structural unit in the document to be segmented. For each first candidate segment, if the length of the first candidate segment is greater than a first threshold, the first candidate segment is segmented according to a first semantic to obtain at least one target segment. For each target segment, the target segment and the target metadata corresponding to the target segment are associated. The target metadata includes the document location of the target segment and the semantic structural unit to which the target segment belongs. In this application, by associating target segments with their corresponding target metadata, when searching through the RAG system, the target metadata of the retrieved target segments can be used to associate them with other target segments belonging to the same semantic structure unit. By inputting all target segments belonging to the same semantic structure unit into the LLM for integration, it is possible to ensure that complete semantic structure units are input into the LLM, improve the LLM's understanding of the context, and thus improve the reasoning and generation performance of the RAG system. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating a document segmentation method disclosed in an embodiment of this application; Figure 2 This is a schematic diagram of the operation of a RAG system disclosed in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a document segmentation device disclosed in an embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device disclosed in an embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0013] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the electrically connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0014] The document segmentation method, apparatus, electronic device, and readable storage medium disclosed in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0015] This application discloses a document segmentation method. Figure 1 This is a flowchart illustrating a document segmentation method disclosed in an embodiment of this application. Figure 1 As shown, the method includes the following steps: S120. Identify semantic structure units in the document to be segmented, wherein the semantic structure units include at least one of causal relationship chain, process step chain, and entity-action-result (EAR) pattern.

[0016] It should be noted that the document segmentation method in this application is used in the RAG system.

[0017] For example, the causal chain defined in this application can be “[Cause 1]→[Result 1 / Cause 2]→[Result 2]→…→[Final Result]”. In the causal chain identification, firstly, the standard causal identifier is constructed to locate causal markers (such as “because”, “due to”, “leading to”, “therefore”, “so”, etc.) to obtain possible cause-effect sentences. Then, dependency parsing combined with a large model is used to identify causal relationships within sentences, and further, cross-sentence causal chains are constructed to output the causal chain and the relationship between the linked sentences.

[0018] In process step chain recognition, the "process chain" in the document to be segmented, which is a sequential progression of operations or events, is identified in the form of "[Step 1] → [Step 2] → [Step 3] → … → [Result / Final State]". First, process markers (such as "1.", "First step", "First", "Then", "Finally") or conditional statements (such as "When…", "If…then") are identified. Then, based on the large model, a flowchart or step chain is further constructed, and the process chain and sentence relationships are output.

[0019] In Entity-Action-Result (EAR) pattern recognition, the common semantic structure of technical documents is identified as: "Main Entity + Action / Operation + Result State". Specifically, semantic role labels are used to extract Agent, Action, and Patient / Result, recognizing state changes such as "from X to Y", "increasing / decreasing X", or effect descriptions such as "achieving effect Z" or "achieving goal W", thus completing the overall EAR content extraction. It should be noted that: Entity: The participant or object of the event, usually a noun or noun phrase, such as person, organization, place, item, etc.; Action: The core behavior performed or experienced by the entity, usually a verb or verb phrase, which is the driving force of the event; Result: The state change, output, or consequence caused by the action, usually a noun phrase or descriptive clause.

[0020] S140. The document to be segmented is segmented based on the semantic structure unit to obtain at least one first candidate segment, wherein the at least one first candidate segment includes a first candidate segment corresponding to the semantic structure unit.

[0021] In other words, this application places the document content corresponding to a semantic structural unit in the document to be segmented into a first candidate segmentation block, maintaining the integrity of semantic structural units such as causal relationship chains, process step chains, and entity-action-result patterns, and avoiding their splitting at the segmentation block boundaries. For example, the first candidate segmentation block A includes the content corresponding to causal relationship chain A in the document to be segmented, the first candidate segmentation block B includes the content corresponding to causal relationship chain B in the document to be segmented, the first candidate segmentation block C includes the content corresponding to process step chain A in the document to be segmented, and the first candidate segmentation block D includes the content corresponding to entity-action-result pattern A in the document to be segmented.

[0022] S160. For each of the first candidate segments, if the length of the first candidate segment is greater than the first threshold, the first candidate segment is segmented according to the first semantic to obtain at least one target segment.

[0023] For example, the first threshold can be 200 characters, 500 characters, etc. It should be noted that this application does not limit the specific value of the first threshold, and the specific value of the first threshold can be set according to actual needs. In addition, the first semantic and the second semantic in this application are only used to distinguish different semantic segmentation granularities.

[0024] If the length of the first candidate segment is greater than the first threshold and the first candidate segment is semantically separable, the first candidate segment is segmented according to semantics to obtain multiple target segments. If the length of the first candidate segment is less than or equal to the first threshold, the first candidate segment is treated as a single target segment.

[0025] S180. For each target segment, associate the target segment with the target metadata corresponding to the target segment, wherein the target metadata includes the document location of the target segment and the semantic structure unit to which the target segment belongs.

[0026] In this application, the document location of the target segment may include document ID (identifier), chapter, paragraph, page number, and the semantic structure unit to which the target segment belongs may include causal relationship chain, process step chain, or EAR pattern.

[0027] In this application, by associating target segments with their corresponding target metadata, when searching through the RAG system, other target segments belonging to the same semantic structure unit can be associated with the target metadata corresponding to the retrieved target segment. For example, the target segment B, which belongs to the same document A chapter A and process step chain A as target segment A, can be associated with the target metadata corresponding to target segment A. Then, both target segment A and target segment B are input into the LLM for integration, which ensures that complete semantic structure units are input into the LLM, improves the LLM's understanding of the context, and thus improves the reasoning and generation effects of the RAG system.

[0028] This application provides a document segmentation method. By identifying semantic structural units in the document to be segmented, the semantic structural units include at least one of causal relationship chains, process step chains, and entity-action-result patterns. Based on the semantic structural units in the document to be segmented, the document is segmented to obtain at least one first candidate segment. The at least one first candidate segment includes a first candidate segment corresponding to a semantic structural unit in the document to be segmented. For each first candidate segment, if the length of the first candidate segment is greater than a first threshold, the first candidate segment is segmented according to a first semantic to obtain at least one target segment. For each target segment, the target segment and the target metadata corresponding to the target segment are associated. The target metadata includes the document location of the target segment and the semantic structural unit to which the target segment belongs. In this application, by associating target segments with their corresponding target metadata, when searching through the RAG system, the target metadata of the retrieved target segments can be used to associate them with other target segments belonging to the same semantic structure unit. By inputting all target segments belonging to the same semantic structure unit into the LLM for integration, it is possible to ensure that complete semantic structure units are input into the LLM, improve the LLM's understanding of the context, and thus improve the reasoning and generation performance of the RAG system.

[0029] In this embodiment of the application, the step of segmenting the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment may include: segmenting the document to be segmented based on paragraphs to obtain at least one second candidate segment; for each second candidate segment, segmenting the second candidate segment according to a second semantic to obtain at least one third candidate segment, wherein the granularity of the second semantic is greater than the granularity of the first semantic; and determining at least one first candidate segment based on the semantic structure unit and at least one third candidate segment.

[0030] In this application, the adaptive segmentation generation module performs the following process: First, the document to be segmented is segmented based on its paragraphs, with each paragraph serving as a second candidate segment. Then, the second candidate segments are semantically segmented to obtain at least one third candidate segment, effectively breaking down the document. After obtaining at least one third candidate segment, the segment boundaries are verified based on the semantic structural units within the document. If a semantic structural unit is found to be truncated, a merging operation is performed, incorporating the entire semantic structural unit into the same segment. This means merging the third candidate segments belonging to the same semantic structural unit to form a first candidate segment. In this application, the adaptive segmentation generation module generates multi-level segments based on the identified semantic structural units, maintaining the integrity of semantic structural units such as causal chains, process step chains, and entity-action-result patterns, and preventing them from being split at segment boundaries.

[0031] In this application, to support retrieval needs at different granularities, a three-level segmentation hierarchy is constructed. The segmentation system includes a three-level structure: coarse-grained (chapter level), medium-grained (paragraph group / semantic unit), and fine-grained (sentence group), as follows: document ├─ Coarse-grained blocks (chapter level) │├─ Medium-grained blocks (paragraph groups / semantic units) ││├─ Fine-grained blocks (sentence groups) ││└─ Fine-grained blocks (sentence groups) │└─ Medium-sized blocks └─ Coarse-grained blocks In one implementation, after obtaining at least one target slice, the process may further include: updating the target slices based on the granularity level corresponding to each target slice, the entities contained in each target slice, and the semantics of each target slice.

[0032] For example, multiple fine-grained target segments with semantic relevance greater than the merging threshold, shared entities, or constituting a complete semantic chain can be merged into a new target segment to update the target segment. This can enhance semantic coherence, improve retrieval hit and generation effects, improve retrieval efficiency, and flexibly switch between different granularities to adapt to query complexity.

[0033] In this embodiment of the application, the target metadata may further include at least one of granularity level, entity metadata, relation metadata, topic and keyword metadata, inter-segment link metadata, and quality metadata.

[0034] In other words, the target metadata corresponding to the target segment can include the document location of the target segment (document ID, chapter, paragraph, page number), the semantic structure unit to which the target segment belongs (causal relationship chain / process step chain / EAR pattern), the granularity level of the target segment (coarse / medium / fine granularity), the entity metadata of the target segment (named entity, type, sentence ID), the relational metadata of the target segment (entity-entity relationship, action relationship, causal or process relationship, confidence level), the topic and keyword metadata of the target segment, the inter-segment link metadata of the target segment (parent-child link, sibling association, reference / referenced), and the quality metadata of the target segment (such as semantic integrity score, segment length rationality, relevance index, etc.). By associating the target segment with the aforementioned target metadata, complex reasoning queries can be supported, enabling the segment to effectively support complex query requirements such as multi-hop reasoning and cross-paragraph reasoning.

[0035] In this application, the metadata enhancement module constructs rich structured metadata for each generated segment to achieve accurate retrieval and multi-dimensional filtering based on semantic relationships. For example, metadata generation can be implemented through the following steps: determining the granularity level of the segment (coarse / medium / fine granularity); determining the document location of the segment (document ID, chapter, paragraph, page number); extracting entity information from the segment and generating entity metadata (named entity, type, sentence ID); extracting relational information from the segment and generating relational metadata (entity-entity relationship, action relationship, causal or process relationship, confidence level); identifying the semantic structural units involved in the segment (causal relationship chain, process step chain, EAR pattern, etc.) and determining semantic structure metadata (belonging to the causal relationship chain / process step chain / EAR pattern); extracting the topic and keywords of the segment and generating topic and keyword metadata; establishing link relationships between segments and generating inter-segment link metadata (parent-child link, sibling association, reference / being referenced); obtaining segment quality scores and generating quality metadata (such as semantic integrity score, segment length reasonableness, relevance index, etc.). It should be noted that the initial quality score of the chunk can be determined based on the granularity and length of the chunk, and then the chunk quality score can be adjusted based on user feedback on the generated search results. Fine-grained semantic integrity scores are low, while coarse-grained semantic integrity scores are high.

[0036] During the retrieval phase, the metadata supports multiple advanced retrieval methods, combining these methods based on a large model to meet various query scenarios. Specific retrieval methods include the following: Entity-based retrieval (e.g., "retrieve chunks that mention entity A and are of type ..."); Semantic structure-based retrieval (e.g., "querying chunks containing complete causal chains or process step chains"); Search based on link relationships (e.g., "search for slices that are associated with / referenced by slice X").

[0037] Through this metadata enhancement module, the chunks not only store text content, but also have semantic tags and structured background, thereby improving retrieval accuracy, supporting complex query scenarios, and enhancing the interpretability of the generated results.

[0038] In one implementation, identifying semantic structural units in the document to be segmented may include: obtaining attribute data of the document to be segmented, wherein the attribute data includes at least one of hierarchical structure, structural meta-information, entity information, and semantic relationships between sentences; and identifying semantic structural units in the document to be segmented based on the attribute data.

[0039] In this application, before identifying semantic structural units in the document to be segmented, the document to be segmented is preprocessed by the document preprocessing module to obtain the attribute data of the document to be segmented. Then, the semantic structure recognition module identifies the semantic structural units of the document to be segmented based on the attribute data of the document to be segmented, which can improve the accuracy of the identified semantic structural units.

[0040] The document preprocessing module acquires attribute data from the document to be segmented, which may include: First, performing structured parsing on the input documents of various formats (i.e., the documents to be segmented) (such as PDF, Word, HTML, Markdown, TXT, etc.), identifying and extracting their hierarchical structure such as chapters, titles, paragraphs, tables, charts, and lists, while retaining structural meta-information (such as chapter IDs, page numbers, and title text); Second, using a fine-tuned large model combined with specific prompts and few-shots to perform NLP (Natural Language Processing) on ​​the extracted text content. Natural Language Processing (NLP) applications segment, word segmentation, and part-of-speech tagging to mark the boundary positions and sentence IDs of each sentence. Then, at the sentence level, entity recognition is performed to identify entities such as names, organizations, locations, times, values, and technical terms, and to label their type (name, organization, location, time, value, technical term), location in the document, and the sentence ID to which they belong. Simultaneously, an entity-sentence inverted index is built. Finally, preliminary relation extraction is performed: identifying the subject-verb-object structure within sentences ("entity A-action-entity B"), action verbs, and modifying relationships, and identifying conjunctions between sentences (such as "therefore," "because," "firstly," "further") to mine preliminary semantic relationships for subsequent segmentation and semantic structure analysis.

[0041] In this application, the aforementioned high-level semantic structural units (causal relationship chain, process step chain, EAR pattern) can be identified and extracted from sentences, word segments, entities and preliminary semantic relationships extracted from the document preprocessing module, thereby providing a semantically complete basis for block boundary decisions.

[0042] In this embodiment of the application, after associating the target segment and the target metadata corresponding to the target segment, the method may further include: receiving a retrieval instruction; determining at least one fourth candidate segment from a knowledge base storing the target segment based on the retrieval instruction, wherein the fourth candidate segment is a target segment in the knowledge base whose relevance score to the retrieval instruction is greater than a second threshold; determining at least one fifth candidate segment from the knowledge base based on the target metadata corresponding to at least one fourth candidate segment, wherein the fifth candidate segment is a target segment in the knowledge base that belongs to the same semantic structure unit as the fourth candidate segment; and obtaining the retrieval result corresponding to the retrieval instruction by inputting the retrieval instruction, the at least one fourth candidate segment, and the at least one fifth candidate segment into a large language model.

[0043] For example, the received search instruction can be a question entered by the user, and the relevance score between the search instruction and the target segment can be determined based on the vector corresponding to the search instruction and the vector corresponding to the target segment.

[0044] In this application, when a fourth candidate segment is obtained by retrieving the target segment knowledge base based on the retrieval instruction, the target metadata corresponding to the fourth candidate segment is associated with other target segments belonging to the same semantic structure unit (i.e., the fifth candidate segment). All target segments belonging to the same semantic structure unit are input into the LLM for integration, which can ensure that the complete semantic structure unit is input into the LLM, improve the LLM's understanding of the context, and thus improve the reasoning and generation effect of the RAG system.

[0045] The indexing and retrieval optimization module achieves efficient retrieval of chunks and optimal utilization of the context window by constructing a hybrid index structure (i.e., vector index + relation index) for the (RAG) system. During the indexing phase, each chunk in the knowledge base not only generates a vector representation for semantic similarity retrieval (i.e., semantic relevance analysis), but also establishes a relation index to associate semantic chains and links between chunks, and extracts chunks that include entities identical to those in the retrieval command.

[0046] The retrieval phase models the block selection as an optimization problem of "maximizing semantic coverage under the constraint of context window length". The algorithm structure is as follows: Objective function: maximize Σ(relevance_score_i × chunk_i) Constraint: Σ(length_i × chunk_i) ≤ context_window_limit Here, `relevance_score_i` represents the relevance score between chunk i and the query, `length_i` represents the length of chunk i (number of tokens), `chunk_i` is a binary variable where 1 indicates selection of the chunk and 0 indicates non-selection, and `context_window_limit` represents the context window limit. By optimizing the utilization of the context window, the optimal combination of chunks is selected within a limited context length to maximize semantic coverage.

[0047] In this application, after the indexing and retrieval optimization module receives the retrieval instruction and determines multiple candidate chunks to be input into the LLM through a hybrid index structure, it first calculates the relevance score between each candidate chunk and the retrieval instruction, as well as the length of each candidate chunk. It then checks whether the chunks form a complete semantic chain (such as a causal chain or a process step chain), prioritizing the retention of chunks with complete semantic chains. Next, dynamic programming is performed on the remaining candidate chunks based on the aforementioned objective function and constraints to determine the chunks to be input into the LLM. Finally, post-processing is performed to ensure that the selected chunks are arranged according to the original document order, and adjacent chunks belonging to the same semantic structure unit are automatically merged to avoid semantic fragmentation. Furthermore, the indexing and retrieval optimization module includes a chunk redundancy detection and deduplication mechanism to avoid selecting chunks with highly repetitive content, thereby improving context window utilization, reducing interfering content, and enhancing the quality of the generated LLM integration results. The context window-optimized chunk selection and combination algorithm models chunk selection as a constrained optimization problem, maximizing semantic coverage under context length constraints, and improving context utilization and generation accuracy through dynamic programming and a semantic chain priority strategy.

[0048] In one implementation, the quality assessment and optimization module establishes a multi-dimensional evaluation system for slicing quality and combines it with feedback from actual retrieval and generation results to achieve automatic optimization and continuous improvement of the document slicing strategy. After obtaining the retrieval results output by the large language model corresponding to the retrieval instruction, the module may further include: obtaining the evaluation results corresponding to the retrieval results, wherein the evaluation results include at least one of semantic integrity, retrieval recall, generation quality of retrieval results, and context utilization; and adjusting the slicing strategy parameters based on the evaluation results.

[0049] In this application, a comprehensive quality score can be determined based on the semantic completeness, retrieval recall, retrieval result generation quality, and context utilization of the search results. For example, semantic completeness can be determined based on the search results output by the LLM and the chunks input to the LLM; retrieval recall can be determined based on the relevance score between the search results output by the LLM and the search command; retrieval result generation quality can be determined based on user feedback ratings of the search results; and context utilization can be determined based on the search results output by the LLM and the chunks input to the LLM.

[0050] The segmentation strategy parameters may include at least one of semantic weight, maximum segment size, segment size, and merging threshold. It should be noted that the merging threshold here is the same as the merging threshold at the target segment updated above. In addition, the quality of the search results generated in the evaluation results is the LLM generation quality.

[0051] In the evaluation of the RAG system, the quality assessment optimization module dynamically adjusts the chunking strategy parameters based on the evaluation results corresponding to the retrieval results using a Bayesian adaptive optimization algorithm. For example, when semantic integrity is low (below the third threshold), the semantic_priority_weight is increased or the max_chunk_size is increased; when the retrieval recall is low (below the fourth threshold), the chunk_size may be decreased; when the context utilization is low (below the fifth threshold), the merging threshold is increased; if the quality of the generated retrieval results is low (below the sixth threshold) but other indicators are normal, it is determined to be an LLM model problem, and the chunking parameters are not adjusted.

[0052] Furthermore, the RAG system using the proposed solution also supports online learning and continuous optimization. During operation, the system continuously collects user feedback and retrieves / generates data, triggers strategy updates periodically or in real time, and compares the actual effects of different slicing strategies through A / B testing, thereby achieving closed-loop improvement and long-term enhancement of the slicing strategy.

[0053] like Figure 2As shown, the RAG system of this application includes a document preprocessing module, a semantic structure recognition module, an adaptive chunking generation module, a metadata enhancement module, an indexing and retrieval optimization module, and a quality assessment optimization module. The document to be segmented is input into the document preprocessing module, which obtains the attribute data of the document to be segmented through preprocessing and outputs structured document data to the semantic structure recognition module. The semantic structure recognition module identifies and outputs semantic structure units such as causal relationship chains, process step chains, and EAR patterns to the adaptive chunking generation module. The adaptive chunking generation module generates multi-level chunks (coarse, medium, and fine) based on the identified semantic structure units. The metadata enhancement module adds corresponding structured metadata to the chunks and constructs a hybrid index structure (i.e., vector index + relation index), which is stored in the knowledge base. When the indexing and retrieval optimization module receives the retrieval instruction input by the user, it retrieves relevant chunks, optimizes the context window, constructs a prompt, selects the chunk and query combination, generates an answer, and returns it to the user. Then, the quality assessment optimization module collects feedback on the answer and user satisfaction, generates an evaluation result, adjusts the chunking strategy parameters based on the evaluation result, and regenerates the chunks.

[0054] The document segmentation method provided in this application can be executed by a document segmentation device. This application uses a document segmentation device executing the document segmentation method as an example to illustrate the document segmentation device provided in this application.

[0055] Figure 3 This is a schematic diagram of the structure of a document segmentation device disclosed in an embodiment of this application. Figure 3 As shown, the document segmentation device 300 includes: an identification module 310, an acquisition module 320, and an association module 330.

[0056] In this application, the identification module 310 is used to identify semantic structure units in the document to be segmented, wherein the semantic structure unit includes at least one of causal relationship chain, process step chain, and entity-action-result pattern; the obtaining module 320 is used to segment the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment, wherein the at least one first candidate segment includes a first candidate segment corresponding to the semantic structure unit; the obtaining module 320 is further used to segment each first candidate segment according to a first semantic if the length of the first candidate segment is greater than a first threshold to obtain at least one target segment; the association module 330 is used to associate each target segment with the target metadata corresponding to the target segment, wherein the target metadata includes the document location of the target segment and the semantic structure unit to which the target segment belongs.

[0057] In one implementation, the obtaining module 320 segments the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment, including: segmenting the document to be segmented based on paragraphs to obtain at least one second candidate segment; for each second candidate segment, segmenting the second candidate segment according to a second semantic to obtain at least one third candidate segment, wherein the granularity of the second semantic is greater than the granularity of the first semantic; and determining at least one first candidate segment based on the semantic structure unit and at least one third candidate segment.

[0058] In one implementation, the above apparatus further includes an update module, configured to update the target segments based on the granularity level corresponding to each target segment, the entity contained in each target segment, and the semantics of each target segment after obtaining at least one target segment.

[0059] In one implementation, the target metadata further includes at least one of granularity level, entity metadata, relation metadata, topic and keyword metadata, inter-segment link metadata, and quality metadata.

[0060] In one implementation, the identification module 310 identifies semantic structural units in the document to be segmented, including: acquiring attribute data of the document to be segmented, wherein the attribute data includes at least one of hierarchical structure, structural meta-information, entity information, and semantic relationships between sentences; and identifying semantic structural units in the document to be segmented based on the attribute data.

[0061] In one implementation, the above apparatus further includes: a receiving module, configured to receive a retrieval instruction after associating the target segment and the target metadata corresponding to the target segment; a determining module, configured to determine at least one fourth candidate segment from a knowledge base storing the target segment based on the retrieval instruction, wherein the fourth candidate segment is a target segment in the knowledge base whose relevance score to the retrieval instruction is greater than a second threshold; the determining module is further configured to determine at least one fifth candidate segment from the knowledge base based on the target metadata corresponding to at least one fourth candidate segment, wherein the fifth candidate segment is a target segment in the knowledge base that belongs to the same semantic structure unit as the fourth candidate segment; and the obtaining module 320 is further configured to input the retrieval instruction, the at least one fourth candidate segment, and the at least one fifth candidate segment into a large language model to obtain a retrieval result output by the large language model corresponding to the retrieval instruction.

[0062] In one implementation, the above apparatus further includes: an acquisition module, configured to acquire an evaluation result corresponding to the search result after obtaining the search result output by the large language model corresponding to the search instruction, wherein the evaluation result includes at least one of semantic completeness, search recall, search result generation quality, and context utilization; and an adjustment module, configured to adjust the segmentation strategy parameters based on the evaluation result.

[0063] The document segmentation device provided in this application embodiment can implement all the processes implemented in the document segmentation method embodiment, and will not be described again here to avoid repetition.

[0064] Optionally, such as Figure 4 As shown, this application embodiment also provides an electronic device 400, including a processor 401 and a memory 402. The memory 402 stores a program or instructions that can run on the processor 401. When the program or instructions are executed by the processor 401, they implement the various steps of the above-described document segmentation method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0065] It should be noted that the electronic devices in the embodiments of this application include mobile electronic devices and non-mobile electronic devices.

[0066] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described document segmentation method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0067] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0068] This application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to perform the steps of the document segmentation method described above.

[0069] The above embodiments of this application focus on describing the differences between the various embodiments. As long as the different optimization features between the various embodiments are not contradictory, they can be combined to form a better embodiment. For the sake of brevity, they will not be described in detail here.

[0070] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A document segmentation method, characterized in that, include: Identify semantic structural units in the document to be segmented, wherein the semantic structural units include at least one of causal relationship chain, process step chain, and entity-action-result pattern; The document to be segmented is segmented based on the semantic structure unit to obtain at least one first candidate segment, wherein the at least one first candidate segment includes a first candidate segment corresponding to the semantic structure unit; For each of the first candidate segments, if the length of the first candidate segment is greater than a first threshold, the first candidate segment is segmented according to the first semantics to obtain at least one target segment; For each target segment, the target segment and the target metadata corresponding to the target segment are associated, wherein the target metadata includes the document location of the target segment and the semantic structure unit to which the target segment belongs.

2. The method according to claim 1, characterized in that, The step of segmenting the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment includes: The document to be segmented is segmented based on its paragraphs to obtain at least one second candidate segment. For each second candidate slice, the second candidate slice is segmented according to the second semantics to obtain at least one third candidate slice, wherein the granularity of the second semantics is greater than the granularity of the first semantics; Based on the semantic structural unit and at least one of the third candidate segments, at least one first candidate segment is determined.

3. The method according to claim 1, characterized in that, After obtaining at least one target slice, the process further includes: The target segments are updated based on the granularity level corresponding to each target segment, the entities contained in each target segment, and the semantics of each target segment.

4. The method according to claim 1, characterized in that, The target metadata also includes at least one of the following: granularity level, entity metadata, relation metadata, topic and keyword metadata, inter-segment link metadata, and quality metadata.

5. The method according to claim 1, characterized in that, The process of identifying semantic structural units in the document to be segmented includes: Obtain the attribute data of the document to be segmented, wherein the attribute data includes at least one of hierarchical structure, structural meta-information, entity information, and semantic relationships between sentences; Based on the attribute data, semantic structural units in the document to be segmented are identified.

6. The method according to claim 1, characterized in that, After associating the target slice with the target metadata corresponding to the target slice, the method further includes: Receive search instructions; Based on the retrieval instruction, at least one fourth candidate segment is determined from the knowledge base storing the target segment, wherein the fourth candidate segment is a target segment in the knowledge base whose relevance score to the retrieval instruction is greater than a second threshold; Based on the target metadata corresponding to at least one of the fourth candidate segments, at least one fifth candidate segment is determined from the knowledge base, wherein the fifth candidate segment is a target segment in the knowledge base that belongs to the same semantic structure unit as the fourth candidate segment; By inputting the search command, the at least one fourth candidate segment, and the at least one fifth candidate segment into the large language model, the search results corresponding to the search command are obtained from the output of the large language model.

7. The method according to claim 6, characterized in that, After obtaining the search results corresponding to the search command output by the large language model, the method further includes: Obtain the evaluation results corresponding to the search results, wherein the evaluation results include at least one of semantic completeness, search recall, search result generation quality, and context utilization. Based on the evaluation results, the slicing strategy parameters are adjusted.

8. A document segmentation device, characterized in that, include: The identification module is used to identify semantic structure units in the document to be segmented, wherein the semantic structure unit includes at least one of causal relationship chain, process step chain, and entity-action-result pattern; The module is configured to segment the document to be segmented based on the semantic structure unit to obtain at least one first candidate segment, wherein the at least one first candidate segment includes a first candidate segment corresponding to the semantic structure unit; The obtaining module is further configured to, for each first candidate segment, if the length of the first candidate segment is greater than a first threshold, segment the first candidate segment according to a first semantic to obtain at least one target segment; The association module is used to associate each target segment with the target metadata corresponding to the target segment, wherein the target metadata includes the document location of the target segment and the semantic structure unit to which the target segment belongs.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the document segmentation method as described in any one of claims 1-7.

10. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the document segmentation method as described in any one of claims 1-7.

11. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions that, when executed by a computer, cause the computer to perform the steps of the document segmentation method as described in any one of claims 1-7.

Images