Semantic enhancement adaptive partitioning method and system for natural resource large model question and answer

Abstract

The application provides a natural resource large model question and answer semantic enhancement adaptive segmentation method and system, aiming to solve the problems of difficult term boundary recognition and semantic integrity destruction. The method integrates three core technologies: topic-aware coarse-grained paragraph division, adaptive sliding window topic level division, and context-adaptive text segmentation with embedding awareness. The system first analyzes the structure of long natural resource text, identifies the title and topic level, and aligns the associated content; extracts paragraphs according to the topic-aware strategy, and subdivides them into sentence sets according to the grammar rules; and divides the sentences into window sentence block groups using an improved sliding window mechanism. The core is to introduce a dynamic aggregation threshold mechanism, calculate the semantic correlation between adjacent sentence blocks through context-aware semantic segmentation technology, and dynamically adjust the threshold to determine whether to merge the sentence blocks, realize adaptive demarcation of semantic boundaries, and generate text segmentation with clear structure and coherent semantics.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of natural language text, more particularly, to a natural resource large model question and answer semantic enhancement adaptive block method and system, through text data, researches automatic block of long text as the object, aiming at the problems such as semantic fragmentation and context loss of traditional block method in processing complex structure and semantic intensive government text, and innovatively constructs the semantic enhancement adaptive text block technology for natural resource large model question and answer. BACKGROUND

[0002] Research is intended to introduce RAG technology in the field of natural resource intelligent question and answer, based on large language model (LLMs), knowledge graph, retrieval enhancement generation (RAG), multi-agent (Multi-Agent) and other technologies, to carry out efficient and intelligent natural resource government question and answer system theory and technology research. Through optimizing the key technologies such as text data processing, knowledge graph construction, intelligent retrieval and agent collaboration, the intelligence, response speed and user experience of the system are improved, so as to improve the accuracy and convenience of natural resource government information query, and help the efficient development of intelligent government.

[0003] Natural language text data processing is crucial in the field of natural language processing, especially in large-scale language models (LLMs) and information retrieval systems. Intelligent text block, semantic enhancement, and context completion technologies significantly affect text analysis and model generation capabilities.

[0004] In recent years, domestic and foreign researchers have carried out in-depth research on sliding window and context window adaptive block, semantic enhancement context completion and other technologies. Text block is an important part of text data processing, and its main purpose is to divide long text into data segments suitable for downstream tasks (such as text retrieval, summary generation, question and answer, etc.) without destroying the text semantics. Current intelligent text block methods mainly include sliding window-based methods, syntax structure-based methods and deep learning-based methods. Among them, the sliding window-based method is a common block strategy, which is mainly used for local context segmentation of long text.

[0005] In 2020, researchers employed a fixed-length sliding window in the Dense Passage Retrieval (DPR) model to improve recall in question-answering tasks. However, the fixed window size strategy cannot adapt to variations in different text types, potentially leading to semantic fragmentation. Syntactic structure-based methods primarily address the structured needs of text chunking. In 2025, researchers proposed a dynamic chunking method based on dependency parsing. This method analyzes the syntactic relationships of the text to ensure semantic integrity after chunking, improving performance in summarization and text classification tasks. In recent years, researchers have attempted to use deep learning models to predict optimal chunking positions. For example, in 2022, researchers proposed an adaptive chunking method in the FiD (Fusion-in-Decoder) model. This method uses a Transformer model to learn the optimal text fragment length for different tasks, improving the ability to aggregate information across documents.

[0006] While sliding window techniques are effective, fixed window sizes can lead to information fragmentation or redundancy. Therefore, adaptive windowing has become a research hotspot in recent years, with researchers attempting to combine sliding windows with context windows for adaptive text segmentation. In 2020, researchers introduced a dynamic window strategy in REALM (Retrieval-Augmented Language Model), which adjusts the window size based on the query content, improving the text matching performance of subsequent Retrieval Augmentation Generation (RAG) models. Also in 2020, researchers combined sliding windows and cross-document attention mechanisms to perform adaptive text segmentation at the document level, improving the performance of text retrieval and question-answering systems. In 2021, researchers used a variable-length window segmentation method in legal retrieval tasks, calculating the optimal window segmentation point through semantic matching, improving the accuracy of legal clause comparison. In 2024, researchers proposed a method combining sliding windows and dynamic context windows, automatically learning window adjustment strategies through reinforcement learning algorithms to improve performance on long document reading comprehension tasks.

[0007] To address the semantic gaps caused by text segmentation, semantic enhancement is used to expand contextual information and improve text modeling performance. In recent years, researchers have combined pre-trained language models (such as BERT and GPT-4) with knowledge enhancement methods to improve contextual completion capabilities. For example, a knowledge graph-based contextual completion model, combining entity linking and relational reasoning for text semantic enhancement, has achieved good results in text understanding tasks. Another example is the Prompt-Tuning method, which incorporates external knowledge bases to enhance the semantic richness of text in low-resource scenarios, thereby improving contextual reasoning capabilities in question-answering tasks. Furthermore, a BERT-based bidirectional contextual completion method proposed in 2024 optimizes text representation through contrastive learning, addressing the difficulty of capturing long-range dependencies in traditional methods and achieving significant improvements in dialogue generation tasks. Finally, a hierarchical contextual completion mechanism combining BERT and knowledge distillation techniques, also proposed in 2024, improves the summarization quality of long texts.

[0008] In summary, natural language text data processing enhancement technologies have developed rapidly in recent years. Significant progress has been made both domestically and internationally in intelligent text segmentation, adaptive segmentation combined with sliding windows, and semantically enhanced contextual completion techniques. International research primarily focuses on model architecture optimization and cross-document information aggregation to improve the performance of text retrieval and question-answering systems. Domestic research emphasizes combining pre-trained models (such as BERT and GPT) and reinforcement learning techniques to enhance the adaptability of text segmentation and contextual modeling capabilities. With the development of large-scale language models, how to better integrate knowledge enhancement techniques during text segmentation will become an important direction for further improving text processing capabilities. Summary of the Invention

[0009] This invention addresses the needs of existing technologies by providing a semantically enhanced adaptive segmentation method and system for question answering in a large natural resource model. It gradually constructs fine-grained text segments through a dynamic text segmentation method, overcoming the shortcomings of static text segmentation in existing technologies.

[0010] According to a first aspect of the present invention, a semantically enhanced adaptive segmentation method for question answering in a large natural resource model is provided, comprising:

[0011] Step 1: Perform heterogeneous content parsing on the input long text in the field of natural resources, identify the title hierarchy in the long text, maintain the topic tree structure, and align the title hierarchy with the related text.

[0012] Step 2: Obtain paragraph blocks under each heading level by using a topic-aware coarse-grained paragraph segmentation strategy;

[0013] Step 3: For each paragraph block, based on Chinese grammar rules and syntactic structure, automatically identify sentence boundaries and split each paragraph block into multiple fine-grained sentences to obtain the sentence set of each paragraph block;

[0014] Step 4: Based on the improved adaptive sliding window segmentation mechanism, all fine-grained sentences in the sentence set are divided into multiple window sentence block groups;

[0015] Step 5: Using the context-aware adaptive text segmentation technique, calculate the semantic relevance of adjacent window block groups. Combine the context awareness capability with the changing trend of similarity distribution to dynamically adjust the aggregation threshold to determine whether to merge multiple window block groups, adaptively delineate semantic boundaries, and obtain multiple semantically coherent semantic blocks after long text segmentation.

[0016] According to a second aspect of the present invention, a semantically enhanced adaptive segmentation system for question answering of a large natural resource model is provided, comprising:

[0017] The processing module is used to perform heterogeneous content parsing processing on the input long text in the field of natural resources, identify the title level in the long text, maintain the topic tree structure, and align the title level with the associated text.

[0018] The acquisition module is used to acquire paragraph blocks under each heading level by using a topic-aware coarse-grained paragraph segmentation strategy.

[0019] The splitting module is used to automatically identify sentence boundaries for each paragraph block based on Chinese grammar rules and syntactic structure, and split each paragraph block into multiple fine-grained sentences to obtain a set of sentences for each paragraph block;

[0020] The segmentation module is used to divide all fine-grained sentences in the sentence set into multiple window sentence block groups based on an improved adaptive sliding window segmentation mechanism;

[0021] The merging module is used for context-adaptive text segmentation technology based on embedding awareness. By calculating the semantic relevance of adjacent window block groups, and combining context awareness and the changing trend of similarity distribution, it dynamically adjusts the aggregation threshold to determine whether to merge multiple window block groups, adaptively delineates semantic boundaries, and obtains multiple semantically coherent semantic blocks after long text segmentation.

[0022] This invention provides a semantically enhanced adaptive segmentation method and system for question answering in a large-scale natural resource model. First, based on a topic-aware coarse-grained paragraph segmentation strategy, paragraph blocks are obtained at each topic level. Then, according to Chinese grammar rules and syntactic structure, sentence boundaries are automatically identified, and each paragraph block is split into multiple fine-grained sentences, resulting in a sentence set. Based on an improved adaptive sliding window segmentation mechanism, all fine-grained sentences in the sentence set are divided into multiple window sentence block groups. Using an embedding-aware context-adaptive text segmentation technique, the semantic relevance of adjacent sentence blocks is calculated, and the aggregation threshold is dynamically adjusted based on context awareness and similarity distribution trends. This allows for the merging of multiple window sentence block groups, resulting in multiple semantic blocks after long text segmentation. This invention, when grouping sentences, gradually constructs fine-grained text blocks through an adaptive dynamic window text segmentation method, dynamically adjusting the aggregation threshold based on context awareness and similarity distribution trends. This effectively improves the adaptability of text segmentation to downstream question answering and knowledge extraction tasks, enhancing the performance of intelligent question answering systems in the natural resource domain. Attached Figure Description

[0023] Figure 1 A schematic diagram of the structure of three models for implementing a semantic enhancement adaptive segmentation method for question answering in a large natural resource model, provided in one embodiment of the present invention;

[0024] Figure 2 A flowchart of a semantic enhancement adaptive segmentation method for question answering in a large natural resource model, provided in one embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of an adaptive sliding window-driven topic hierarchy partitioning model according to an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of an embedding-aware context-adaptive text segmentation model according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the structure of a semantically enhanced adaptive segmentation system for question answering of a large natural resource model, provided as an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. In addition, the technical features of the various embodiments or individual embodiments provided by the present invention can be arbitrarily combined with each other to form feasible technical solutions. Such combinations are not constrained by the order of steps and / or structural composition patterns, but must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0029] The field of natural resources encompasses multiple aspects such as mineral resources, water resources, forest resources, and land use. Corresponding texts are characterized by high specialization, a mixture of structured and unstructured data, strong contextual dependence, and uneven information distribution. Traditional question-answering methods for natural resources suffer from slow data updates, insufficient semantic understanding capabilities, and limited support for unstructured data. Therefore, intelligent question-answering in the field of natural resources requires the use of knowledge graph retrieval enhancement generation techniques based on large models to improve question-answering accuracy. Text segmentation is a crucial step in the preprocessing of natural resource texts.

[0030] In practical applications, existing natural resource text segmentation technologies have the following problems: difficulty in identifying the boundaries of complex professional terms; improper segmentation leading to the truncation of long professional phrases, which will destroy the integrity of the contextual semantics; natural resource documents often contain semi-structured content such as charts, numbered entries, and measurement point records, which increases the design complexity of segmentation strategies; uneven text information density, such as some paragraphs presenting important indicators and analysis conclusions, while other paragraphs are background explanations, making it difficult for fixed-length strategies to effectively extract the key points of the text.

[0031] Based on this, the present invention provides a semantic enhancement adaptive segmentation method for question answering of a large natural resource model, which is mainly implemented based on three core modules.

[0032] See Figure 1The following is an architecture diagram of the three core modules provided by this invention. The first module is the Topic-Aware Coarse-Grained Paragraph Segmentation (TACPS) model; the second module is the Adaptive Sliding-Window Driven Topic Hierarchical Segmentation Model (ASW-THS); and the third module is the Embedding-Aware Context-Adaptive Text Segmentation Algorithm (ECA-TS).

[0033] See Figure 2 The flowchart below shows a semantic enhancement adaptive segmentation method for question answering in a large natural resource model provided by the present invention. The method includes the following steps:

[0034] Step 1: Perform heterogeneous content parsing on the input long text in the field of natural resources, identify the title hierarchy in the long text, maintain the topic tree structure, and align the topic hierarchy with the related text.

[0035] Step 2: Obtain paragraph blocks under each topic level by using a topic-aware coarse-grained paragraph segmentation strategy.

[0036] Understandably, documents in the field of natural resources are often lengthy or have varied themes, frequently containing semi-structured content such as charts, numbered entries, and measurement point records, increasing the complexity of the segmentation strategy design. This invention innovatively constructs a topic-aware coarse-grained paragraph segmentation algorithm. In this embodiment, steps 1 and 2 are performed by... Figure 1 It is implemented using a topic-aware coarse-grained paragraph segmentation model.

[0037] The following section introduces the topic-aware coarse-grained paragraph segmentation model.

[0038] Existing documents in the field of natural resources are often lengthy or have varied themes, frequently embedding semi-structured content such as charts, numbered entries, and measurement point records, increasing the complexity of designing segmentation strategies. Traditional segmentation relies on text paragraph boundaries and fixed thresholds, treating charts as non-text elements and hierarchically segmenting numbered entries, which can easily lead to semantic breaks. This invention designs a topic-aware coarse-grained paragraph segmentation algorithm that uses topic consistency as the segmentation criterion, merging charts and related text into topic blocks and aggregating numbered entries by topic. The topic-aware coarse-grained paragraph segmentation model mainly includes steps such as multi-source heterogeneous text parsing and topic aggregation segmentation.

[0039] (1) Multi-source heterogeneous text parsing:

[0040] Multi-source heterogeneous text parsing primarily analyzes heterogeneous content from input documents such as DOCX and PDF, including topics, paragraph text, images, and tables. It maintains a topic tree structure by providing heading hierarchy recognition (e.g., based on paragraph styles); image parsing extracts images and simultaneously parses the context text, aligning recognized chart titles with related text entities; table parsing normalizes rows and columns of unstructured table content and extracts keywords relevant to the topic. Based on multi-source heterogeneous text parsing, a unified representation of heterogeneous content such as topics, paragraph text, images, and tables is constructed.

[0041] In subsequent processing, images in the document are not involved; for tables, the text content of the tables is extracted and used in subsequent processing.

[0042] (2) Coarse-grained paragraph segmentation strategy based on topic awareness:

[0043] This invention uses regular expressions to uniformly represent and construct heterogeneous text content. It allocates topics based on a domain dictionary (such as the terminology set of the "Technical Specification for Soil Environmental Monitoring"), resulting in paragraph blocks under each topic. To maintain the hierarchical structure of the document, the topic allocation strategy employs a stack structure to manage heading paths. When a new heading is encountered, the current heading path stack is updated according to its level, ensuring the correct heading paths for each paragraph.

[0044] Let the set of document titles be ,in This indicates the h-th heading element, whose level is... The title path stack S is used to dynamically maintain the title hierarchy at the current parsing position. The initial state is... .

[0045] When a new title is parsed At that time, the stack update rule is:

[0046] ;

[0047] In the formula, This represents the top element of the stack, i.e., the current highest-level heading. This means extracting the first k elements from the stack. This represents the level of the stack's top title, initially defined as... , This represents the last element in the stack. This indicates that the new title is being added to the stack. Indicates new title The hierarchy.

[0048] The topic-aware coarse-grained paragraph segmentation algorithm is designed as follows:

[0049] Algorithm: Topic-aware coarse-grained paragraph segmentation algorithm;

[0050] Input: DOC, DOCX, PDF, and other document formats;

[0051] Output: Thematic text structured chunks in Markdown format (a lightweight markup language format).

[0052] S11: Compare the hierarchical relationships to determine the new heading level. With the current top level of the stack The size relationship.

[0053] S12: If The level is raised, and the new title will be... Push onto the top of the stack and update the stack: .

[0054] S13: If The stack depth decreases, popping the top element until the stack depth equals the top element. New title Push onto the stack and update the stack: .

[0055] S14: If Create a new heading at the same level. Replace the top element of the stack and update the stack: .

[0056] S15: Output the following uniformly represented topic-text structured chunking results:

[0057] Taking a table as an example, the Markdown format theme - structured text block output:

[0058] #First-level heading - Paragraph 1;

[0059] #Heading 1 - Paragraph 2;

[0060] ...

[0061] ## Level 2 heading - paragraph n.

[0062] Step 3: For each paragraph block, based on Chinese grammar rules and syntax structure, automatically identify sentence boundaries and split each paragraph block into multiple fine-grained sentences to obtain the sentence set of each paragraph segment.

[0063] Step 4: Based on the improved adaptive sliding window segmentation mechanism, all fine-grained sentences in the sentence set are divided into multiple window sentence block groups.

[0064] Understandably, steps 1 and 2 divide the input long text in the natural resources field into multiple paragraph blocks based on a topic segmentation mechanism. Steps 3 and 4 require further segmentation of these paragraph blocks. Traditional fixed-length segmentation strategies fail to account for the uneven density of text information, leading to the mechanical splitting of high-density information paragraphs such as important indicators and analytical conclusions, or their forced merging with low-density content like background information. This results in the fragmentation of core semantics and reduced information extraction efficiency. This invention innovatively constructs an adaptive sliding window-driven topic hierarchy segmentation model. Steps 3 and 4 divide each paragraph block into multiple window sentence blocks using this adaptive sliding window-driven topic hierarchy segmentation model.

[0065] The following section introduces the topic hierarchy partitioning model driven by the adaptive sliding window.

[0066] In the field of natural resources, documents exhibit uneven information density. Some paragraphs emphasize key indicators and analyses, while others provide background information. Traditional fixed-length block segmentation strategies struggle to accommodate this unevenness, leading to the mechanical splitting of high-density information paragraphs (such as important indicators and analyses) or their forced merging with low-density content like background information. This results in a fragmentation of core semantics and reduced information extraction efficiency. This invention proposes a novel adaptive sliding window-driven topic hierarchy segmentation model. By dynamically calculating the content density and topic coherence within the window in real time, the model dynamically adjusts the window size, step size, and overlap rate to achieve adaptive hierarchical segmentation of natural resource documents. The dynamic adjustment of window size, step size, and overlap rate, combined with word segmentation and sentence segmentation preprocessing, enables differentiated segmentation of high- and low-density information paragraphs in natural resource documents. The adaptive sliding window-driven topic hierarchy segmentation model mainly includes steps such as fine-grained text formatting segmentation and adaptive sliding window block segmentation.

[0067] (1) Fine-grained text formatting segmentation:

[0068] Content formatting and segmentation are based on the unified Markdown format for representing topic-text structured text blocks, namely the paragraph blocks output in steps 1 and 2 above. According to Chinese grammar rules and syntax structure, sentence boundaries are automatically identified, and long text paragraphs divided by topic are split into fine-grained sentences to generate a standardized set of sentences for each paragraph block.

[0069] (2) Adaptive sliding window segmentation:

[0070] See Figure 3 For each paragraph block output by fine-grained segmentation, this invention designs an adaptive sliding window segmentation algorithm, designs dynamic parameter optimization technology, and constructs a dynamically adjusted sliding window segmentation mechanism to avoid the semantic fragmentation problem that exists in fixed window text segmentation.

[0071] In one embodiment of the present invention, step 4, based on an improved adaptive sliding window segmentation mechanism, divides all fine-grained sentences in the sentence set into multiple window sentence block groups, including:

[0072] Step 41: Set the sliding window size to represent the number of sentences contained in the window and initialize the window sliding step size.

[0073] Understandably, in the design of dynamic parameter optimization algorithms, a fixed-size sliding window is first preset, encapsulating multiple consecutive sentences as basic processing units, and the sliding iteration starts from the first sentence of the text. Let the size of the sliding window be... This indicates the number of sentences contained within the window and the initial window sliding step size. =1. In this embodiment of the invention, the window sliding step size is dynamically adjusted based on the semantic relationships between sentences. The text sentence sequence is defined as follows: , No. The set of sentences within the second sliding window is represented as follows: .

[0074] Step 42: For each sliding window, calculate the similarity between two adjacent sentences within the sliding window in sequence.

[0075] Specifically, for the first Secondary sliding window Two adjacent sentences within and , Calculate the cosine similarity: ;

[0076] In the formula, and Sentences and The term frequency-inverse document frequency feature vector, For vector dot product, and Let represent the L2 norm of the eigenvectors.

[0077] Step 43: Based on the relationship between the similarity between two adjacent sentences and the preset similarity threshold, determine whether to group the two adjacent sentences into the same window group, dynamically adjust the next sliding step size, and slide the window according to the next sliding step size.

[0078] Among them, for the first Slide the window to group the windows. Initialize to That is, during initialization, window groups The first sentence in the sliding window is included. If... Less than the similarity threshold This indicates that the semantic connection between two adjacent sentences is insufficient, and will As a window clause, the sentence and sentences It will be assigned to two window groups, and the next sliding step size will be dynamically adjusted. .like Greater than or equal to the similarity threshold ,sentence and sentences If they are assigned to the same window group, the comparison will continue. Cosine similarity between sentences, and Add window group This avoids invalid computation of incoherent semantic regions and dynamically adjusts the next sliding step size. .

[0079] The dynamic adjustment of the next sliding step size includes:

[0080] .

[0081] Step 44: Repeat steps 42 to 43 until all fine-grained sentences in the sentence set have been divided, resulting in multiple window sentence block groups.

[0082] Specifically, by repeatedly executing steps 42 and 43, the window is continuously slid until all fine-grained sentences in the sentence set are divided, resulting in multiple window sentence block groups.

[0083] This invention, through its improved adaptive window step size adjustment strategy, enables the adaptive sliding window block model to effectively reduce redundant computation while ensuring the semantic integrity of the text, thereby achieving a dual improvement in the efficiency and accuracy of document block segmentation in the field of natural resources.

[0084] The improved adaptive sliding window block partitioning algorithm is designed as follows:

[0085] Algorithm: Adaptive sliding window-driven topic hierarchy partitioning model;

[0086] Input: Topic - Text structured chunk dataset (Markdown format);

[0087] Output: Topic window - Sentence structured dataset (JSON format).

[0088] S21: Based on Chinese grammar rules and syntax structure, identify sentence boundaries (such as periods, question marks, exclamation marks, etc.) and break down long text paragraphs into fine-grained sentence sequences.

[0089] S22: Initialize the fixed window size W=k and sliding step size Similarity threshold Current window position Window start position .

[0090] S23: By constructing a window, extracting features, calculating similarity, dynamically adjusting the step size, and performing window sliding iterations, an adaptive block-based sentence set is generated.

[0091] S24: Perform hierarchical construction on the input adaptive sentence segmentation results, calculate the topic similarity between adjacent sentences and merge them into window sentence blocks to form higher-level topic units.

[0092] Step 5: By calculating the semantic relevance of adjacent sentence blocks, the multiple window sentence blocks are grouped and merged to obtain multiple semantic blocks after the long text is divided.

[0093] Understandably, the window segmentation generated in step 4 may truncate long technical terms, disrupting the integrity of the contextual semantics. Addressing the difficulty in identifying the boundaries of complex technical terms in natural resource documents, and the fact that improper segmentation can lead to the truncation of long technical terms and disrupt the integrity of the contextual semantics, this invention innovatively constructs an embedding-aware context-adaptive text segmentation model.

[0094] For the multiple window sentence block groups obtained by the adaptive sliding window segmentation algorithm, this embodiment of the invention further merges the window sentence block groups based on the embedding-aware context adaptive text segmentation model to obtain the final semantic blocks.

[0095] The following section introduces the embedding-aware context-adaptive text segmentation model.

[0096] In documents related to natural resources, identifying the boundaries of complex technical terms is difficult, and improper segmentation leads to the truncation of long technical phrases, compromising the integrity of the semantic context. This invention proposes a novel embedding-aware, context-adaptive text segmentation model. This model improves the threshold adjustment strategy in the pre-trained model for extracting semantic context. The strategy dynamically adjusts the threshold by analyzing the semantic similarity between different window block groups in the text, automatically determining how to aggregate and segment according to semantics. This method considers the contextual relationships of the text, combining a semantic model and a dynamic adjustment mechanism, effectively improving the quality of the segmentation results. See also... Figure 4 The specific implementation steps of the embedding-aware context-adaptive text segmentation model are as follows:

[0097] (1) Semantic embedding model construction:

[0098] This invention uses a Sentence-Transformer model (e.g., multilingual-e5-large) to process semantic information in topic-sentence structured data text. This model groups window segments of sentences based on the input topic-sentence structured dataset. Calculate the embedding vector for each window block group to form an embedding vector set. ,in (d is the vector dimension).

[0099] Calculate the semantic association between two adjacent window block groups:

[0100] ;

[0101] In the formula, adjacent window blocks are grouped. The degree of semantic association, with a value range of . A larger value indicates greater semantic similarity.

[0102] (2) Dynamic threshold calculation:

[0103] This invention uses a topic-sentence segmentation structured dataset to construct window sentence block grouping. and the set of embedding vectors generated based on Sentence-Transformer The similarity threshold is then used for comparison based on the structured data aggregation threshold within the window. To ensure flexibility, the similarity threshold is dynamically adjusted based on the length of the window block group and the context content. Longer window block groups will have a higher threshold to avoid unnecessary merging. The dynamic adjustment strategy is designed as follows:

[0104] ;

[0105] ;

[0106] In the formula, To and The corresponding aggregation threshold, Based on the threshold, Group the current window block Sentence length (normalized to) ), This is a length adjustment factor, with longer sentences corresponding to a higher threshold to avoid cross-topic merging. The context fluctuation coefficient reflects the standard deviation of the semantic vector within the window, suppressing erroneous merging of semantically abrupt regions. express arrive The set of embedded vectors within.

[0107] It should be noted that this invention groups two adjacent window segments and sets different similarity thresholds. The similarity thresholds are not fixed and can be better adapted to semantic segmentation.

[0108] (3) Contextual semantic segmentation based on dynamic threshold calculation:

[0109] This invention uses dynamically calculated thresholds to determine whether different window block groups should be merged into a single semantic block. ≥ Then adjacent window blocks are grouped. Grouping of window blocks They can be merged. If Then adjacent window blocks are grouped. Grouping of window blocks No merging is performed. After merging window sentence blocks, the final semantic blocks are obtained.

[0110] The content under each heading is reorganized according to the semantic segmentation results. Each segment contains the number of sentences and their specific content. All segmentation results are saved as a structured JSON file for easy subsequent analysis and processing.

[0111] The design process of the embedding-aware context-adaptive text segmentation algorithm is as follows:

[0112] Algorithm: Embedding-aware context-adaptive text segmentation model;

[0113] Input: Topic-Sentence Structured Dataset (JSON format);

[0114] Output: Structured semantic chunked dataset (JSON format).

[0115] S31: Load the Sentence-Transformer pre-trained model, configure the model parameters, and specify the embedding vector dimension d.

[0116] S32: Calculate the embedding vector of the input window block grouping.

[0117] S33: Use regular expressions to match Markdown heading lines, extract each heading and its level, and generate a heading-content mapping relationship.

[0118] S34: Group window blocks Grouping window blocks according to the dynamic threshold formula Calculate its dynamic threshold.

[0119] S35: Traverse window block groups and compare the blocks in adjacent windows. and ,like ≥ ,Will Add to the current block; otherwise, save the current block as a separate text block and create a new semantic block.

[0120] S36: Generate a set of semantic chunks M represents the final semantic blocks, each containing consecutive and semantically similar sentences, and a structured semantic block dataset is constructed.

[0121] Finally, the semantic enhancement adaptive text segmentation algorithm provided in this embodiment of the invention will be described. The specific design of the algorithm is as follows:

[0122] Algorithm: Context-aware iterative block segmentation technique for natural resource texts for large-scale question answering;

[0123] Input: DOC, DOCX, PDF, and other document formats;

[0124] Output: Structured semantic chunked dataset (JSON format).

[0125] S41: Input documents such as DOC, DOCX, and PDF.

[0126] S42: Perform parsing of heterogeneous content such as topics, paragraph text, images, and tables, and construct a unified representation of multi-source heterogeneous text content.

[0127] S43: Use regular expressions to assign topics to heterogeneous text content with a unified representation, and construct a Markdown-formatted topic-text structured block result.

[0128] S44: Set the initial adaptive sliding window size, step size, and overlap rate.

[0129] S45: Input topic - Text structured segmentation results, based on Chinese grammar rules and syntax structure, identify sentence boundaries, and split long text paragraphs into fine-grained sentence sequences.

[0130] S46 generates adaptive block-based window sentence groups by constructing windows, extracting features, calculating similarity, dynamically adjusting step size, and performing window sliding iterations.

[0131] S47: Construct a semantic embedding pre-trained model, configure model parameters, and calculate the semantic similarity relationship of each group of window sentence blocks.

[0132] S48: Group window blocks Based on the dynamic threshold formula, the sentences in each window are... Calculate its dynamic threshold.

[0133] S49: Traverse window block groups and compare the blocks in adjacent windows. and ,like ≥ ,Will Add to the current block; otherwise, save the current block as a separate text block and create a new semantic block.

[0134] S410: Generate a set of semantic chunks M represents the final semantic blocks, each containing consecutive and semantically similar sentences, and a structured semantic block dataset is constructed.

[0135] See Figure 5 The present invention also provides a semantically enhanced adaptive segmentation system for question answering in a large natural resource model, the system comprising:

[0136] The processing module 501 is used to perform heterogeneous content parsing processing on the input long text in the field of natural resources, identify the title level in the long text, maintain the topic tree structure, and align the title level with the associated text.

[0137] The acquisition module 502 is used to acquire paragraph blocks under each heading level by using a topic-aware coarse-grained paragraph segmentation strategy.

[0138] The splitting module 503 is used to automatically identify sentence boundaries for each paragraph block according to Chinese grammar rules and syntactic structure, split each paragraph block into multiple fine-grained sentences, and obtain a set of sentences for each paragraph block;

[0139] The segmentation module 504 is used to divide all fine-grained sentences in the sentence set into multiple window sentence block groups based on an improved adaptive sliding window segmentation mechanism;

[0140] The merging module 505 is used to merge multiple window sentence block groups by calculating the semantic relevance of adjacent window sentence block groups, so as to obtain multiple semantic blocks after the long text is divided.

[0141] It is understood that the semantic enhancement adaptive segmentation system for question answering of a large natural resource model provided by this invention corresponds to the semantic enhancement adaptive segmentation method for question answering of a large natural resource model provided in the foregoing embodiments. The relevant technical features of the semantic enhancement adaptive segmentation system for question answering of a large natural resource model can be referred to the relevant technical features of the semantic enhancement adaptive segmentation method for question answering of a large natural resource model, and will not be repeated here.

[0142] The following example illustrates the semantic enhancement adaptive segmentation method for question answering in the natural resource big model provided by this invention.

[0143] 1) Dataset Overview

[0144] Product Introduction: This document first provides a brief introduction to a company, then details various product features from that company under different themes. The dataset contains 18 themes in total, with the first theme, "Company Introduction," containing the most sentences, consisting of seven sentence groups. A portion of the dataset is shown in Table 1.

[0145] Table 1 Product Introduction

[0146]

[0147] Settlement Experiment Analysis Report: This document mainly introduces the research background, objectives, content, and current research status at home and abroad of the settlement project, covering a total of seven topics. The first topic, the research background, contains the most sentences, totaling 16. Partial datasets are shown in Table 2.

[0148] Table 2 Settlement Experiment Analysis Report

[0149]

[0150] 2) Overview of the study area

[0151] Texts in the natural resources field encompass various aspects such as minerals, water, forests, and land use, exhibiting characteristics such as high specialization, a mixture of structured and unstructured content, high context dependence, and uneven information distribution. Traditional question-answering methods are insufficient to meet these needs, necessitating knowledge graph retrieval enhancement technologies based on large-scale models. Text segmentation, as a crucial preprocessing step, faces challenges such as difficulty in identifying the boundaries of complex technical terms, complexity in processing semi-structured content, and the inability of fixed-length strategies to adapt to differences in information density. This study selects two datasets: "Product Introduction" and "Settlement Experiment Analysis Report." The former covers introductions to company business and product functions, totaling 18 topics, with the "Company Introduction" topic having the longest sentence cluster. The latter focuses on 7 topics, including the research background and objectives of settlement projects, with the "Project Research Background" topic having the most sentences. These two document types represent business application-oriented and academic research-oriented texts, respectively, comprehensively validating the effectiveness of intelligent segmentation technology in different scenarios.

[0152] 3) Hardware and software environment

[0153] The experimental hardware and software configuration is shown in Table 3. The operating system is Linux (64-bit); the GPU is NVIDIA Colorful 4060Ti GPU; the memory is 16G; the hard disk is 50T; the deep learning platform is PyTorch; and the programming language is Python 3.6.

[0154] Table 3 Experimental Environment

[0155]

[0156] 4) Experimental data

[0157] To analyze the effectiveness of the semantic enhancement adaptive text segmentation technology for question answering of the large-scale natural resource service model proposed in the invention, the datasets "Product Introduction" and "Settlement Experiment Analysis Report" were selected for the experiment.

[0158] Dataset Construction: Documents in the natural resources field are characterized by their length, diverse themes, and complex semi-structured content, making it difficult for traditional segmentation strategies to guarantee semantic integrity. When constructing the experimental dataset for natural text segmentation, we first collected documents covering various types, including academic reports and policy documents, and screened them to ensure representativeness in terms of length, theme, and structure. Then, through multi-source heterogeneous text parsing, we identified themes, paragraphs, images, tables, etc., constructed a theme tree using heading levels, aligned image text, normalized tables, and extracted keywords to form a unified representation. Next, we used regular expressions combined with a domain dictionary to assign topics, managed heading paths based on a stack structure, and employed a topic-aware coarse-grained paragraph segmentation algorithm to merge charts and related text, aggregate numbered entries by topic, and annotate the segmentation results in Markdown format. Finally, we performed data cleaning.

[0159] 5) Case Analysis

[0160] 1. Benchmark Experiment

[0161] Table 4 Experimental results of various text segmentation models

[0162]

[0163] The experiment uses the large model segmentation results as a benchmark to compare and evaluate two benchmark text segmentation algorithms and two text segmentation algorithms designed in the invention: (1) Fixed-size segmentation algorithm (Fixed-BA); (2) Recursive segmentation algorithm (Recursive-BA); (3) Adaptive sliding window driven topic hierarchical segmentation model (ASW-THS); (4) Embedding-aware context-adaptive text segmentation algorithm (ECA-TS). Evaluation metrics include: Semantic consistency (SC): the semantic relevance of sentences within a segment, calculated by BERT embedding of the average cosine similarity of sentence pairs; Segment integrity (CC): the semantic matching degree between the predicted segment and the large model segment, using the mean of maximum similarity in segment-level embedding; F1 score: a comprehensive evaluation of the precision and recall of segment boundary recognition.

[0164] Table 4 shows the evaluation metrics of various text chunking models. Different models exhibit significant differences in performance across different task metrics. Fixed-BA performs poorly overall, with a CC of only 86.03%. Furthermore, because it segments the text according to a fixed length (e.g., number of characters, words, or tokens), each chunk is of uniform size, leading to… The F1 scores were only 15.57% and 20.69%, respectively, significantly lower than other comparative models. Recursive-BA first initially divides the text by topic or paragraph, then recursively subdivides the extremely long blocks until the size limit is met. Its CC (Corrective Conformity) is superior to Fixed-BA, reaching 90.65%. The ECA-TS model improves the threshold adjustment strategy in its pre-trained model for extracting semantic context. By dynamically adjusting the threshold through analyzing the semantic similarity between different paragraphs or sentences in the text, it automatically determines how to aggregate blocks semantically. Experimental results show that this model outperforms the ASW-THS model in both Block Completeness (CC) and Semantic Consistency (SS). For example, in paragraphs such as "Smart Government" and "Fintech," semantic block segmentation merges topic-related long sentences, achieving a consistency of 76.90%, higher than the 70.34% of ASW-THS segmentation, validating the effectiveness of semantic association modeling. Both segmentation algorithms achieve a completeness higher than 95%, indicating a high degree of overlap with the large model's block segmentation in terms of content coverage. Among them, ECA-TS segmentation (96.99%) outperformed ASW-THS segmentation (95.67%), mainly because semantic segmentation preserves the semantic unit boundaries of large model segments through semantic embedding vector matching.

[0165] 2. Ablation test

[0166] Table 5 Ablation Experiment Results

[0167]

[0168] The table above shows the ablation experiment results of ECA-TS. ASW-THS dynamically adjusts the window size, step size, and overlap rate by calculating the content density and topic coherence within the window in real time, thus achieving adaptive hierarchical segmentation of natural resource documents. The model's chunk integrity (CC) and semantic consistency (SS) are 71.22% and 87.87%, respectively. Among them, ECA-TS chunking (96.99%) is better than ASW-THS chunking (95.67%). The main reason is that ECA-TS is based on the window sentence block grouping generated by the ASW-THS model, and dynamically adjusts the threshold by analyzing the semantic similarity between different window sentence block groups in the text, automatically deciding how to aggregate chunks according to semantics.

[0169] This invention provides a semantically enhanced adaptive text segmentation method and system for question answering in a large-scale natural resource model. Addressing the problems existing in long text segmentation for natural resources, it proposes a semantically enhanced adaptive text segmentation technology for question answering in a large-scale natural resource model. The method mainly includes three core technologies: a topic-aware coarse-grained paragraph segmentation algorithm, an adaptive sliding window-driven topic-level segmentation model, and an embedding-aware context-adaptive text segmentation model. Through adaptive dynamic windows and text segmentation methods that dynamically adjust similarity thresholds, fine-grained text segments are gradually constructed, effectively improving the adaptability of text segments to downstream question answering and knowledge extraction tasks, enhancing the performance of intelligent question answering systems in the natural resource field, and providing an effective technical solution.

[0170] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0171] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0173] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0175] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0176] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A semantic enhancement adaptive partitioning method for natural resource large model question answering, characterized in that, The method comprises the following steps: Step 1, heterogeneous content parsing is performed on an input long text in a natural resource field, title levels in the long text are identified, a topic tree structure is maintained, and the title levels are aligned with associated texts; Step 2, a topic-aware coarse-grained paragraph division strategy is used to obtain paragraph blocks under each title level; Step 3, for each paragraph block, sentence boundaries are automatically identified according to Chinese grammar rules and syntactic structures, and each paragraph block is split into a plurality of fine-grained sentences to obtain a sentence set of each paragraph block; Step 4, an improved adaptive sliding window segmentation mechanism is used to divide all fine-grained sentences in the sentence set into a plurality of window sentence block groups; Step 5, a context-aware adaptive text segmentation technology is used to calculate semantic correlation degrees of adjacent window sentence block groups, a dynamic adjustment threshold is combined with a context-aware capability and a similarity distribution change trend to determine whether to merge the plurality of window sentence block groups, semantic boundaries are adaptively delimited, and a plurality of semantically coherent semantic segments obtained after long text division are obtained; In the step 2, the topic-aware coarse-grained paragraph division strategy is used to obtain the paragraph blocks under each title level, which comprises the following steps: A stack structure is used to manage a title path, when a new title is encountered, a current title path stack is updated according to a level of the new title and a level of a top title of the stack, and the paragraph blocks under each title level are obtained; In the step 4, the improved adaptive sliding window segmentation mechanism is used to divide all fine-grained sentences in the sentence set into the plurality of window sentence block groups, which comprises the following steps: Step 41, a sliding window size representing a number of sentences contained in the window and an initial window sliding step are set; Step 42, for each window after sliding, similarity degrees between adjacent two sentences in the window are calculated in sequence; Step 43, a size relationship between the similarity degrees between the adjacent two sentences and a preset similarity threshold is determined, whether the adjacent two sentences are divided into a same window group is determined according to the size relationship, a next window sliding step is dynamically adjusted according to the size relationship, and the window is slid according to the next window sliding step; Step 44, the steps 42 to 43 are repeatedly executed until all fine-grained sentences in the sentence set are divided, and the plurality of window sentence block groups are obtained; In the step 5, the calculation of the aggregation threshold of the adjacent window sentence block groups comprises the following steps: The aggregation threshold of the adjacent window sentence block groups is calculated according to a normalized value of a sentence length of a current window sentence block group and a context fluctuation coefficient reflecting a standard deviation of semantic vectors in the window.

2. The method of claim 1, wherein, In the step of using the stack structure to manage the title path, when the new title is encountered, the current title path stack is updated according to the level of the new title and the level of the top title of the stack, which comprises the following steps: Let the set of long text titles be , and m is the total number of titles, where represents the hth title element, and its level is ; The title path stack S is used to dynamically maintain the title hierarchy of the current parsing position, and the initial state is ; When a new title is parsed The update rule of the title path stack S is as follows: ； wherein, denotes the top element of the stack, i.e. the current highest level heading, denotes the top k elements of the stack, denotes the level of the top heading of the stack, initially defined as , denotes the last element in the stack, denotes the addition of a new heading to the stack, denotes the level of the new heading .

3. The semantically enhanced adaptive tiling method of claim 1, wherein, In the step 41, the sliding window size representing the number of sentences contained in the window and the initial window sliding step are set, which comprises the following steps: Set the sliding window size to represent the number of sentences contained in the window, initialize the first window sliding step size =1; In the step 42, for each window after sliding, the similarity degrees between the adjacent two sentences in the window are calculated in sequence, which comprises the following steps: define a sequence of sentences in the set of sentences as , the th sliding window, the set of sentences is denoted as ; For the sentence set in the window after the first sliding, the TF-IDF algorithm is called to calculate the term frequency-inverse document frequency feature vector of each sentence; For the first sliding window of adjacent two sentences and , , the cosine similarity is calculated: ; where, and are the term frequency-inverse document frequency feature vectors for the sentences and respectively, is the vector dot product, and denote the L2 norm of the feature vectors respectively.

4. The method of claim 3, wherein, The step 43 comprises: determining whether to divide the two adjacent sentences into the same window group according to a size relationship between the similarity between the two adjacent sentences and a preset similarity threshold, and dynamically adjusting a next window sliding step length, and the step 43 comprises: For the After the first slide, group the windows. Initialize to ,like Less than the preset similarity threshold ,Will Grouping windows into blocks and dynamically adjusting the window slide step size for the next iteration. ;like Greater than or equal to the preset similarity threshold We will continue to compare Cosine similarity between sentences, and Add window group And dynamically adjust the window sliding step size for the next time. ; The dynamically adjusting the next sliding step length comprises: 。 5. The semantically enhanced adaptive tiling method of claim 1, wherein, The step 5 comprises: calculating a semantic correlation degree of adjacent window sentence block groups by using a context-aware adaptive text segmentation technology, combining a context-aware capability and a change trend of the similarity distribution to dynamically adjust an aggregation threshold, and determining whether to merge the multiple window sentence block groups to adaptively demarcate a semantic boundary, so as to obtain a plurality of semantic blocks obtained after long text division. Let the window sentence chunk group be , for the gth window sentence chunk group, n is the number of window sentence chunk groups, computing an embedding vector for each windowed sentence chunk grouping , composing a set of embedding vectors wherein , d is a vector dimensionality calculating semantic correlation degree of two adjacent window sentence chunk groups: ; In the formula, Grouping of adjacent window sentences With Semantic correlation degree, the value range is ; The step 5 comprises: The step 5 comprises:

6. The semantically enhanced adaptive tiling method of claim 5, wherein, Adjacent window sentence chunk grouping With The corresponding aggregation threshold is: ； ； In the formula, Based on the threshold, Group the current window block The sentence length, normalized to , This is the length adjustment factor. The context fluctuation coefficient reflects the standard deviation of the semantic vector within the window. express arrive The set of embedded vectors within.

7. A semantic enhancement adaptive partitioning system for natural resource large model question answering, used to implement the semantic enhancement adaptive partitioning method for natural resource large model question answering in claim 1, characterized in that, The step 5 comprises: The processing module is configured to perform heterogeneous content analysis processing on the input long text, identify title levels in the long text, maintain a topic tree structure, and align the title levels with associated texts. The acquisition module is configured to acquire a paragraph block under each title level by using a topic-aware coarse-grained paragraph division strategy. The splitting module is configured to, for each paragraph block, automatically identify a sentence boundary according to Chinese grammar rules and syntax structures, split each paragraph block into a plurality of fine-grained sentences, and obtain a sentence set of each paragraph block. The division module is configured to divide all the fine-grained sentences in the sentence set into a plurality of window sentence block groups based on an improved adaptive sliding window division mechanism. The merging module is configured to, based on the context-aware adaptive text segmentation technology, calculate a semantic correlation degree of adjacent window sentence block groups, combine a context-aware capability and a change trend of the similarity distribution to dynamically adjust an aggregation threshold, determine whether to merge the multiple window sentence block groups, adaptively demarcate a semantic boundary, and obtain a plurality of semantic blocks obtained after long text division.

Images