Government affair large model semantic perception text segmentation technology

Abstract

The invention belongs to the field of natural language processing, and provides a government affair large model semantic perception text segmentation technology which comprises the steps of government affair knowledge base construction, government affair large model training, question input, retrieval document extraction, text partitioning, answer generation and answer fusion. According to the method, document partitioning is carried out through a semantic degree partitioning strategy, a large government affair model can be assisted to understand and retrieve documents, and the answer accuracy and the answer efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a semantically-aware text segmentation technology for a large government model. Background Art

[0002] Retrieval-augmented generation, or RAG, is an advanced technical paradigm that aims to overcome several challenges encountered by large language models, or LLMs, including data update lags, incorrect information generation, and lack of domain-specific knowledge. This technology is particularly important in dealing with knowledge-intensive tasks such as open-domain question answering. By combining two core components, the retriever and the generator, this technology can improve the accuracy of responses to input queries. Although the practicality of retrieval-augmented strategies has been widely verified in practice, their effectiveness depends greatly on the relevance and accuracy of the retrieved documents. If too much redundant or incomplete information is introduced during the retrieval process, it will not only fail to improve the performance of the generation model, but may also reduce the overall quality of the answer.

[0003] In the face of these challenges, current research work mainly focuses on two directions: one is to improve the accuracy of retrieval, and the other is to enhance the resistance of LLMs to harmful information. However, in RAG systems, the chunking of text content is often overlooked, and this step is directly related to the effectiveness of intensive retrieval. Traditional text chunking techniques mostly rely on established rules or semantic similarity. Although they can provide certain structured segmentation, they often cannot fully capture the subtle differences in the logical relationship between sentences. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a semantic-aware text segmentation technology for a large government model, build an efficient and accurate government question and answer system, and improve the performance of the government question and answer system based on retrieval enhanced generation.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A semantic-aware text segmentation technology for a large government model, comprising:

[0007] Collect and organize relevant documents in the field of government affairs to obtain a government affairs knowledge base; the relevant documents include: policies, regulations and service guidelines;

[0008] Using the government affairs knowledge base to train the government affairs big model;

[0009] Receive question input from target users;

[0010] Determine a retrieval model, and use the retrieval model to retrieve documents related to the question input in the government affairs knowledge base to obtain retrieval documents;

[0011] Using a semantic block strategy to segment the retrieved document into blocks to obtain text blocks;

[0012] Input the text blocks into the trained government affairs model to generate answers, and obtain generated answers;

[0013] All the generated answers are integrated to obtain the target output answer.

[0014] Preferably, the search document is segmented into blocks using a semantic block strategy to obtain text blocks, including:

[0015] Preliminarily segmenting the search document into sentences;

[0016] Using LLM to perform binary classification on the sentences in the retrieved document to obtain segmentation probability data;

[0017] Calculate the merging probability data and the separating probability data of any two sentences in the search document by using LLM;

[0018] Calculate the difference between the combined probability data and the separated probability data to obtain the margin value; the calculation formula of the margin value is: Margin M (x i )=P split -P merge Among them, Margin M (x i ) is the sentence x i The marginal value; the P split is the separation probability data; P merge is the combined probability data;

[0019] The semantic degree of the sentences in the search document is calculated using LLM, and a boundary threshold is set according to the distribution of the semantic degree; the calculation formula of the semantic degree is: Among them, PPL M (x i ) is the sentence x i The semantic degree of the sentence x i Total number of tokens; Represents the sentence x i The Mth token; Represents the sentence x i The semantic degree of the Mth token;

[0020] According to a preset segmentation threshold, a merging threshold and a boundary threshold, the search document is segmented using the segmentation probability data, the marginal value and the semantic degree to obtain the text segmentation.

[0021] Preferably, the retrieval model is any one of BM25 and Anserini.

[0022] Preferably, it also includes:

[0023] The generated answer corresponding to each of the text blocks is cached.

[0024] Preferably, the government affairs knowledge base supports multiple rounds of question and answer for the government affairs big model

[0025] Preferably, it also includes:

[0026] The boundary threshold is dynamically adjusted according to the distribution of the semantic degrees of different retrieved documents.

[0027] The present invention discloses the following technical effects:

[0028] The present invention provides a semantic-aware text segmentation technology for a large government model. Document segmentation is performed through a semantic degree segmentation strategy, which solves the defect that conventional large-scale language models rely on established rules or semantic similarity and cannot fully capture the subtle differences in the logical relationship between sentences. The function of assisting document segmentation according to the semantic degree and boundaries of the retrieved documents is realized. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 A schematic diagram of the question-answering process of the government affairs big model provided in an embodiment of the present invention;

[0031] Figure 2 A question-and-answer flow chart of the government affairs big model provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0033] The purpose of the present invention is to provide a semantically-aware text segmentation technology for a large government model, to build an efficient and accurate government question-and-answer system, and to improve the performance of the government question-and-answer system based on retrieval-enhanced generation.

[0034] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] Figure 1 A schematic diagram of the question-answering process of the government affairs big model provided by an embodiment of the present invention, Figure 2 A flowchart of a government affairs model question and answer provided by an embodiment of the present invention is shown in FIG. Figure 1 and Figure 2 As shown, the present invention provides a semantic-aware text segmentation technology for a large government model, including:

[0036] Step 100: Collect and organize relevant documents in the field of government affairs to obtain a government affairs knowledge base; the relevant documents include: policies, regulations and service guidelines;

[0037] Step 200: Using the government affairs knowledge base to train the government affairs model;

[0038] Step 300: receiving a question input from a target user;

[0039] Step 400: determining a retrieval model, and using the retrieval model to retrieve documents related to the question input in the government affairs knowledge base to obtain retrieval documents;

[0040] Step 500: segment the search document into blocks using a semantic block strategy to obtain text blocks;

[0041] Step 600: Input the text blocks into the trained government affairs model to generate answers, and obtain generated answers;

[0042] Step 700: Fuse all the generated answers to obtain the target output answer.

[0043] Specifically, the search document is segmented into blocks using a semantic block strategy to obtain text blocks, including:

[0044] Preliminarily segmenting the search document into sentences;

[0045] Using LLM (Large Language Model) to classify the sentences in the search document into two categories to obtain segmentation probability data;

[0046] Calculate the merging probability data and the separating probability data of any two sentences in the search document by using LLM;

[0047] Calculate the difference between the combined probability data and the separated probability data to obtain the margin value; the calculation formula of the margin value is: Margin M (x i )=P split -Pmerge Among them, Margin M (x i ) is the sentence x i The marginal value; the P split is the separation probability data; P merge is the combined probability data;

[0048] The semantic degree of the sentences in the search document is calculated using LLM, and a boundary threshold is set according to the distribution of the semantic degree; the calculation formula of the semantic degree is: Among them, PPL M (x i ) is the sentence x i The semantic degree of the sentence x i Total number of tokens; Represents the sentence x i The Mth token; Represents the sentence x i The semantic degree of the Mth token;

[0049] According to a preset segmentation threshold, a merging threshold and a boundary threshold, the search document is segmented using the segmentation probability data, the marginal value and the semantic degree to obtain the text segmentation.

[0050] Optionally, the retrieval model is any one of BM25 and Anserini.

[0051] Preferably, it also includes:

[0052] The generated answer corresponding to each of the text blocks is cached.

[0053] Specifically, the government affairs knowledge base supports multiple rounds of question and answer for the government affairs big model

[0054] Preferably, it also includes:

[0055] The boundary threshold is dynamically adjusted according to the distribution of the semantic degrees of different retrieved documents.

[0056] Specifically, the concept of Chunk is introduced: the text is divided into blocks with a granularity between sentences and paragraphs, and each Chunk consists of sentences with deep linguistic logical connections.

[0057] Further, edge sampling and block segmentation: Use LLM to classify continuous sentences into two categories, determine whether they need to be segmented, and make decisions based on probability differences. Marginal probability: Use LLM to calculate the probability that two sentences should be merged into one block, and the probability that they should remain separate. Marginal value: The marginal value is the difference between these two probabilities. Threshold: Set a threshold to determine whether to merge two sentences into one block. If the marginal value is less than the threshold, merge the sentences; otherwise, keep them separate. The method is expressed by the formula:

[0058] Margin M (x i )=P split -P merge

[0059] Advantages of the above method: Capturing logical relationships: It can effectively distinguish sentences with similar semantics but different logical relationships, and retain the logical connection between sentences. Reducing model dependence: It has low dependence on the size of the LLM model, and even a small model can effectively segment text. Simple and easy to use: The algorithm is simple and easy to implement, without complex model training or parameter adjustment. Semantic degree segmentation: By analyzing the semantic distribution characteristics of sentences, the boundaries of text blocks can be accurately identified. Semantic degree is a measure of the difficulty of LLM generating text. The lower the semantic degree, the smoother the text generated by LLM and the clearer the semantics. Semantic degree distribution: Analyze the semantic degree distribution characteristics of sentences to find potential block boundaries. The minimum value in the semantic degree distribution often corresponds to the block boundary. Threshold: Set a threshold to determine whether a sentence is located at a block boundary.

[0060] Furthermore, text segmentation: divide the text into sentences. Semantic degree calculation: use LLM to calculate the semantic degree of each sentence in the current context. Distribution analysis: analyze the semantic degree distribution characteristics of sentences to find potential block boundaries. Peak detection, valley detection and other methods can be used. Threshold setting: set a threshold based on the semantic degree distribution to identify block boundaries. The threshold can be set to the mean, median or specific percentage of the semantic degree distribution. Block division: divide the text into multiple blocks based on the semantic degree distribution and threshold. This method is expressed as:

[0061]

[0062] Preferably, dynamic merging strategy: combining the above two strategies, flexibly adjusting the text block size according to user needs, and balancing fine-grained and coarse-grained segmentation. Advantages: More effectively capturing the logical structure of the text: improving the performance of the RAG system in knowledge-intensive tasks. Reduce dependence on the size of the LLM model: enabling small models to effectively segment text. Improve processing efficiency: reduce computing costs, allowing the RAG system to respond to user requests faster. Application scenarios: Question and answer system: help users find the information they need more accurately. Text summarization: automatically generate a concise summary of the text. Machine translation: improve the accuracy and fluency of translation. Content creation: automatically generate various types of text content, such as news reports, product descriptions, etc.

[0063] Specifically, the implementation case is: a large model of government affairs based on RAG, which is used to provide citizens with policy consultation, service guide and other services. In order to improve the accuracy and efficiency of the question-answering system, the semantic degree chunking (Perplexity Chunking) strategy proposed in this embodiment is used to segment the retrieved documents. Implementation steps:

[0064] 1) Construction of government affairs knowledge base: Collect and organize relevant policies, regulations, service guidelines and other documents in the government affairs field to build a government affairs knowledge base.

[0065] 2) Government affairs big model training: Use the government affairs knowledge base to train your government affairs big model so that it can understand the professional knowledge in the government affairs field and generate questions and answers.

[0066] 3) Retrieval model selection: Select a suitable retrieval model, such as BM25 (BestMatching 25), Anserini, etc., to retrieve documents related to the question from the government knowledge base.

[0067] 4) Semantic Chunking: Use the semantic chunking strategy to split the retrieved documents into multiple chunks.

[0068] 5) Government Q&A Generation: Use your government big model to ask and answer questions for each block and generate answers.

[0069] 6) Answer fusion: Fusion the answers from multiple blocks to generate the final answer.

[0070] 7) Government service integration: Integrate the government question and answer system with the government service platform to provide convenient government services to citizens.

[0071] Advantages of the above methods: Government knowledge understanding: Semantic chunking can help the government big model better understand the documents in the government knowledge base and generate more accurate answers. Improved question-answering efficiency: Semantic chunking can divide documents into multiple chunks and answer questions separately, thereby improving the efficiency of the question-answering system. Government service optimization: The government question-answering system can provide citizens with more convenient government services and improve the efficiency of government services. Improved accuracy: Semantic chunking can effectively divide documents into question-related parts and improve the accuracy of the question-answering system. Experimental results show that compared with the traditional text chunking method, semantic chunking can improve the accuracy of the question-answering system by 20%. Improved accuracy of large models: Semantic chunking can provide clearer contextual information for the government big model, help the model better understand the problem, and thus improve the accuracy of the model. Experimental results show that after using semantic chunking, the accuracy of the government big model can be improved by 20%.

[0072] Optionally, case extension:

[0073] 1) Dynamic threshold: According to the semantic distribution characteristics of different documents, the threshold is dynamically adjusted to obtain the best segmentation effect.

[0074] 2) Block-level caching: Cache the answers to each block to avoid repeated questions and answers, further improving efficiency.

[0075] 3) Cross-language application: Apply the semantic segmentation strategy to multilingual documents to achieve cross-language question answering.

[0076] 4) Intelligent recommendation: Recommend relevant government affairs knowledge based on the user’s query history and interests, and provide personalized government affairs services.

[0077] 5) Multi-round Q&A: Supports multi-round Q&A to help users gain a deeper understanding of government affairs knowledge.

[0078] 6) Natural language understanding: Improve the natural language understanding capabilities of the government affairs model so that it can better understand the user's query intent.

[0079] The beneficial effects of the present invention are as follows:

[0080] The present invention performs document segmentation through a semantic segmentation strategy, which can assist the government affairs big model in understanding and retrieving documents, thereby improving the accuracy and efficiency of answers.

[0081] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0082] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A semantic-aware text segmentation technology for a large government model, characterized in that: include: Collect and organize relevant documents in the field of government affairs to obtain a government affairs knowledge base; The relevant documents include: policies, regulations and service guidelines; Using the government affairs knowledge base to train the government affairs big model; Receive question input from target users; Determine a retrieval model, and use the retrieval model to retrieve documents related to the question input in the government affairs knowledge base to obtain retrieval documents; Using a semantic block strategy to segment the retrieved document into blocks to obtain text blocks; Input the text blocks into the trained government affairs model to generate answers, and obtain generated answers; All the generated answers are integrated to obtain the target output answer.

2. According to claim 1, a semantic-aware text segmentation technology for a large government model is characterized in that: The search document is segmented into blocks using a semantic block strategy to obtain text blocks, including: Preliminarily segmenting the search document into sentences; Using LLM to perform binary classification on the sentences in the retrieved document to obtain segmentation probability data; Calculate the merging probability data and the separating probability data of any two sentences in the search document by using LLM; Calculate the difference between the combined probability data and the separated probability data to obtain the margin value; the calculation formula of the margin value is: Margin M (x i )=P split -P merge Among them, Margin M (x i ) is the sentence x i The marginal value; the P split is the separation probability data; P merge is the combined probability data; The semantic degree of the sentences in the search document is calculated using LLM, and a boundary threshold is set according to the distribution of the semantic degree; the calculation formula of the semantic degree is: Among them, PPL M (x i ) is the sentence x i The semantic degree of the sentence x i Total number of tokens; Represents the sentence x i The Mth token; Represents the sentence x i The semantic degree of the Mth token; According to a preset segmentation threshold, a merging threshold and a boundary threshold, the search document is segmented using the segmentation probability data, the marginal value and the semantic degree to obtain the text segmentation.

3. According to claim 1, the semantic-aware text segmentation technology for a large government model is characterized in that: The retrieval model is any one of BM25 and Anserini.

4. According to claim 1, the semantic-aware text segmentation technology for a large government model is characterized in that: Also includes: The generated answer corresponding to each of the text blocks is cached.

5. According to claim 1, the semantic-aware text segmentation technology for a large government model is characterized in that: The government affairs knowledge base supports multiple rounds of question and answer for the government affairs big model.

6. According to claim 2, a semantic-aware text segmentation technology for a large government model is characterized in that: Also includes: The boundary threshold is dynamically adjusted according to the distribution of the semantic degrees of different retrieved documents.

Images