Method and system for enhancing focusing mode context retrieval in knowledge retrieval system

Abstract

The invention discloses a focused mode context retrieval enhancement method and system in a knowledge retrieval system, and the method comprises the steps: carrying out the format conversion and preprocessing of a knowledge document, segmenting a text through a focused overlapping partitioning algorithm, and extracting structural features through a natural language processing technology to generate a directory; the text blocks are converted into semantic vectors through a BERT-Context model, and the semantic vectors are stored in a vector library; during retrieval, text blocks are quickly positioned according to question similarity, a long context is formed through splicing, similarity ranking is calculated, cue words are generated by high-correlation text blocks and questions together, and answers are obtained by inputting the cue words into a generation model. According to the method, the problems of semantic deficiency, incomplete context information and undetailed user retrieval context in the existing knowledge retrieval system can be effectively solved, the accuracy and integrity of knowledge retrieval are improved, and particularly, the method is excellent in performance when processing problems in complex fields.

Description

Technical Field

[0001] The present invention relates to the technical field of knowledge retrieval, and particularly relates to a method and system for enhancing context retrieval in a focused mode in a knowledge retrieval system. Background Art

[0002] Traditional Retrieval-augmented Generation (RAG) systems have many limitations, such as insufficient semantic understanding, lack of context for multiple user inputs, especially the semantic loss caused by document chunking and the lack of user search context, which limits the retrieval accuracy and user experience.

[0003] In traditional RAG systems, document chunking is often performed according to a fixed length or simple rules, such as splitting a document according to a fixed number of tokens. This method does not consider the semantic structure of the document content and may cut a complete semantic unit (such as a paragraph that expounds a complete concept). For example, a paragraph that expounds a certain scientific principle is cut into different text blocks, so that each text block loses part of the information of the complete principle when understood alone, resulting in semantic loss. There is no effective association mechanism between the text blocks after chunking, and they are only simply split without considering the position and mutual relationship of the text blocks in the overall document semantic structure, resulting in the destruction of the overall semantic coherence.

[0004] At the same time, most traditional RAG systems process each search independently and do not fully consider the connection between consecutive user search behaviors. For example, when a user conducts a series of searches on related topics, the system cannot effectively utilize the context information of previous searches. Each search seems to start anew, ignoring the coherence and progression of the user's search intent. When processing search results, the system does not combine the user's question with the context information formed by previous search results and cannot well understand the complete context of the user's search, resulting in the search results may deviate from what the user actually wants.

[0005] Existing methods, including text block clustering and summarization methods and text block semantic supplementation methods, all have deficiencies in efficiency and accuracy to varying degrees. Summary of the Invention

[0006] Object of the Invention: To solve the above technical problems, the present invention provides a method and system for enhancing context retrieval in a focused mode in a knowledge retrieval system. The method adopts a directory structure model to consider the context of the user input and the global information structure of the text blocks, and at the same time introduces a long context fusion model and a BERT-Context model to ensure that the model understands the context of the text blocks during the re-rank stage. The present invention has the characteristics of high efficiency and accuracy, breaking through the constraints of traditional systems in terms of accuracy and efficiency.

[0007] Technical solution: In the first aspect, a method for enhancing context retrieval in the focus mode of a knowledge retrieval system, the method comprising the following steps:

[0008] Perform format conversion and preprocessing on the original knowledge document to generate text data for retrieval;

[0009] Use the focus overlapping chunking algorithm to split the text so that there is an overlapping part between adjacent text chunks, and maintain the integrity and coherence of semantics;

[0010] Adopt natural language processing technology to identify and extract the structural features of the document, form the chapter and section directory of the document, merge similar directory items, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to the text chunks according to the directory structure;

[0011] Use the BERT-Context model to convert the text chunks containing path information into semantic vectors and store them in the vector database;

[0012] In the retrieval stage, quickly focus on the text chunks according to the similarity between the user's question and the directory items, splice multiple candidate text chunks according to the directory path, splice the text chunks in the same article in the original order, and splice the text chunks in different articles according to the similarity of the key sentences in the directory structure to form a long context text, and calculate the similarity score between the long context text and the user's question;

[0013] Sort the text chunks according to the similarity score between the long context text and the user's question to obtain highly relevant text chunks, and construct the highly relevant text chunks and the user's question into prompt words and input them into the generation model to generate the final answer.

[0014] Further, using the focus overlapping chunking algorithm to split the text so that there is an overlapping part between adjacent text chunks, and maintaining the integrity and coherence of semantics, including:

[0015] Initialize an empty list to store text chunks, mark the document content, and initialize the starting position of the current chunk as 0, the chunk number as 1, and the previous chunk as null;

[0016] When the starting position of the current chunk is less than the length of the document content, calculate the ending position of the current chunk, extract the content of the current chunk, if the current chunk is not the first chunk, extract the overlapping content from the previous chunk and pre-set the overlapping content on the current chunk content;

[0017] Apply focus mode and intelligent truncation, including: identifying the nearest semantic boundary in the current block content, truncating the block content to the nearest semantic boundary if the block size exceeds the text block size, updating the current block end position, and filling the block content by adding content from the previous block or the next block if the block size is less than the text block size; updating the current block content with the truncated or filled content;

[0018] Create a text block using the current block information and add the text block to a list;

[0019] Update the variables for the next block, repeat the above process, and finally return a list of text blocks.

[0020] Furthermore, adopt natural language processing techniques to identify and extract the structural features of the document, form the chapter and section directory of the document, merge similar directory entries, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to the text blocks according to the directory structure, including:

[0021] Extract text, titles, chapters, and tables from the document and convert them into structured text data;

[0022] Initialize the directory tree as an empty nested dictionary, for each chapter in the text data, use natural language processing techniques to extract the title, keywords, and key sentences to form directory entries, and add the directory entries to the directory tree according to the title hierarchy;

[0023] For each directory entry in the directory tree, calculate the similarity with other directory entries. If the similarity is greater than the specified threshold, merge the directory entries by combining the attributes of the directory entries, update the directory tree with the merged directory entries, and return a hierarchical directory tree including titles, keywords, and key sentences;

[0024] Add the path information in the directory tree to the corresponding text blocks and adjust and optimize the text blocks across directories.

[0025] Furthermore, the BERT-Context model additionally introduces the parameter directory structure information encoding directory_ids and the overlap part information encoding overlap_mask as input parameters on the basis of the BERT model, adds a context attention module before each transformer layer, and additionally adds a fully connected layer, using Tanh for nonlinear transformation, where the parameter directory_ids is a tensor representing the directory path to which the text block belongs, and overlap_mask is a tensor of the same length as the input sequence representing the overlapping part in the text block.

[0026] Furthermore, the process of the BERT-Context model extracting semantic vectors is as follows:

[0027] First, in the input embedding layer, the standard input embedding is calculated based on the input text block ID, and the directory embedding is calculated by encoding the directory_ids according to the directory structure information. Then, these two embeddings are combined so that the model can perceive the information of both the input text and the directory simultaneously;

[0028] Next, enter the BERT encoder layer. The combined new input embedding is passed to the BERT encoder. Each Transformer layer in the BERT encoder will operate separately. First, the standard Transformer output is calculated, and then the context attention output is calculated by combining the overlap_mask of the overlapping part information. The influence of context information on the output is considered through the attention mechanism. Finally, the encoder output and the context attention output are combined to improve the model's ability to process context;

[0029] After that, it comes to the output optimization layer. The CLS output vector is extracted from the final hidden state, and the CLS output vector is linearly transformed and feature-extracted through a fully connected layer. Then, the Tanh activation function is applied to increase the non-linearity of the output, and dropout is used for regularization operation. Finally, the optimized vector representation for similarity calculation after these processes is returned, that is, the optimized output.

[0030] Furthermore, calculating the similarity score between the long context text and the user's question includes:

[0031] Measuring semantic similarity by calculating the cosine similarity between the user's question vector and the text block vector;

[0032] The semantic similarity is weighted and adjusted according to the importance of the text block in the directory structure and the relevance weight to the user's question. Let the original similarity score of text block T k be S k , its importance weight in the directory structure be α k , and the relevance weight to the user's question be β k . Then the adjusted similarity score S k ' is S k ' = S k ×(α k + β k ), where the importance weight α k of the directory structure is determined according to the directory depth, and the relevance weight β k to the user's question is determined according to the number of keywords and key sentences in the text block that matches the user's question.

[0033] Furthermore, the method for constructing the prompt is as follows:

[0034] Let the user's question be Q, and the set of highly relevant text blocks be {T1, T2, …, T n}, where n is the number of highly relevant text blocks. The prompt P is constructed by the following formula: denotes the text concatenation operation.

[0035] In a second aspect, a context retrieval enhancement system in a focused mode in a knowledge retrieval system includes:

[0036] A knowledge document preprocessing module for performing format conversion and preprocessing on the original knowledge document to generate text data for retrieval;

[0037] A text segmentation module for segmenting the text using a focused overlapping chunking algorithm so that there is an overlapping part between adjacent text blocks, and maintaining semantic integrity and coherence;

[0038] A table of contents structure model processing module for using natural language processing techniques to identify and extract the structural features of the document, form the chapter and section headings of the document, merge similar table of contents items, generate a table of contents structure including titles, keywords, and key sentences, and add corresponding path information to the text blocks according to the table of contents structure;

[0039] An embedding representation module for using the BERT-Context model to convert the text blocks containing path information into semantic vectors and storing them in a vector database;

[0040] A long context fusion module for quickly focusing on text blocks according to the similarity between the user's question and the table of contents items during the retrieval stage, concatenating multiple candidate text blocks according to the table of contents path, concatenating text blocks in the same article in the original order, and concatenating text blocks from different articles according to the similarity of key sentences in the table of contents structure to form a long context text, and calculating the similarity score between the long context text and the user's question;

[0041] An answer generation module for sorting the text blocks according to the similarity score between the long context text and the user's question, obtaining highly relevant text blocks, and constructing a prompt with the highly relevant text blocks and the user's question and inputting it into a generation model to generate a final answer.

[0042] In a third aspect, the present invention also provides a computer device, which includes: one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the program is executed by the processor, it implements the steps of the focused mode context retrieval enhancement method in the knowledge retrieval system as described in the first aspect of the present invention.

[0043] Fourthly, the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for enhancing context retrieval in the focus mode in the knowledge retrieval system according to the first aspect of the present invention are implemented.

[0044] Advantages: Compared with the prior art, the present invention has the following advantages:

[0045] (1) By identifying the document content and extracting the directory entries of the previous search results to generate a directory structure (including titles, keywords, and key sentences), and optimizing the text chunking in combination with this directory structure during the text chunking stage, the text chunks can follow the original semantic structure of the document, avoiding the situation of splitting complete semantic units, thus retaining the global structure information of the text chunks and helping to restore the semantic integrity of the document. When chunking, corresponding directory chain information is added to each text chunk, which enables each text chunk to establish connections with other text chunks through the directory structure during subsequent processing, enhancing the semantic association between text chunks.

[0046] (2) In the retrieval stage, the long context fusion model focuses on the text chunks according to the directory structure, and splices all the text chunks under the most relevant directory in the original order or according to relevance to form a long context text. Splicing the text chunks under the same chapter together can recombine the scattered semantic units, restore the semantic coherence that may be lost due to chunking, and solve the problem of semantic loss. Moreover, while ensuring that there is no need to store articles separately to save storage resources, the relevance between multiple similar articles is increased, improving the accuracy and credibility of retrieval. Through the focus mode, the computational burden and inference efficiency are also reduced, optimizing the system operation efficiency and reducing the hardware resource requirements.

[0047] (3) When outputting the answer, the spliced long context text and the user's question are input into the model to calculate the similarity score. This method considers the user's question in the context of the long context, rather than processing the user's question in isolation, thus taking into account the complete context of the user's search and avoiding search result deviations caused by context loss. Description of the Drawings

[0048] Figure 1 is a schematic flowchart of context knowledge enhanced search according to an embodiment of the present invention;

[0049] Figure 2 is a schematic architecture diagram of context knowledge enhanced search according to an embodiment of the present invention;

[0050] Figure 3 is a directory structure generation framework according to an embodiment of the present invention;

[0051] Figure 4It is a long context fusion model framework according to an embodiment of the present invention. Detailed implementation mode

[0052] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below through specific implementations and in conjunction with the accompanying drawings.

[0053] The present invention proposes a method for enhancing context retrieval in the focus mode in a knowledge retrieval system, aiming to solve problems such as semantic loss, ambiguity caused by document chunking, and the lack of the user's continuous search context in existing retrieval-augmented generation (RAG) systems. Different from the traditional RAG system's processing of documents, the present invention introduces the focus mode, that is, adopts a directory structure model, generates a directory structure in the form of titles, keywords, and key sentences by identifying the document content and extracting the directory entries of the previous search results, so as to consider the context of the user input. The present invention optimizes chunking in combination with the directory structure at the text chunking stage, and then through a long context fusion model, splices all text chunks under the most relevant directory in the original order or according to relevance to form a long context text, and inputs it together with the user's question into the model for similarity calculation, thereby restoring the semantic integrity of the document and enhancing the relevance between multiple articles.

[0054] Referring to Figure 1 and Figure 2 , a method for enhancing context retrieval in the focus mode in a knowledge retrieval system proposed by the present invention includes the following steps:

[0055] Step S1: Perform format conversion and preprocessing on the original knowledge document to generate text data available for retrieval;

[0056] Step S2: Use the focus overlapping chunking algorithm to chunk the text, so that there is an overlapping part between adjacent text chunks, and maintain the semantic integrity and coherence, and improve the recall ability of the retrieval model for continuous text chunks;

[0057] Step S3: Use natural language processing technology to identify and extract the structural features of the document, form the chapter and section directory of the document, merge similar directory entries, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to the text chunks according to the directory structure;

[0058] Step S4: Use the BERT-Context model to convert the text chunks containing path information into semantic vectors and store them in the vector database;

[0059] Step S5: In the retrieval stage, quickly focus on text chunks according to the similarity between the user's question and the catalog items, splice multiple candidate text chunks according to the catalog path. For text chunks in the same article, they are spliced in the original order, and for text chunks in different articles, they are spliced according to the similarity of key sentences in the catalog structure to form a long context text, and calculate the similarity score between the long context text and the user's question;

[0060] Step S6: Sort the text chunks according to the similarity score between the long context text and the user's question to obtain highly relevant text chunks, and construct the highly relevant text chunks and the user's question into a prompt, and input it into the generation model to generate the final answer.

[0061] According to an embodiment of the present invention, in step S1, first, through document parsing, various knowledge documents are converted into text data or pure text format that can be processed by downstream models. Then, text chunking is performed, and the focused overlapping chunking method is used to split the text, so that adjacent text chunks are more likely to be recalled together in the retrieval stage, thereby providing a basis for subsequent context restoration. Finally, vectorization processing is performed, and the text chunks are converted into semantic vectors through the BERT-Context model and stored in the vector database. To support subsequent efficient retrieval operations, the system can quickly find text chunks similar to the semantic vector of the user's question.

[0062] As an example, the data structure of the document data is as follows.

[0063] Data Structure 1 Document Data Data Structure

[0064]

[0065] In step S2, when performing text chunking, the overlapping length of adjacent text chunks is determined by the overlapping ratio. For example, the overlapping length of adjacent text chunks can be 1 / n of the text chunk length, where n can be regarded as the overlapping ratio factor. Let the original text length be L and the text chunk length be W, then the number of text chunks N is obtained according to formula (1).

[0066]

[0067] According to an embodiment of the present invention, the focused overlapping segmentation algorithm is used to divide the text into chunks. The preset length of the text chunks is 800 tokens, the initial overlapping ratio is 0.2, intelligent truncation is used to ensure semantic integrity, and the overlapping ratio is readjusted for text chunks exceeding the predetermined length. The specific implementation method of the focused overlapping segmentation algorithm is as follows:

[0068] First, create an empty list to store the segmented text chunks later. Then, tokenize the input document content, for example, split it into words or sub - words, etc. Next, initialize some variables, including the current starting position as 0, the chunk number as 1, and the previous chunk initialized as null.

[0069] When the current starting position is still less than the length of the document content, start the loop processing. In each loop, first calculate the ending position of the current chunk, which is the current starting position plus the specified chunk size. Then extract the content of the current chunk, from the current starting position to the ending position of the document content.

[0070] If this is not the first chunk (chunk number is greater than 1), extract the overlapping content from the previous chunk, that is, take the specified overlapping size length from the end of the previous chunk's content, and then prefix the overlapping content to the front of the current chunk's content.

[0071] Next, apply the focusing mode and intelligent truncation logic. Identify the nearest semantic boundary (usually the end of a sentence) in the current chunk's content. If the size of the current chunk exceeds the maximum tokens limit, truncate the chunk content to the nearest semantic boundary within the limit range and update the current chunk's ending position accordingly; if the chunk size is less than the maximum tokens limit, fill the chunk content by adding content from the previous chunk or the next chunk, and ensure that the filled content can maintain semantic integrity.

[0072] After that, add directory path information to the current chunk. Obtain the directory path by calling a specific function based on the current starting position, ending position, and the directory tree. The directory path added here is empty. This is to ensure the consistency of the global text chunk data structure. The generation of the specific directory structure is described in detail below. Then, create a TextChunk object using the information of the current chunk and add it to the previously created list.

[0073] Finally, update the relevant variables for the next chunk. Set the current chunk as the previous chunk, update the current starting position to the current ending position minus the overlapping size, and increment the chunk number by 1. After the loop ends, return the list containing all the segmented text chunk objects.

[0074] The pseudocode of the focused overlapping segmentation algorithm is given as follows:

[0075] Pseudocode 1 Focused Overlapping Chunking Algorithm

[0076]

[0077]

[0078] The text chunk data structure is shown in Data Structure 2.

[0079] Data Structure 2 Text Block Data Structure

[0080]

[0081] In step S4 of the present invention, the BERT-Context model is used to convert the text block into a semantic vector. The BERT-Context model additionally introduces the parameter directory structure information encoding directory_ids and the overlap part encoding overlap_mask on the basis of the BERT model, and adds a context attention module before each transformer layer. At the same time, a fully connected layer is added specifically, and Tanh is used for non-linear transformation. The parameter directory_ids is a list of integers, representing the directory path to which the text block belongs, and is a one-dimensional tensor in the form of [seq_len]. The ids are mapped from the paths of each chapter and sub-chapter, in the form of 1, 2, etc. The overlap_mask is a tensor with the same length as the input sequence, and is a two-dimensional tensor with the shape of [batch_size, seq_len]. It marks the token positions of the overlapping parts, with 1 representing the overlapping parts and 0 representing the non-overlapping parts. The overlap_mask is determined by the overlapping parts in the text block. The specific method is to initialize a mask tensor of all zeros with the shape of [batch_size, seq_len], and for each text block, the overlapping parts are marked as 1. The specific processing process of the BERT-Context model is as follows:

[0082] First, in the input embedding layer, the standard input embedding is calculated according to the input text block ID, and at the same time, the directory embedding is calculated according to the directory ID. Then, these two embeddings are combined so that the model can perceive the information of the input text and the directory at the same time.

[0083] Next, enter the BERT encoder layer, and pass the combined new input embedding to the BERT encoder. Each Transformer layer in the BERT encoder will operate separately. First, calculate the standard Transformer output, then calculate the context attention output, and consider the influence of the context information on the output through the attention mechanism in combination with the overlap_mask. Finally, combine the encoder output and the context attention output to further improve the model's ability to process the context.

[0084] After that, it reaches the output optimization layer, and extracts the CLS token output from the final hidden state (in the input of BERT, the CLS embedding vector is used to represent the semantic information of the entire sequence). The CLS output is further linearly transformed and feature-extracted through a fully connected layer. Then, an activation function such as Tanh is applied to increase the non-linearity of the output. Next, dropout is used for regularization operations to prevent overfitting. Finally, the optimized vector representation for similarity calculation after these processes is returned, which is the optimized output.

[0085] Among them, the process of the directory embedding layer: Initialize an embedding layer with specific dimensions (num_directory_membeddings, hidden_size), which is like a dictionary and can map different directory_ids to corresponding embedding vectors. Then, according to the input directory_id, the directory embedding is calculated through the embedding layer function EmbeddingLayer, that is, the embedding vector corresponding to this directory ID is obtained, and finally this directory embedding is returned.

[0086] The process of the context attention module: First, perform linear transformations on the hidden state respectively to calculate the query, key, and value vectors, which are used for subsequent attention mechanism calculations. Then, calculate the attention scores, which are obtained by multiplying the query by the transpose of the key and dividing by a scaling factor sqrt(d_k), reflecting the correlation between different positions. Next, apply the overlap_mask to the attention scores to consider the impact of the overlapping part on the attention and adjust the attention mechanism. Then, use Softmax to normalize the adjusted attention scores and convert them into a probability distribution to select relevant context information. Finally, calculate the context output by multiplying the normalized attention probabilities by the value vectors, that is, the output that comprehensively considers the attention mechanism and value vector information, and return this context output.

[0087] The algorithm code of the BERT-Context model is as follows:

[0088] Pseudo-code 2 BERT-Context

[0089]

[0090]

[0091] The present invention can identify the text and image information in the document through the directory structure model (the framework is shown in Figure 3 ), summarize the content of the previous structure directory, and generate a chapter and section directory tree in the form of titles, keywords, and key sentences to retain the global structure information of the text block and the user retrieval context.

[0092] Specifically, the optimization of the directory structure model includes the following steps:

[0093] Text parsing: Use natural language processing technology (NLP) and optical character recognition technology (OCR) to identify and extract the structural features of the document. Extract key sentences by evaluating sentence importance scores using TF-IDF. Use BERT-based to extract document keywords. The keywords and key sentences extracted here are used to generate the data structure of the table of contents items, so they are extracted from the text content contained in this directory. The data structure of the table of contents items is title, keyword, and key sentence. Through in-depth parsing of the document, text parsing obtains the hierarchical structure information of the document, providing basic data for the generation of the table of contents tree.

[0094] Table of contents merging: Calculate the semantic similarity between table of contents items and merge those with high similarity. After converting the table of contents items into semantic vectors using the BERT-Context model, calculate the cosine similarity between each pair of table of contents items (see Formula 2, where the semantic vectors corresponding to table of contents items D i , D j are v i and v j ), and perform similarity aggregation on the obtained results to get the overall similarity.

[0095]

[0096] Merge table of contents items with a similarity higher than 0.7.

[0097] Table of contents tree generation: Generate a table of contents tree based on the table of contents items generated from the document and the previously generated table of contents results (see Data Structure 3 for the table of contents data structure). The directory structure is stored in the form of a nested dictionary.

[0098] Data Structure 3 Table of Contents Data Structure

[0099]

[0100] Optimized chunking: Optimize the chunking of the text in combination with the directory structure to enhance the context of the text chunks. At the same time, add the corresponding table of contents chain information in front of each text chunk so that each text chunk retains its hierarchical identifier in the original text. After splitting, the text chunks only contain an empty directory structure, and the text chunks themselves may contain content from different directories. Therefore, after completing the directory structure, while adding table of contents information to the text chunks, the optimization module determines whether the text chunks cross directories. If they do, further optimization is performed.

[0101] The algorithm of the directory structure model is shown in Pseudocode 3.

[0102] Pseudocode 3 Directory Structure Model

[0103]

[0104] In the retrieval stage, the present invention can focus on the directory structure through a long context fusion model (the framework is shown in Figure 4 ), process multiple text blocks simultaneously, and assign more accurate similarity scores to each text block by evaluating the semantic associations between text blocks. It mainly includes several parts: focusing on the directory, text splicing, and similarity calculation.

[0105] First, focus on the directory and quickly locate the directory with the greatest similarity to the search content. The methods include: merging the directory structures of directory entries, converting the text into vectors using TF-IDF, calculating the similarity between the user's question and directory entries, and taking the top 10 directory entries with the highest similarity as candidates; using the BERT-Context model to encode the text of the user's question and candidate directory entries into semantic vectors, calculating the cosine similarity between the vectors, and selecting the directory entry with the highest similarity as the final matching result.

[0106] Then, according to the directory structure information, splice the retrieved multiple text blocks. For text blocks in the same article, splice them in the original order, and for text blocks in different articles, splice them according to the similarity of key sentences in the directory structure to form a long context text. The data structure of text blocks is shown in Data Structure 4.

[0107] Data Structure 4 Data Structure of Long Context Fusion Text Blocks

[0108]

[0109] The methods of text splicing include: focusing on selecting the text blocks most relevant to the current task, using the MoBA mechanism and the Late Chunking method to reduce the computational burden and inference efficiency. Use a transformer for semantic fusion to restore the context. Calculate the cosine similarity of key sentences in the directory structure of text blocks and splice text blocks from different articles.

[0110] Perform similarity calculation: Calculate the similarity score between the long context text and the user's question, including: using the BERT-Context model to perform semantic encoding on text blocks to form semantic vectors. Measure semantic similarity by calculating the cosine similarity between the user's question vector and the text block vector. Adjust the similarity by weighting according to the importance of the text block in the directory structure and its relevance to the user's question. Let the original similarity score of text block T k be S k , its importance weight in the directory structure be α k , and its relevance weight to the user's question be β k , then the adjusted similarity score S k ' is obtained from formula (3).

[0111] S'k = S k × (α k + β k ) (3) The importance weight α of the directory structure k is determined according to the directory depth. For example, the top level is 1.0 and the second level is 0.8. The keyword weight is determined according to the number of keyword occurrences in the text block. For example, for each additional keyword, the weight increases by 0.1, and the same applies to key sentences. Denote the sum of the two weights as ω k , ω k = α k + β k , and its specific formula is as follows:

[0112] ω k = α1·f1(Q, T k ) + α2·f2(Q, T k ) + α3·f3(Q, T k ) (4)

[0113] where Q is the user's question, f represents the feature function, f1 is the keyword matching degree, f2 is the key sentence matching degree, f3 is the directory structure depth, and the weight coefficients α 1， α 2， α3 are 0.5, 0.5, and 0.8 respectively.

[0114] The algorithm of the long context fusion model is shown in Pseudocode 4.

[0115] Pseudocode 4 Long Context Fusion Model

[0116]

[0117]

[0118] In step S6 of the present invention, the text blocks are sorted, and the highly relevant text blocks and the user's question are constructed into prompt words, which are input into the generation model to generate the final answer. The generation model used is a pre-trained large language model such as GPT-4. Specifically, it includes:

[0119] Sort the text blocks according to the similarity scores to obtain a set of highly relevant text blocks; construct the highly relevant text blocks and the user's question into prompt words. Let the user's question be Q and the set of highly relevant text blocks be {T1, T2…, T n}, and the prompt word P can be constructed by formula 4, where represents the text splicing operation:

[0120]

[0121] Generate the final answer according to the prompt. Further, grammar checking, logical verification, and format adjustment can be performed on the generated answer. The generated answer can be optimized using natural language processing technology (NLP), and a user feedback mechanism can be provided to optimize the performance of the model.

[0122] The data structure of the answer is shown in Data Structure 5.

[0123] Data Structure 5 Generation Module Data Structure

[0124]

[0125] The present invention proposes a new directory structure model. By retrieving documents and merging existing directory structures, a new directory structure in the form of titles, keywords, and key sentences is formed. A focusing mode is incorporated to improve efficiency.

[0126] The context fusion process proposed by the present invention can achieve semantic restoration of text blocks. While ensuring that articles do not need to be stored separately, the relevance between multiple similar articles is increased, and the accuracy and credibility of retrieval are improved.

[0127] The BERT-Context proposed by the present invention, based on the BERT model, pays more attention to context semantics and improves the accuracy of retrieval.

[0128] Another embodiment of the present invention provides a focusing mode context retrieval enhancement system in a knowledge retrieval system, including:

[0129] A knowledge document preprocessing module for performing format conversion and preprocessing on the original knowledge document to generate text data available for retrieval;

[0130] A text segmentation module for segmenting the text using the focusing overlapping block algorithm, so that there is an overlapping part between adjacent text blocks, and the semantic integrity and coherence are maintained;

[0131] A directory structure model processing module for using natural language processing technology to identify and extract the structural features of the document, form the chapter and section directory of the document, merge similar directory items, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to the text blocks according to the directory structure;

[0132] An embedding representation module for using the BERT-Context model to convert the text blocks containing path information into semantic vectors and store them in a vector database;

[0133] A long context fusion module is used to quickly focus on text blocks according to the similarity between the user's question and the table of contents items during the retrieval stage, splice multiple candidate text blocks according to the table of contents path. The text blocks in the same article are spliced in the original order, and the text blocks in different articles are spliced according to the similarity of key sentences in the table of contents structure to form a long context text, and calculate the similarity score between the long context text and the user's question;

[0134] An answer generation module is used to sort the text blocks according to the similarity score between the long context text and the user's question, obtain highly relevant text blocks, and construct the highly relevant text blocks and the user's question into a prompt, and input it into the generation model to generate the final answer. Through the design of these modules, the accuracy and efficiency of knowledge search can be improved, and the problem of the lack of multi-question context for users can be solved.

[0135] It should be understood that the focus mode context retrieval enhancement system in the knowledge retrieval system in the embodiments of the present invention can implement all the technical solutions in the above method embodiments. The functions of its various components can be specifically implemented according to the focus mode context retrieval enhancement method in the knowledge retrieval system in the above method embodiments. The specific implementation process can refer to the relevant descriptions in the above embodiments and will not be elaborated here.

[0136] The present invention also provides a computer device, which includes: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors. When the program is executed by the processor, it implements the steps of the focus mode context retrieval enhancement method in the knowledge retrieval system as described above.

[0137] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the focus mode context retrieval enhancement method in the knowledge retrieval system as described above.

[0138] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device (system), a computer device, or a computer program product. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0139] The present invention is described with reference to the flowchart of a method according to an embodiment of the present invention. It should be understood that each process in the flowchart and the combination of processes in the flowchart can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in one process Figure 1 or a plurality of processes.

[0140] These computer program instructions can also be stored in a computer-readable memory capable of guiding a computer or other programmable data processing devices to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one process Figure 1 or a plurality of processes.

[0141] These computer program instructions can also be loaded onto a computer or other programmable data processing devices, so that a series of operation steps are executed on the computer or other programmable devices to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable devices provide steps for implementing the functions specified in one process Figure 1 or a plurality of processes.

Claims

1. A method for enhancing context retrieval in the focus mode of a knowledge retrieval system, characterized in that, It includes the following steps: Perform format conversion and preprocessing on the original knowledge document to generate text data for retrieval; Use the focused overlapping chunking algorithm to split the text so that there is an overlapping part between adjacent text chunks, and maintain the integrity and coherence of semantics; Adopt natural language processing techniques to identify and extract the structural features of the document, form the chapter and section directory of the document, merge similar directory entries, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to the text chunks according to the directory structure; Use the BERT-Context model to convert the text chunks containing path information into semantic vectors and store them in the vector database; In the retrieval stage, quickly focus on the text chunks according to the similarity between the user's question and the directory entries, splice multiple candidate text chunks according to the directory path, splice the text chunks in the same article in the original order, and splice the text chunks in different articles according to the similarity of the key sentences in the directory structure to form a long context text, and calculate the similarity score between the long context text and the user's question; Sort the text chunks according to the similarity score between the long context text and the user's question to obtain highly relevant text chunks, and construct the highly relevant text chunks and the user's question into prompt words and input them into the generation model to generate the final answer.

2. The method according to claim 1, characterized in that, Use the focused overlapping chunking algorithm to split the text so that there is an overlapping part between adjacent text chunks, and maintain the integrity and coherence of semantics, including: Initialize an empty list to store text chunks, mark the document content, and initialize the starting position of the current chunk to 0, the chunk number to 1, and the previous chunk to null; When the starting position of the current chunk is less than the length of the document content, calculate the ending position of the current chunk, extract the content of the current chunk, if the current chunk is not the first chunk, extract the overlapping content from the previous chunk and pre-set the overlapping content on the current chunk content; Apply the focused mode and intelligent truncation, including: identify the nearest semantic boundary in the current chunk content, if the chunk size exceeds the text chunk size, truncate the chunk content to the nearest semantic boundary and update the current chunk ending position, if the chunk size is less than the text chunk size, fill the chunk content by adding content from the previous chunk or the next chunk; update the current chunk content with the truncated or filled content; Create a text chunk using the current chunk information and add the text chunk to the list; Update the variables of the next chunk, repeat the above process, and finally return the list of text chunks.

3. The method according to claim 1, characterized in that, Adopt natural language processing techniques to identify and extract the structural features of the document, form the chapter and section directory of the document, merge similar directory entries, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to the text chunks according to the directory structure, including: Extract text, titles, chapters, and tables from the document and convert them into structured text data; Initialize the directory tree as an empty nested dictionary, for each chapter in the text data, use natural language processing techniques to extract titles, keywords, and key sentences to form directory entries, and add the directory entries to the directory tree according to the title hierarchy; For each directory entry in the directory tree, calculate the similarity with other directory entries. If the similarity is greater than the specified threshold, merge the directory entries by combining their attributes, update the directory tree with the merged directory entries, and return a hierarchical directory tree containing titles, keywords, and key sentences. Add the path information in the directory tree to the corresponding text blocks and adjust and optimize the text blocks across directories.

4. The method according to claim 1, wherein The BERT-Context model additionally introduces the parameter directory structure information encoding directory_ids and the overlapping part information encoding overlap_mask as input parameters on the basis of the BERT model, adds a context attention module before each transformer layer, and additionally adds a fully connected layer, using Tanh for non-linear transformation. Among them, the parameter directory_ids is a tensor representing the directory path to which the text block belongs, and overlap_mask is a tensor of the same length as the input sequence representing the overlapping part in the text block.

5. The method according to claim 4, wherein The process of the BERT-Context model extracting semantic vectors is as follows: First, in the input embedding layer, calculate the standard input embedding according to the input text block ID, and calculate the directory embedding according to the directory structure information encoding directory_ids. Then combine these two embeddings so that the model can perceive the information of the input text and the directory at the same time. Next, enter the BERT encoder layer. Pass the combined new input embedding to the BERT encoder. Each Transformer layer in the BERT encoder will operate separately. First, calculate the standard Transformer output, then calculate the context attention output by combining the overlapping part information encoding overlap_mask, consider the influence of context information on the output through the attention mechanism, and finally combine the encoder output and the context attention output to improve the model's ability to process context. After that, go to the output optimization layer. Extract the CLS output vector from the final hidden state, perform linear transformation and feature extraction on the CLS output vector through the fully connected layer, then apply the Tanh activation function to increase the non-linearity of the output, and then use dropout for regularization operation. Finally, return the optimized vector representation for similarity calculation after these processes, that is, the optimized output.

6. The method according to claim 1, wherein Calculate the similarity score between the long context text and the user's question, including: Measure semantic similarity by calculating the cosine similarity between the user's question vector and the text block vector. Adjust the semantic similarity by weighting according to the importance of the text block in the directory structure and the relevance weight of the user's question. Let the text block be T k with the original similarity score of S k , its importance weight in the directory structure is α k , and the relevance weight to the user's question is β k . Then the adjusted similarity score S k ' is S k ′ = S k ×(α k + β k ), where the importance weight α k of the directory structure is determined according to the directory depth, and the relevance weight β k to the user's question is determined according to the number of keywords and key sentences in the text block that matches the user's question.

7. The method according to claim 1, characterized in that, The method for constructing prompts is as follows: Let the user's question be Q, and the set of highly relevant text blocks be {T1, T2, …, T n}, where n is the number of highly relevant text blocks. The prompt P is constructed by the following formula: denotes the text concatenation operation.

8. A context retrieval enhancement system for the focused mode in a knowledge retrieval system, characterized in that, Including: A knowledge document preprocessing module for performing format conversion and preprocessing on the original knowledge document to generate text data available for retrieval. A text segmentation module for segmenting the text using the focused overlapping chunking algorithm so that there is an overlapping part between adjacent text blocks and the semantic integrity and coherence are maintained. A directory structure model processing module, which is used to identify and extract the structural features of a document by using natural language processing technology, form the chapter and section directory of the document, merge similar directory entries, generate a directory structure including titles, keywords, and key sentences, and add corresponding path information to text blocks according to the directory structure; An embedding representation module, which is used to convert text blocks containing path information into semantic vectors by using the BERT-Context model and store them in a vector database; A long context fusion module, which is used to quickly focus on text blocks according to the similarity between the user's question and directory entries during the retrieval phase, splice multiple candidate text blocks according to the directory path, text blocks in the same article are spliced in the original order, and text blocks in different articles are spliced according to the similarity of key sentences in the directory structure to form a long context text, and calculate the similarity score between the long context text and the user's question; An answer generation module, which is used to sort text blocks according to the similarity score between the long context text and the user's question, obtain highly relevant text blocks, and construct the highly relevant text blocks and the user's question into prompt words, and input them into a generation model to generate a final answer.

9. A computer device, characterized in that, The computer device includes: one or more processors; a memory; and one or more programs, where the one or more programs are stored in the memory and are configured to be executed by the one or more processors, and when the program is executed by the processor, it implements the steps of the enhanced method for context retrieval in the focused mode in the knowledge retrieval system according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the enhanced method for context retrieval in the focused mode in the knowledge retrieval system according to any one of claims 1-7.

Images