Retrieval enhancement generation method based on token-level word embedding

Abstract

The invention discloses a retrieval enhancement generation method based on token-level word embedding, and relates to the technical field of artificial intelligence and natural language processing. The method specifically comprises the following steps of: segmenting a text block, namely segmenting a document into paragraphs; enhancing the document information; the text coding is used for converting a text into a numerical vector; segmenting the problem to obtain a character array, encoding the character array, and converting the problem into a numerical vector; calculating the similarity between the problem and the text block by adopting the average maximum similarity to obtain a related text block; performing Prompt construction on the question and the related text block to obtain a prompt text; the prompt text is input into a large language model LLMs, and question answers conforming to the answer format are output; and in combination with the question answers and the related text blocks, quoting and marking the question answers to obtain answers with quotations. According to the method, a token-level text embedding method is utilized to vectorize questions and text blocks, so that the accuracy of RAG retrieval is improved, and the confidence of LLMs answers is increased.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence and natural language processing, and in particular to a retrieval enhancement generation method based on token-level word embedding. Background Art

[0002] With the rapid development of deep learning and neural networks, word-based tokenization has been widely used in various NLP tasks, including machine translation, text classification, sentiment analysis, and knowledge question answering.

[0003] Traditional knowledge question answering systems use large language models (LLMs) as the core of question answering. By inputting text questions to LLMs, LLMs can output some text answers. However, this approach also has inherent limitations, including hallucinations, lack of interpretability of generated text, poor understanding of professional domain knowledge, and limited understanding of the latest knowledge. In order to overcome these limitations and improve the capabilities of the model, there are two main approaches: one is to fine-tune (FineTune) to update the model, and the other is to enable them to interact with the outside world and acquire knowledge in different forms and ways. Based on this, a retrieval augmented generation method is proposed, which can realize the functions of real-time knowledge update, reliable knowledge source, and control of LLMs' answer content by plugging in a real-time updated knowledge base.

[0004] The traditional RAG method uses text segmentation to divide the knowledge base into some relatively large text blocks, and then directly performs text encoding (text embedding) on ​​the entire text block. After that, based on the question, the most relevant knowledge or relevant conversation history of the user's question is found, and combined with the original question (query), an information-rich prompt is created to guide the model to generate accurate output.

[0005] Traditional RAG methods may also face the problem of low accuracy for two reasons: first, the text encoding method directly adopts the text block level. This encoding method cannot encode the deeper voice information of the text block, and the results retrieved by similarity may be inaccurate; second, the granularity of the text block is relatively coarse, and its semantic information is likely to be too different from the existence of the question, which will also lead to inaccurate retrieval results. Summary of the invention

[0006] The purpose of the present invention is to provide a retrieval enhancement generation method based on token-level word embedding, and to propose a technology that can improve the retrieval accuracy without affecting the RAG retrieval speed through an innovative method based on token-level encoding and text block information alignment. It solves the problems of untimely update of traditional large-model question-and-answer knowledge, unreliable answer knowledge, hallucination of answers, and inaccurate traditional RAG retrieval methods. This method only acts on the knowledge base preprocessing process, thereby improving accuracy while ensuring speed and promoting the development of the RAG field.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a retrieval enhancement generation method based on token-level word embedding, which adopts character-level token encoding. In natural language processing, tokenization is to decompose a piece of text into smaller units, which are usually words or phrases, but in this solution, we focus on character-level tokens. Character-level tokenization provides a fine-grained way to understand and process text, which is crucial to improving the accuracy of knowledge question-answering systems. This solution further proposes a knowledge question-answering system based on character-level token encoding, which does not require the addition of new knowledge input while maintaining a high operating speed. By tokenizing the input questions and text blocks at the character level and then encoding these tokens using a deep learning model, relevant answers can be retrieved and provided more accurately.

[0008] Specifically include:

[0009] Step S1, text block segmentation, is used to segment the document into paragraphs, specifically including:

[0010] S11, determine the document type of the knowledge base, and parse the document into plain text;

[0011] S12, setting the maximum length of each plain text slice and the maximum overlapping length between slices;

[0012] S13, traversing the plain text of the document, and dividing the document into text blocks using separators;

[0013] Step S2, document information enhancement, specifically includes:

[0014] S21, for the text block obtained in step S1, using large language models LLMs to generate a text block title and hypothetical questions;

[0015] S22, concatenating the text block title, the hypothetical question, and the text content, and performing character-level segmentation, removing stop words, so that each text block has a phrase, and obtaining an enhanced text block;

[0016] Step S3, segmenting the enhanced text block obtained in step S2 to obtain a character array, and performing text encoding to convert the text into a numerical vector to obtain a numerical matrix D;

[0017] Step S4, segmenting the question using a character-level segmentation method to obtain a character array, and encoding the question to convert it into a numerical vector to obtain a numerical matrix Q;

[0018] Step S5, using the average maximum similarity, calculate the similarity between the question and the text block to obtain the relevant text block. The specific formula is:

[0019]

[0020] Where Q = Q 1 Q 2 ...Q n For question, Q i is the word embedding corresponding to the token of the question, Q i =[v 1 ,v 2 ,...,v 768 ], D=D 1 D 2 ...D m For the text block, D j is the word embedding corresponding to the token of the text block, D j =[v 1 ,v 2 ,...,v 768 ];

[0021] Step S6, constructing a prompt by combining the question with the related text block obtained in step S5 to obtain a prompt text;

[0022] Step S7, inputting the prompt text obtained in step S6 into the large language model LLMs, organizing the input question and related text blocks, and summarizing them into answers that conform to the answer format;

[0023] Step S8, combining the answer to the question obtained in step S7 with the relevant text block, marking the answer to the question at the sentence level to obtain a quoted answer.

[0024] Furthermore, the document types of the knowledge base in step S1 include structured formats: PDF and word documents, and unstructured data: web page text or pure document files.

[0025] Furthermore, in step S3, the text is encoded using a Multi-Modal Multi-Task Embedding model.

[0026] Further, the Prompt style described in step S6 is as follows:

[0027] '<Instruction>Answer the question concisely and professionally based on the known information. If no answer can be obtained from it, say "The question cannot be answered based on the known information.",'

[0028] 'Do not allow fabricated content in the answer. The answer should be in Chinese.< / Instruction>\n'

[0029] '<Known Information>{{context}}< / Known Information>\n'

[0030] '<Question>{{question}}< / Question>\n'

[0031] Further, for the large language models LLMs described in step S7, the underlying Transformer model is adopted, and the Transformer model completes the modeling of the global dependencies of the source language sequence and the target language sequence through the attention mechanism.

[0032] Further, step S8 is further as follows:

[0033] S81. Adopt text-level encoding to calculate the similarity between each sentence of the answer and the relevant text block;

[0034] S82. Select the citation number with the highest similarity and higher than the preset similarity threshold as the citation label of the sentence in the answer to the question.

[0035] Advantages of the present invention: Compared with the traditional RAG method, the present invention adopts a more refined token-level text embedding method, divides the text into smaller units, and can capture the semantic and context information of the text more accurately; by adopting a deep learning-based semantic similarity model for similarity calculation, it can more accurately evaluate the matching degree between the question and the text, and improve the retrieval accuracy.

[0036] This solution introduces an answer citation generation method to post-process the answer generated by the model to mark the credibility level of the answer, which helps to screen and recommend the best answer and improve the overall performance of the system; this solution organically combines text embedding, similarity calculation, and answer citation generation to achieve a more comprehensive and efficient question-answer system, and optimizes the system performance at different levels by integrating multiple modules, bringing significant technical advantages to the RAG field. Description of the Drawings

[0037] Figure 1 is the main flowchart of the computer program of the present invention.

[0038] Figure 2It is a diagram of the text segmentation process.

[0039] Figure 3 It is the similarity calculation graph of step S5.

[0040] Figure 4 This is the Transformers model architecture diagram. DETAILED DESCRIPTION

[0041] In the following description, a large number of specific details are provided to provide a more thorough understanding of the present invention. However, it is apparent to those skilled in the art that the present invention can be implemented without one or more of these details. In other examples, in order to avoid confusion with the present invention, some technical features well known in the art are not described.

[0042] A retrieval-enhanced generation method based on token-level word embedding, referring to Figure 1 ,The knowledge base based RAG proposed in this scheme includes the following problems: segmentation and encoding, document loading, document information enhancement, document encoding, similarity calculation, large model question answering, and answer reference generation.,The specific steps are:

[0043] Step 1: Question segmentation and encoding: Use character-level segmentation to segment the question and obtain character-level word phrases. The specific steps are as follows:

[0044] S11. Use Jieba word segmenter to segment the question into individual words;

[0045] S12. Based on the commonly used stop word library, remove the stop words therein to obtain a word array: Q = Q 1 Q 2 …Q n ;

[0046] S13, the text encoder M3E using deep learning encodes each word into a 768-dimensional numerical vector, where Q i The numerical vector is: Q i =[v 1 ,v 2 ,…,v 768 ], all word encodings form a matrix, specifically:

[0047]

[0048] Step S2, document loading, according to the different types of knowledge base documents, the document loading function provided by the open source tool langchain is used to convert the document into text form. This function supports documents in doc / docx, pdf, markdown, csv, xslx, txt, HTML, PPT and other formats. The document types and corresponding functions are shown in the following table:

[0049]

[0050]

[0051] Step S3, document information enhancement, specifically includes:

[0052] S31, using a text segmentation method to segment the plain text into text blocks, and using LLMs to generate titles and hypothetical query questions for the text blocks;

[0053] S32, concatenate the text block title, hypothetical question and text content, perform character-level segmentation, remove stop words, so that each text block has a phrase, and obtain an enhanced text block:

[0054] Reference Figure 2 , preset the maximum length of a text block and the maximum overlap length (256) between adjacent text blocks (1024), and set the key characters for segmentation to "\n", ".", ".", "\t";

[0055] Call the Recursive Character Text Splitter method of langchain to split each text block. To ensure the traceability of information, set each text block to the following format:

[0056] Field Name illustrate doc_id The document's identity id, unique in the current knowledge base text The text of the text block doc_name The name of the document in which the text block is located doc_title The title of the text block, which needs to be generated in subsequent steps syn_question The text block hypothetical query problem requires subsequent steps to generate emb_matrix Numerical matrix, Numerical matrix of text blocks

[0057] Construct an LLMs input through the Prompt project, and the output is the required text block title. The Prompt template is as follows:

[0058] “I will give you an article below. Create a title for the article below.

[0059] This is the article content: {document}

[0060] Note: Only provide a title, no additional text. The answer format starts with 'Title:' followed by the question. Do not write an empty title. "

[0061] The Prompt project constructs an LLMs input, and the output is the required hypothetical query. The Prompt template is as follows:

[0062] “I’ll give you an article below. What search queries or questions are relevant to this article or that this article can answer?

[0063] Each query is on a separate line.

[0064] This is the article content: {document}

[0065] Provide only the user query, without additional text. Format each query as 'Query:' followed by the question. Do not write an empty query. "

[0066] Step S4, document encoding, concatenate the text title, hypothetical question, and text block content of each text block, segment the text block into words using character-level segmentation, and encode each word after removing stop words, specifically:

[0067] S41. Connect the text block title, hypothetical question, and text block content in the following order:

[0068] "Title: {text title}

[0069] Query:{hypothetical query}

[0070] Body: {text content}"

[0071] S42, use jieba word segmenter to segment the concatenated content, and use the common stop word list to remove stop words, and obtain the word array: D = D 1 D 2 ...D m ., convert each word in the word array into a 768-dimensional numerical vector using the M3E text encoding model: D j =[v 1 ,v 2 ,...,v 768 ], all words form a numerical matrix:

[0072]

[0073] Step S5, similarity calculation, using cosine similarity to measure the similarity between characters, so as to effectively capture the directionality and relative relationship between character vectors, so as to more accurately evaluate the similarity between them. When calculating the similarity between the question and the text block, the average maximum similarity method is used to calculate the maximum similarity between each character in the question and the characters in the text block, and the average of these maximum similarities is taken as the overall similarity between the question and the text block;

[0074] Reference Figure 3 , by calculating the similarity between the question encoding and all the text block encodings in the knowledge base, the top 10 text blocks most relevant to the question are selected, specifically:

[0075] For the question encoding matrix: and the text block encoding matrix: Calculate the MaxSim value of Q and D as the text similarity between question Q and text block D, where the calculation formula of MaxSim is:

[0076]

[0077] For all text blocks in the knowledge base: K = D 1 D 2 ...D v , calculate their MaxSim values with question Q respectively to obtain the score vector: S = s 1 s 2 ...s v , for s i sort them from largest to smallest, and select the top 10 s i corresponding text blocks D 1 D 2 ...D 10 ;

[0078] Step S6, large model question answering. Refer to Figure 4 , construct the input of LLMs by using the question and the content of the top 10 most relevant text blocks through Prompt engineering. The output of LLMs is the answer to the given question. The underlying of LLMs uses the Transformers model, and the Transformer structure completely completes the modeling of the global dependencies of the source language sequence and the target language sequence through the attention mechanism. Among them, the template of Prompt is:

[0079] " <Instruction>According to the known information, answer the question concisely and professionally. If you cannot get the answer from it, say 'The question cannot be answered according to the known information',

[0080] Do not add fabricated content to the answer, and the answer should be in Chinese.< / Instruction>

[0081] <Known information>{{The text of text block 1 to text block 10}}< / Known information>

[0082] <Question>{{The question asked}}< / Question>"

[0083] Step S7, answer citation generation. Add citation tags to the answer of LLMs to increase the accuracy of the knowledge question and answer result. Specifically:

[0084] Use "。", "!", "?", "\n", etc. as the end symbols of sentences to perform sentence-level segmentation on the answer of LLMs;

[0085] Each sentence is segmented using the Jieba tokenizer, and after removing stop words, word-level encoding is performed to obtain the numerical matrix S of each sentence:

[0086] Calculate the MaxSim value of sentence S and the 10 related text blocks obtained, and compare them;

[0087] If the maximum MaxSim value of these 10 related text blocks is greater than the preset threshold value of 0.7, it is considered that the sentence needs to be marked with a reference tag, and the reference tag is the sequence number of the text block with the largest MaxSim in the 10 related text blocks. Otherwise, it is considered that the sentence only plays an excessive role and does not need to be marked with a reference tag.

[0088] As described above, although the present invention has been shown and described with reference to specific preferred embodiments, it should not be construed as limiting the present invention itself. Various changes may be made to it in form and detail without departing from the spirit and scope of the present invention as defined in the appended claims.

Claims

1. A retrieval enhancement generation method based on token-level word embedding, characterized in that: It includes the following steps: Step S1, text block segmentation, which is used to split the document into paragraphs, specifically including: S11. Determine the document type of the knowledge base and parse the document into plain text; S12. Set the maximum length of each plain text slice and the maximum overlap length between slices; S13. Traverse the plain text of the document and split the document into text blocks using delimiters; Step S2, document information enhancement, specifically including: S21. For the text blocks obtained in step S1, use large language models (LLMs) to generate text block titles and hypothetical questions; S22. Concatenate the text block titles, hypothetical questions, and text content, and perform character-level segmentation and stop word removal to obtain a phrase for each text block, thereby obtaining enhanced text blocks; Step S3, split the enhanced text blocks obtained in step S2 to obtain character arrays, and perform text encoding to convert the text into numerical vectors, resulting in a numerical matrix D; Step S4, split the question using a character-level segmentation method to obtain a character array, and perform encoding to convert the question into a numerical vector, resulting in a numerical matrix Q; Step S5, use the average maximum similarity to calculate the similarity between the question and the text blocks, and obtain relevant text blocks. The specific formula is: Where, Q=Q1Q2...Q n For question, Q i is the word embedding corresponding to the token of the question, Q i =[v1,v2,...,v 768 ], D=D1D2...D m For the text block, D j is the word embedding corresponding to the token of the text block, D j =[v1,v2,...,v 768 ]; Step S6, construct a Prompt for the question and the relevant text blocks obtained in step S5 to obtain a prompt text; Step S7, input the prompt text obtained in step S6 into the large language model (LLMs). By organizing the input question and relevant text blocks, summarize them into an answer in the format of a question answer; Step S8, combine the question answer obtained in step S7 with the relevant text blocks, and perform sentence-level citation tagging on the question answer to obtain a cited answer; 2. The token-level word embedding-based retrieval enhancement generation method according to claim 1, characterized in that: The document types of the knowledge base described in step S1 include structured formats: PDF and word documents, and unstructured data: web page text or plain document files; 3. The token-level word embedding-based retrieval enhancement generation method according to claim 1, characterized in that: In step S3, the text is encoded using a Multi-Modal Multi-Task Embedding model; 4. The token-level word embedding-based retrieval enhancement generation method according to claim 1, characterized in that: The Prompt style in step S6 is: '<Instruction>Based on the known information, answer the question concisely and professionally. If the answer cannot be obtained from it, say "The question cannot be answered based on the known information",' 'Do not add fabricated content to the answer. The answer should be in Chinese.< / Instruction>\n' '<Known Information>{{context}}< / Known Information>\n' '<Question>{{question}}< / Question>\n' 5. The token-level word embedding-based retrieval enhancement generation method according to claim 1, characterized in that: The large language model (LLMs) described in step S7 uses a Transformers model at the bottom layer, and the Transformers model completes the global dependency modeling of the source language sequence and the target language sequence through an attention mechanism; 6. The token-level word embedding-based retrieval enhancement generation method according to claim 1, characterized in that: Step S8 is further as follows: S81. Use text-level encoding to calculate the similarity between each sentence of the answer and the relevant text blocks; S82. Select the citation number with the highest similarity and higher than the preset similarity threshold as the citation label for the sentence in the answer to the question.

Images