RAG retrieval process optimization method and system based on fine tuning embedding model

Abstract

The invention relates to the technical field of natural language processing, in particular to an RAG retrieval process optimization method and system based on a fine-tuning embedding model, and the method comprises the steps: determining the topic expression distribution degree of each topic in each long text document; obtaining all window sequences of each text segmentation topic based on the topic intensity sequence of each text segmentation topic by adopting a sliding window algorithm, and determining the topic mutation degree of each window sequence of each text segmentation topic based on the difference of the average distribution condition between the first half element and the second half element in each window sequence, short texts under all text segmentation themes are obtained; and determining the text topic concentration ratio of each short text, and optimizing the RAG retrieval process in combination with a large language model. According to the method, the text segmentation quality is improved by analyzing the topic expression distribution conditions of different topics in the long text document.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a RAG retrieval process optimization method and system based on a fine-tuning embedding model. Background Art

[0002] Retrieval Augmented Generation (RAG) is a method that uses a large model to assist data retrieval. The general technical solution for implementing RAG is to obtain text data, segment the text data, vectorize the segmented text data, and then store the segmented text data and vectorized data in the database. When a user asks a question to the large model, the user's question will be matched with the vector of the segmented text data, and then the matched segmented text data will be combined with the user's question to form an enhanced context, and then the large model will generate answers to the question based on the enhanced context. Among them, RAG retrieval generates new question text by performing natural language processing on the user's question, which enables the large model to understand the user's question more accurately, thereby improving the quality of the output answer.

[0003] When using RAG for large-scale model-assisted data retrieval, the data quality of segmented text data is extremely important. For the original text data, since the data is too long, it needs to be segmented, but segmentation easily leads to cross-block information loss, thereby reducing the quality of segmented text data. For cross-block information loss, the traditional segmentation method uses the method of overlapping blocks when segmenting, which easily leads to information duplication and data redundancy, reducing the quality of text segmentation data. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide a RAG retrieval process optimization method and system based on a fine-tuned embedding model. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a RAG retrieval process optimization method based on a fine-tuned embedding model, the method comprising the following steps:

[0006] Obtain each long text document, use a topic model to obtain the probability of various topics in each natural language paragraph in each long text document, and determine the topic strength of various topics in each natural language paragraph by combining the total number of words in each natural language paragraph with the probability;

[0007] Based on the topic strength of each topic in all natural language paragraphs in each long text document, determine the topic strength sequence of each topic in each long text document, and based on the average distribution of all elements in the topic strength sequence, determine the distribution strength of each topic in each long text document; based on the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic and the discrete degree of all elements in the topic strength sequence, determine the distribution concentration of each topic in each long text document, and determine the topic expression distribution degree of each topic in each long text document in combination with the distribution strength;

[0008] Arrange the topic expression distribution degrees of all topics in each long text document in descending order, and then use the topics corresponding to the first preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopt a sliding window algorithm based on the topic strength sequence of each text segmentation topic, obtain all window sequences of each text segmentation topic, and determine the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half of the elements and the second half of the elements in each window sequence, so as to obtain short texts under all text segmentation topics;

[0009] Based on the total number of natural language paragraphs in each short text and the total number of identical natural language paragraphs between each short text and the other short texts, the topic concentration of each short text is determined to obtain the text topic concentration of various short texts, and the RAG retrieval process is optimized in combination with the large language model.

[0010] Preferably, the method for determining the topic strength sequence of each topic in each long text document is:

[0011] Among the multiple topics of all natural language paragraphs in each long text document, the topic strengths of all natural language paragraphs under the same topic are combined into a topic strength sequence of the topic.

[0012] Preferably, the distribution intensity of each topic in each long text document is the mean of all topic intensities of each topic in each long text document.

[0013] Preferably, the distribution concentration of each topic in each long text document is expressed as: Where Md i,j represents the distribution concentration of the jth topic in the long text document i; σ1 i,j Represents the degree of dispersion of all elements in the topic strength sequence of the jth topic in the long text document i; σ2 i,j It represents the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic intensity sequence of the jth topic in the long text document i; α represents a constant preset to be greater than 0.

[0014] Preferably, the topic expression distribution degree of each topic in each long text document is the product of the distribution intensity and the distribution concentration of each topic in each long text document.

[0015] Preferably, the topic mutation degree of each window sequence of each text segmentation topic is the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence of each text segmentation topic.

[0016] Preferably, the obtaining of short texts under all types of text segmentation topics includes:

[0017] Arrange the topic mutation degrees of all window sequences of each text segmentation topic in descending order, and the window sequences corresponding to the first preset number of topic mutation degrees are used as the sequences to be analyzed. Set a dividing point between the last element value in the first half of the elements and the first element value in the second half of the elements in each sequence to be analyzed. Utilize all the dividing points of each text segmentation topic, and organize the natural language paragraphs between two adjacent dividing points in the long text document into short texts under each text segmentation topic.

[0018] Preferably, determining the topic concentration of each short text to obtain the text topic concentration of various short texts includes:

[0019] The topic concentration of the m-th short text β m The expression is: Where Ph m,k represents the total number of identical natural language paragraphs between the mth short text and the kth short text; Ph m The total number of natural language paragraphs in the mth short text; Ph k represents the total number of natural language paragraphs of the k-th short text; K represents the number of short texts under all text segmentation topics;

[0020] Among the short texts under all kinds of text segmentation topics, the short texts containing the same natural language paragraphs are regarded as the same kind of short texts, and the cumulative sum of the topic concentrations of all short texts contained in various short texts is regarded as the text topic concentration of various short texts.

[0021] Preferably, the optimization of the RAG search process includes:

[0022] Arrange the text topic concentrations of all short texts in descending order and select the short text corresponding to the first preset text topic concentration as the final segmented text;

[0023] Obtain domain data, and use the large language model and domain data as input to the adapter fine-tuning method, and output the fine-tuned model;

[0024] Obtain user questions, and use the user questions and the final segmented texts of all long text documents as inputs to the summary extraction algorithm, and output the user question summary and the summary of each final segmented text;

[0025] Calculate the similarity between the user question summary and the summaries of all final segmented texts, arrange the similarities in descending order, and then use the final segmented text corresponding to the second numerical similarity as the input of the fine-tuning model in the RAG retrieval process, and output the question and answer results of the fine-tuning model.

[0026] In the second aspect, an embodiment of the present application also provides a RAG retrieval process optimization system based on a fine-tuned embedded model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein when the processor executes the computer program, the system implements the steps of any one of the above-mentioned methods for optimizing the RAG retrieval process based on a fine-tuned embedded model.

[0027] This application has at least the following beneficial effects:

[0028] This application determines the topic strength sequence of each topic in each long text document based on the topic strength of multiple topics in all natural language paragraphs in each long text document, and determines the distribution strength of each topic in each long text document based on the average distribution of all elements in the topic strength sequence. The beneficial effect is that it can reflect the overall expression strength of the topic in the long text document, which helps to determine the dominant position of the topic in the entire document, so as to facilitate the optimization of subsequent text segmentation and RAG retrieval process;

[0029] The present application determines the distribution concentration of each topic in each long text document by analyzing the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic, as well as the discrete degree of all elements in the topic strength sequence. The beneficial effect is that the distribution concentration can reflect the distribution pattern of the topic in the long text document, which helps to ensure that the segmented short texts are coherent in terms of the topic, thereby improving the quality of text processing, as well as the accuracy and efficiency of retrieval.

[0030] This application determines the distribution of topic expression of each topic in each long text document based on the distribution concentration of each topic in each long text document, combined with the distribution strength. Its beneficial effect is that it quantifies the relative importance of each topic in the long text document, which helps to avoid segmenting important topics into different short texts, thereby improving the quality of text segmentation and improving retrieval accuracy and efficiency.

[0031] The present application arranges the topic expression distribution degrees of all kinds of topics in each long text document in descending order, and then uses the topics corresponding to the first preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopts a sliding window algorithm based on the topic strength sequence of each text segmentation topic, obtains all window sequences of each text segmentation topic, and determines the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half elements and the second half elements in each window sequence, so as to obtain short texts under all kinds of text segmentation topics. The beneficial effect is that the topic mutation degree reflects the difference in average distribution between the front and back parts of the elements in the window sequence. By determining the topic mutation degree of each window sequence of each text segmentation topic, short texts under all kinds of text segmentation topics are obtained, which helps to ensure the coherence of topic expression in the short text;

[0032] This application determines the text topic concentration of each short text based on the total number of natural language paragraphs in each short text and the total number of identical natural language paragraphs between each short text and the other short texts, and optimizes the RAG retrieval process in combination with a large language model. The beneficial effect is that the short text with the more concentrated text topic is selected as the final segmentation result, which can improve the optimization effect of the RAG retrieval process.

[0033] This application analyzes the distribution of topic expressions of different topics in long text documents to obtain short text data with more concentrated and coherent meaning, thereby improving the quality of text segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A flowchart of a method for optimizing a RAG retrieval process based on a fine-tuning embedding model provided in one embodiment of the present application;

[0036] Figure 2 A schematic diagram of a process for extracting the distribution of topic expressions provided in one embodiment of the present application;

[0037] Figure 3 A schematic diagram of a process for extracting a subject mutation degree provided in one embodiment of the present application;

[0038] Figure 4 A schematic diagram of the RAG search process provided for one embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following, in combination with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, features and effects of a RAG retrieval process optimization method and system based on a fine-tuned embedded model proposed in the present application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0040] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0041] The following is a detailed description of a specific solution of a RAG retrieval process optimization method and system based on a fine-tuned embedding model provided by the present application in conjunction with the accompanying drawings.

[0042] See also Figure 1 , which shows a flowchart of a RAG retrieval process optimization method based on a fine-tuning embedding model provided by an embodiment of the present application, the method comprising the following steps:

[0043] Step S1: Obtain each long text document, use a topic model to obtain the probability of various topics in each natural language paragraph in each long text document, and combine the total number of words in each natural language paragraph with the probability to determine the topic strength of various topics in each natural language paragraph.

[0044] For RAG search, the corresponding text database must first be constructed, which is obtained by natural language processing of a large number of long texts. RAG search usually uses text segmentation to complete natural language processing for long texts. In this embodiment, each long text document is obtained from the long text database.

[0045] In the process of constructing the text database for RAG retrieval, it is necessary to segment the long text into multiple short texts. When segmenting the text, it is necessary to ensure that a short text can coherently express a theme, that is, a short text must contain the beginning to the end of a text theme, so as to ensure that the short text can be expressed fluently. Therefore, natural language paragraphs are used as segmentation marks, that is, any natural language paragraph in the long text corresponds to a complete natural language paragraph in the segmented text, which can reduce the incoherence of text meaning caused by segmenting the text from the middle of a natural language paragraph.

[0046] In addition, the expression of a text theme is not isolated from other text themes. For example, a long text has 5 paragraphs, the first 3 paragraphs express theme 1, and the last 3 paragraphs express theme 2. The third paragraph expresses both theme 1 and theme 2. Therefore, there is a natural language paragraph in the text that expresses multiple themes, and there are also two different themes that overlap in some natural language paragraphs. For the situation where two different themes overlap in some natural language paragraphs, the traditional method uses block overlap to segment the text, that is, the first 3 paragraphs are a short text, and the last 3 paragraphs are a short text. The two short texts overlap, which is called block overlap.

[0047] Therefore, in the case where a natural language paragraph expresses multiple topics, this embodiment uses the LDA Dirichlet allocation model to extract keywords from each long text document to extract the complex features of the natural language paragraph when expressing the topic, specifically:

[0048] Each natural language paragraph in each long text document is used as the input of the LDA Dirichlet allocation model, and the probabilities of various topics in each natural language paragraph are output.

[0049] Among them, the LDA Dirichlet model is a well-known technology in the field of natural language processing. The LDA model can estimate the probability that each paragraph in a document belongs to a different topic. The specific process of obtaining the probability of various topics in each natural language paragraph will not be repeated here.

[0050] Since the probability of various topics in each natural language paragraph output by the LDA Dirichlet model is a normalized parameter, that is, the sum of the probabilities of all topics is 1, it represents the proportion of topic expression in the natural language paragraph. However, the probability of the topic does not take into account the impact of the length of the natural language paragraph on the topic expression, that is, the longer the natural language paragraph, the more words it has, the more information it expresses, and the more important it is in reflecting the text topic information. Therefore, this embodiment multiplies the probability of various topics in each natural language paragraph by the total number of words in the natural language paragraph, and records it as the topic strength of various topics in each natural language paragraph.

[0051] Step S2: Based on the topic strengths of multiple topics in all natural language paragraphs in each long text document, determine the topic strength sequence of each topic in each long text document, and based on the average distribution of all elements in the topic strength sequence, determine the distribution strength of each topic in each long text document; based on the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic, and the discrete degree of all elements in the topic strength sequence, determine the distribution concentration of each topic in each long text document, and in combination with the distribution strength, determine the topic expression distribution degree of each topic in each long text document.

[0052] For long texts, they have multiple topics, and the RAG retrieval method needs to segment the text according to the topic. This embodiment obtains the topic information in the long text by calculating the topic strength, but the text topics that need to be paid attention to and segmented in the long text need to be further screened. Therefore, this embodiment extracts the topic expression distribution of the long text document according to the topic distribution in the entire long text, and selects appropriate topics for text segmentation according to the overall topic distribution of the long text, specifically:

[0053] S201: Determine a topic strength sequence for each topic in each long text document based on the topic strengths of multiple topics expressed in all natural language paragraphs in each long text document; determine the distribution strength of each topic in each long text document based on the average distribution of all elements in the topic strength sequence of each topic.

[0054] Each long text document contains multiple natural language paragraphs, each natural language paragraph corresponds to one or more topics. It is uncertain whether there are the same topics in any two natural language paragraphs, and it is also uncertain how many topics are involved in each natural language paragraph.

[0055] Therefore, this embodiment determines the topic strength sequence of each topic in each long text document based on the topic strengths of multiple topics expressed in all natural language paragraphs in each long text document; determines the distribution strength of each topic in each long text document based on the average distribution of all elements in the topic strength sequence of each topic, so as to quantify the topic distribution of each topic and determine the importance of each topic in text segmentation, specifically:

[0056] Among the multiple topics of all natural language paragraphs in each long text document, the topic strengths of all natural language paragraphs under the same topic are combined into a topic strength sequence of the topic;

[0057] Further, the mean of all elements in the topic strength sequence of each topic in each long text document is calculated, and recorded as the distribution strength of each topic in each long text document;

[0058] According to the distribution strength of each topic in each long text document, it can be understood that the larger the mean of all elements in the topic strength sequence, the greater the expression strength of the corresponding topic in the entire long text document, and the more likely it is that the text wants to express the main topic, and the more this topic should be referred to when performing text segmentation; conversely, the smaller the mean of all elements in the topic strength sequence, the smaller the expression strength of the corresponding topic in the entire long text document, and the topic can be ignored when performing text segmentation.

[0059] S202: Determine the distribution concentration of each topic in each long text document based on the discreteness of all elements in the first-order difference sequence of the topic strength sequence of each topic and the discreteness of all elements in the topic strength sequence.

[0060] The same topic may be expressed significantly differently in different natural language paragraphs. This difference may manifest in two situations: first, the topic is significantly concentrated in a specific area of ​​the long text document, indicating that the topic is expressed more densely in this area; second, the expression intensity of the topic in the entire long text document fluctuates greatly, which means that it appears with different frequencies and intensities in different parts of the text. Therefore, by analyzing the discrete degree of all elements in the first-order difference sequence of the topic intensity sequence of each topic, and the discrete degree of all elements in the topic intensity sequence, the distribution concentration of each topic in each long text document is determined to quantify the distribution of each topic in the long text, to ensure that each segmented short text is coherent in the topic when performing text segmentation, to improve the quality of text processing, as well as the accuracy and efficiency of retrieval, specifically:

[0061] The distribution concentration of the jth topic in the long text document i is Md i,j The expression is: Where, σ1 i,j Represents the degree of dispersion of all elements in the topic strength sequence of the jth topic in the long text document i; σ2 i,j Represents the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence of the j-th topic in the long text document i; α represents a constant preset to be greater than 0, which is used to prevent the denominator from being 0, wherein the value of α is artificially set. In this embodiment, the value of α is 0.01. Under the premise of ensuring that the denominator is not 0 and does not excessively affect the calculation results, the implementer can also set it by himself based on the specific situation. This embodiment does not impose any special restrictions.

[0062] It should be noted that there are many methods for measuring the degree of discreteness of a set of data. In this embodiment, the degree of discreteness of all elements in the topic strength sequence is measured by calculating the standard deviation of all elements in the topic strength sequence of each topic in each long text document. The degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence of each topic in each long text document is measured by calculating the standard deviation of the absolute values ​​of all elements in the first-order difference sequence. The implementer may also use methods such as variance or dispersion coefficient to measure the degree of discreteness of data. This embodiment does not impose any special restrictions on the selection of methods for measuring the degree of discreteness of a set of data.

[0063] Among them, the method for obtaining the first-order difference sequence is a well-known technology, and its specific acquisition process will not be repeated here.

[0064] Furthermore, according to the distribution concentration of each topic in each long text document, it can be understood that the greater the degree of discreteness of all elements in the topic strength sequence, the more uneven the distribution of the topic in the text, and the more likely it is that the topic is concentrated in certain paragraphs or fluctuates greatly. The smaller the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence, the less frequent the fluctuation of the topic strength sequence. Therefore, combining the two degrees of discreteness, it can be seen that if the degree of discreteness of all elements in the topic strength sequence is greater and the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence is smaller, the distribution concentration is greater, indicating that the corresponding topic is mainly concentrated in a part of continuous paragraphs in the long text document. On the contrary, if the degree of discreteness of all elements in the topic strength sequence is smaller and the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence is greater, the distribution concentration is smaller, indicating that the corresponding topic fluctuates greatly in the long text document and is more dispersed.

[0065] S203: Determine the topic expression distribution degree of each topic in each long text document based on the distribution concentration and distribution strength of each topic in each long text document.

[0066] Distribution strength and distribution concentration describe the characteristics of topics in text from different perspectives. Distribution strength reflects the overall expression strength of topics in text, while distribution concentration reflects whether the distribution pattern of topics in text is concentrated or dispersed. Combining the distribution strength and distribution concentration, we can get the topic expression distribution to quantify the relative importance of each topic in a long text document, avoid segmenting important topics into different short texts, improve the quality of text segmentation, and thus improve retrieval accuracy and efficiency. Specifically:

[0067] The distribution degree of topic expression of each topic in each long text document is the product of the distribution concentration and the distribution intensity of each topic.

[0068] Furthermore, according to the distribution degree of topic expression of each topic in each long text document, it can be understood that the greater the distribution intensity, the more widely and evenly the corresponding topic is distributed in the long text document, and the greater the topic expression distribution degree; and the greater the distribution concentration, the more concentrated the distribution of the corresponding topic in the text is, and it may appear frequently in certain paragraphs, and the greater the topic expression distribution degree; conversely, the smaller the distribution intensity, the less significant the expression of the corresponding topic in the long text document, and the smaller the topic expression distribution degree; and the smaller the distribution concentration, the more dispersed the corresponding topic is in the text, and there is no obvious high-intensity area, and the topic expression distribution degree will decrease, resulting in the topic not being particularly emphasized when the text is segmented.

[0069] Preferably, the schematic diagram of the topic expression distribution extraction process provided in this embodiment is as follows: Figure 2 shown.

[0070] Step S3: After the topic expression distribution degrees of all the topics in each long text document are arranged in descending order, the topics corresponding to the first preset number of topic expression distribution degrees are used as the text segmentation topics based on each long text document; based on the topic strength sequence of each text segmentation topic, a sliding window algorithm is used to obtain all window sequences of each text segmentation topic, and based on the difference in the average distribution between the first half of the elements and the second half of the elements in each window sequence, the topic mutation degree of each window sequence of each text segmentation topic is determined to obtain the short texts under all the text segmentation topics.

[0071] The long text documents are segmented based on the topic expression distribution in step S2, specifically: the topic expression distribution of all topics in each long text document is arranged in descending order, and then a preset number of topics are selected as text segmentation topics based on each long text document.

[0072] It should be noted that the value of the preset number is artificially set. In this embodiment, the value of the preset number is set to 50. The implementer can also set it by himself according to the specific situation. This embodiment does not impose any special restrictions.

[0073] In particular, if the number of all kinds of topics is less than a preset number, all topics are taken as text segmentation topics of the long text document.

[0074] The text segmentation topic may be expressed at different positions in a long text document. When performing text segmentation, the place where the topic is concentrated should be divided into a short text. Therefore, in this embodiment, a sliding window algorithm is used to obtain all window sequences of each text segmentation topic, and based on the difference in the average distribution between the first half of the elements and the second half of the elements in each window sequence, the topic mutation degree of each window sequence of each text segmentation topic is determined to obtain short texts under all text segmentation topics, specifically:

[0075] A sliding window algorithm is used to divide the topic strength sequence of each text segmentation topic into multiple window sequences to obtain all window sequences of each text segmentation topic. The sliding window size is set to A, the sliding step is set to B, and the sliding window starts from the first element in the topic strength sequence and slides to the last element. Each time it slides, all elements in the window form a window sequence.

[0076] It should be noted that the values ​​of the sliding window size A and the sliding step size B are both manually set. In this embodiment, the value of the sliding window size A is 10, and the value of the sliding step size B is 1. The implementer can also set them according to the specific situation. This embodiment does not impose any special restrictions.

[0077] Furthermore, the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence of each text segmentation topic is taken as the topic mutation degree of each window sequence of each text segmentation topic;

[0078] It should be noted that there are many methods for measuring the difference between data. In this embodiment, the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence is measured by calculating the absolute value of the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence. The implementer may also use other methods for measuring the difference between data, such as ratio. This embodiment does not impose any special restrictions on the selection of the method for measuring the difference between data.

[0079] Preferably, the schematic diagram of the subject mutation degree extraction process provided in this embodiment is as follows: Figure 3 shown.

[0080] Furthermore, the window sequences corresponding to the first preset number of topic mutation degrees among the topic mutation degrees of all window sequences of each text segmentation topic are arranged in descending order as the sequences to be analyzed, and a dividing point is set between the last element value in the first half of the elements and the first element value in the second half of the elements in each sequence to be analyzed. All the dividing points of each text segmentation topic are used to organize the natural language paragraphs between two adjacent dividing points in the long text document into short texts under each text segmentation topic.

[0081] It should be noted that the value of the preset number is set manually. The specific setting process of the preset number in this embodiment is as follows:

[0082] In the formula, F-1 represents the value of the preset number; P represents the total number of natural language paragraphs corresponding to all elements in the window sequence; Q represents a preset value. In this embodiment, the value of Q is 5, which means that when segmenting text, 5 paragraphs tend to constitute a short text. The implementer can also set it according to the specific situation. This embodiment does not impose any special restrictions. Represents the ceiling function.

[0083] It should be understood that the window sequence is obtained based on the segmented topic strength sequence. The topic strength sequence of each topic contains the topic strength of all natural language paragraphs expressing the corresponding topic. Therefore, each element in the window sequence is actually the topic strength. Each topic strength corresponds to a natural language paragraph. Therefore, the above P represents the total number of natural language paragraphs corresponding to all elements in the window sequence.

[0084] This embodiment selects some segmentation methods from these segmentation methods as the final paragraph segmentation strategy. Text segmentation is a common technology in the field of natural language processing, and its specific principle will not be repeated here.

[0085] Step S4: Determine the text topic concentration of each short text based on the total number of natural language paragraphs in each short text and the total number of the same natural language paragraphs between each short text and the other short texts.

[0086] The better the short texts are segmented from the long text, the more concentrated the text topics in the short text are, and the topics expressed in the short text appear less frequently in the adjacent paragraphs of the short text. Therefore, in order to improve the quality of text segmentation, the text topic concentration of each short text is determined based on the total number of natural language paragraphs in each short text and the total number of natural language paragraphs that are the same between each short text and the other short texts, as follows:

[0087] The topic concentration of the m-th short text β m The expression is: Where Ph m,k represents the total number of identical natural language paragraphs between the mth short text and the kth short text; Ph m Indicates the total number of paragraphs in the mth short text; Ph k represents the total number of natural language paragraphs of the kth short text; K represents the number of short texts under all text segmentation topics.

[0088] According to the topic mutation degree of each window sequence of each text segmentation topic, it can be understood that if the total number of identical natural paragraphs between two short texts is larger, the mean number of paragraphs of the two short texts is smaller, it indicates that these short texts contain fewer paragraphs and are more concentrated in distribution, and are more likely to discuss a coherent topic; conversely, if the total number of identical natural paragraphs between two short texts is smaller, the mean number of paragraphs of the two short texts is larger, it indicates that these short texts contain more paragraphs and are more dispersed in distribution, and are less likely to discuss a coherent topic.

[0089] Furthermore, since there may be repeated texts in the short texts under all text segmentation topics, among the short texts under all text segmentation topics, the short texts containing the same natural language paragraphs are regarded as the same type of short texts, and the cumulative sum of the topic concentrations of all short texts contained in various short texts is regarded as the text topic concentration of various short texts.

[0090] It should be noted that for repeated texts in short texts, although the corresponding short texts are the same short texts, they are obtained under different topic divisions. Therefore, the sum of the topic concentrations of the repeated short texts rather than the mean is used to represent the text topic concentration of the short texts, so as to avoid ignoring the topic concentration information from different topics.

[0091] Step S5: Based on the text topic concentration of various short texts, the RAG retrieval process is optimized in combination with the large language model.

[0092] (1) Based on the text topic concentration obtained in step S4, the long text document is divided, specifically: the short text corresponding to the first preset value of text topic concentration among the text topic concentrations of all the short texts in each long text document is arranged in descending order as the final segmented text;

[0093] It should be noted that the value of the preset first value is artificially set. In this embodiment, the value of the preset first value is G. Based on the value obtained in step S4, The implementer may also set it according to the specific situation, and this embodiment does not impose any special restrictions.

[0094] (2) further, obtaining domain data, and using the large language model and the domain data as inputs of the adapter fine-tuning method, and outputting a fine-tuned model;

[0095] It should be noted that there are many commonly used large language models. In this embodiment, a bidirectional encoder representation from a transformer (Bidirectional Encoder Representations from Transformers, BERT) model is used as the input of the adapter fine-tuning method. Implementers may also use other large language models, and this embodiment does not impose any special restrictions.

[0096] Among them, the BERT model and adapter fine-tuning method are both well-known technologies, and their specific principles will not be repeated here.

[0097] Domain data generally refers to a data set related to a specific task or field. These data sets may include industry-specific text data, such as medical literature, legal documents, etc.; or task-specific data, such as product reviews, news reports, etc. The domain data in this embodiment is mainly data in the field of medical question and answer.

[0098] (3) Further, the user question is obtained, and the user question and the final segmented texts of all long text documents are used as inputs of the summary extraction algorithm, and the user question summary and the summary of each final segmented text are output;

[0099] It should be noted that there are many commonly used summary extraction algorithms. This embodiment adopts the TD-IDF term frequency-inverse document frequency algorithm. The implementer can also use the TextRank text ranking algorithm to obtain the text summary. This embodiment does not impose any special restrictions on the selection of the summary extraction algorithm.

[0100] Among them, the TD-IDF term frequency-inverse document frequency algorithm is a well-known technology in the field of natural language processing, and the specific process of obtaining the text summary is not repeated here.

[0101] (4) Calculate the similarity between the user question summary and the summaries of all final segmented texts, arrange the similarities in descending order, and use the final segmented texts corresponding to the second numerical similarity as the input of the fine-tuning model in the RAG retrieval process, and output the question-answering result of the fine-tuning model. It should be noted that the summary is in the form of a vector.

[0102] It should be understood that there are many methods for calculating the similarity between vectors. In this embodiment, the cosine similarity between the user question summary and the summaries of all the final segmented texts is calculated to measure the similarity between the user question summary and the summaries of all the final segmented texts. The implementer may also use other methods for measuring the similarity between vectors, such as the inverse of the Euclidean distance. This embodiment does not impose any special restrictions on the selection of methods for measuring the similarity between vectors.

[0103] The calculation process of cosine similarity is a well-known technology, and its specific calculation steps are not repeated here.

[0104] In addition, it should be understood that the value of the preset second numerical value is manually set. In this embodiment, the value of the preset second numerical value is 3. The implementer can also set it by himself according to the specific situation. This embodiment does not impose any special restrictions.

[0105] So far, this embodiment calculates the distribution intensity according to the occurrence intensity of the topic in the long text, calculates the distribution concentration according to the distribution concentration law of the intensity of different topics in the long text, and obtains the distribution degree of the topic expression in the long text document based on the distribution intensity and distribution concentration. The topics with large distribution intensity and concentrated distribution are regarded as topics with good segmentation effect, and these topics are selected to participate in text segmentation; further, the long text document is segmented based on the dividing point to obtain multiple short texts; and based on the text topic concentration, the short text with more concentrated text topic is selected as the final text segmentation result, and the text segmentation result is added to the RAG retrieval process, which improves the quality of text segmentation, optimizes the RAG retrieval process, and improves the accuracy and efficiency of RAG retrieval.

[0106] Preferably, the RAG search process diagram provided in this embodiment is as follows Figure 4 shown.

[0107] Based on the same inventive concept as the above method, an embodiment of the present application also provides a RAG retrieval process optimization system based on a fine-tuning embedded model, including a memory, a processor, and a computer program stored in the memory and running on the processor, and when the processor executes the computer program, it implements the steps of any one of the above-mentioned RAG retrieval process optimization methods based on a fine-tuning embedded model.

[0108] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0110] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A RAG retrieval process optimization method based on fine-tuning the embedding model, characterized in that: The method comprises the following steps: Obtain each long text document, use a topic model to obtain the probability of various topics in each natural language paragraph in each long text document, and determine the topic strength of various topics in each natural language paragraph by combining the total number of words in each natural language paragraph with the probability; Based on the topic strength of each topic in all natural language paragraphs in each long text document, determine the topic strength sequence of each topic in each long text document, and based on the average distribution of all elements in the topic strength sequence, determine the distribution strength of each topic in each long text document; based on the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic and the discrete degree of all elements in the topic strength sequence, determine the distribution concentration of each topic in each long text document, and determine the topic expression distribution degree of each topic in each long text document in combination with the distribution strength; Arrange the topic expression distribution degrees of all topics in each long text document in descending order, and then use the topics corresponding to the first preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopt a sliding window algorithm based on the topic strength sequence of each text segmentation topic, obtain all window sequences of each text segmentation topic, and determine the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half of the elements and the second half of the elements in each window sequence, so as to obtain short texts under all text segmentation topics; Based on the total number of natural language paragraphs in each short text and the total number of identical natural language paragraphs between each short text and the other short texts, the topic concentration of each short text is determined to obtain the text topic concentration of various short texts, and the RAG retrieval process is optimized in combination with the large language model.

2. A RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The method for determining the topic strength sequence of each topic in each long text document is as follows: Among the multiple topics of all natural language paragraphs in each long text document, the topic strengths of all natural language paragraphs under the same topic are combined into a topic strength sequence of the topic.

3. A RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The distribution strength of each topic in each long text document is the average of all topic strengths of each topic in each long text document.

4. A RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The expression of the distribution concentration of each topic in each long text document is: Where Md i,j represents the distribution concentration of the jth topic in the long text document i; σ1 i,j Represents the degree of dispersion of all elements in the topic strength sequence of the jth topic in the long text document i; σ2 i,j It represents the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic intensity sequence of the jth topic in the long text document i; α represents a constant preset to be greater than 0.

5. The RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The topic expression distribution degree of each topic in each long text document is the product of the distribution intensity and the distribution concentration of each topic in each long text document.

6. A RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The topic mutation degree of each window sequence of each text segmentation topic is the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence of each text segmentation topic.

7. A RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The short texts under all kinds of text segmentation topics are obtained, including: Arrange the topic mutation degrees of all window sequences of each text segmentation topic in descending order, and the window sequences corresponding to the first preset number of topic mutation degrees are used as the sequences to be analyzed. Set a dividing point between the last element value in the first half of the elements and the first element value in the second half of the elements in each sequence to be analyzed. Utilize all the dividing points of each text segmentation topic, and organize the natural language paragraphs between two adjacent dividing points in the long text document into short texts under each text segmentation topic.

8. The RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: Determining the topic concentration of each short text to obtain the text topic concentration of various short texts includes: The topic concentration of the m-th short text β m The expression is: Where Ph m,k represents the total number of identical natural language paragraphs between the mth short text and the kth short text; Ph m The total number of natural language paragraphs in the mth short text; Ph k represents the total number of natural language paragraphs of the k-th short text; K represents the number of short texts under all text segmentation topics; Among the short texts under all kinds of text segmentation topics, the short texts containing the same natural language paragraphs are regarded as the same kind of short texts, and the cumulative sum of the topic concentrations of all short texts contained in various short texts is regarded as the text topic concentration of various short texts.

9. The RAG retrieval process optimization method based on fine-tuning the embedding model as claimed in claim 1, characterized in that: The optimization of the RAG search process includes: Arrange the text topic concentrations of all short texts in descending order and select the short text corresponding to the first preset text topic concentration as the final segmented text; Obtain domain data, and use the large language model and domain data as input to the adapter fine-tuning method, and output the fine-tuned model; Obtain user questions, and use the user questions and the final segmented texts of all long text documents as inputs to the summary extraction algorithm, and output the user question summary and the summary of each final segmented text; Calculate the similarity between the user question summary and the summaries of all final segmented texts, arrange the similarities in descending order, and then use the final segmented text corresponding to the second numerical similarity as the input of the fine-tuning model in the RAG retrieval process, and output the question and answer results of the fine-tuning model.

10. A RAG retrieval process optimization system based on fine-tuning embedded model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of a RAG retrieval process optimization method based on a fine-tuning embedding model as described in any one of claims 1-9 are implemented.

Images