A RAG retrieval process optimization method and system based on fine-tuning embedding model

Abstract

The present application relates to the field of natural language processing technology, and specifically to a RAG retrieval process optimization method and system based on a fine-tuned embedding model, the method comprising: determining the distribution of topic expressions for each topic within each long text document; using a sliding window algorithm to obtain all window sequences of each text segmentation topic based on the topic strength sequence of each text segmentation topic, and determining the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half and the second half of the elements in each window sequence, so as to obtain short texts under all text segmentation topics; determining the text topic concentration of each short text, and optimizing the RAG retrieval process in combination with a large language model. The present application improves the quality of text segmentation by analyzing the distribution of topic expressions of different topics in long text documents.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a RAG retrieval process optimization method and system based on a fine-tuned embedding model. Background Art

[0002] Retrieval Augmented Generation (RAG) is a method that uses a large model to assist in data retrieval. The general technical solution for implementing RAG is: obtain text data, then segment the text data, vectorize the segmented text data, and then store the segmented text data and vectorized data in the database. When a user asks a question to the large model, the user's question will be matched with the vector of the segmented text data, and then the matched segmented text data will be combined with the user's question to form an enhanced context, and then the large model will generate answers to the question based on the enhanced context. Among them, RAG retrieval generates new question text by performing natural language processing on the user's question, which enables the large model to understand the user's question more accurately, thereby improving the quality of the output answer.

[0003] When using RAG for large-scale model-assisted data retrieval, the quality of segmented text data is crucial. Original text data, due to its length, requires segmentation, but segmentation can easily lead to cross-block information loss, which in turn reduces the quality of the segmented text data. Traditional segmentation methods, which overlap blocks during segmentation, can easily lead to information duplication and data redundancy, reducing the quality of the segmented text data. Summary of the Invention

[0004] In order to solve the above technical problems, the purpose of this application is to provide a RAG retrieval process optimization method and system based on a fine-tuned embedding model. The technical solutions adopted are as follows:

[0005] In a first aspect, an embodiment of the present application provides a method for optimizing a RAG retrieval process based on a fine-tuned embedding model, the method comprising the following steps:

[0006] Obtaining each long text document, using a topic model to obtain the probability of various topics in each natural language paragraph in each long text document, and combining the total number of words in each natural language paragraph with the probability to determine the topic strength of the various topics in each natural language paragraph;

[0007] Based on the topic strength of each topic in all natural language paragraphs in each long text document, a topic strength sequence of each topic in each long text document is determined, and based on the average distribution of all elements in the topic strength sequence, the distribution strength of each topic in each long text document is determined; based on the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic and the discrete degree of all elements in the topic strength sequence, the distribution concentration of each topic in each long text document is determined, and combined with the distribution strength, the topic expression distribution degree of each topic in each long text document is determined;

[0008] Arrange the topic expression distribution degrees of all topics in each long text document in descending order, and then use the topics corresponding to the first preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopt a sliding window algorithm based on the topic strength sequence of each text segmentation topic to obtain all window sequences of each text segmentation topic, and determine the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half and the second half of the elements in each window sequence, so as to obtain short texts under all text segmentation topics;

[0009] Based on the total number of natural language paragraphs in each short text and the total number of common natural language paragraphs between each short text and the other short texts, the topic concentration of each short text is determined to obtain the text topic concentration of various short texts, and the RAG retrieval process is optimized in combination with the large language model.

[0010] Preferably, the method for determining the topic strength sequence of each topic in each long text document is:

[0011] Among the multiple topics of all natural language paragraphs in each long text document, the topic strengths of all natural language paragraphs under the same topic are combined into a topic strength sequence of the topic.

[0012] Preferably, the distribution intensity of each topic in each long text document is the average of all topic intensities of each topic in each long text document.

[0013] Preferably, the distribution concentration of each topic in each long text document is expressed as: Where Md i,j Represents the distribution concentration of the jth topic in the long text document i; σ1 i,j Represents the degree of dispersion of all elements in the topic strength sequence of the jth topic in the long text document i; σ2 i,j It represents the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic intensity sequence of the j-th topic in the long text document i; α represents a constant preset to be greater than 0.

[0014] Preferably, the topic expression distribution degree of each topic in each long text document is the product of the distribution intensity and the distribution concentration of each topic in each long text document.

[0015] Preferably, the topic mutation degree of each window sequence of each text segmentation topic is the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence of each text segmentation topic.

[0016] Preferably, obtaining short texts under all types of text segmentation topics includes:

[0017] Arrange the topic mutation degrees of all window sequences of each text segmentation topic in descending order, and then use the window sequences corresponding to the first preset number of topic mutation degrees as the sequences to be analyzed. Set a dividing point between the last element value in the first half of the elements and the first element value in the second half of the elements in each sequence to be analyzed. Utilize all the dividing points of each text segmentation topic, and in the long text document, group the natural language paragraphs between two adjacent dividing points into short texts under each text segmentation topic.

[0018] Preferably, determining the topic concentration of each short text to obtain the text topic concentration of various short texts includes:

[0019] The topic concentration β of the m-th short text m The expression is: Where Ph m,k represents the total number of identical natural language paragraphs between the mth short text and the kth short text; Ph m The total number of natural language paragraphs in the mth short text; Ph k represents the total number of natural language paragraphs of the k-th short text; K represents the number of short texts under all text segmentation topics;

[0020] Among the short texts under all kinds of text segmentation topics, the short texts containing the same natural language paragraphs are regarded as the same kind of short texts, and the cumulative sum of the topic concentrations of all short texts contained in various short texts is regarded as the text topic concentration of various short texts.

[0021] Preferably, the optimization of the RAG search process includes:

[0022] Arrange the text topic concentrations of all short texts in descending order and select the short text corresponding to the first preset text topic concentration as the final segmented text;

[0023] Obtain domain data and use the large language model and domain data as input to the adapter fine-tuning method, outputting a fine-tuned model;

[0024] Obtain user questions, and use the user questions and the final segmented texts of all long text documents as input to the summary extraction algorithm, and output the user question summary and the summary of each final segmented text;

[0025] Calculate the similarity between the user question summary and the summaries of all final segmented texts, sort the similarities in descending order, and then preset the final segmented text corresponding to the second numerical similarity as the input of the fine-tuning model in the RAG retrieval process, and output the question and answer results of the fine-tuning model.

[0026] In the second aspect, an embodiment of the present application also provides a RAG retrieval process optimization system based on a fine-tuning embedded model, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned RAG retrieval process optimization methods based on a fine-tuning embedded model.

[0027] This application has at least the following beneficial effects:

[0028] This application determines the topic strength sequence of each topic in each long text document based on the topic strength of multiple topics in all natural language paragraphs in each long text document, and determines the distribution strength of each topic in each long text document based on the average distribution of all elements in the topic strength sequence. The beneficial effect is that it can reflect the overall expression strength of the topic in the long text document, which helps to determine the dominant position of the topic in the entire document, thereby facilitating the optimization of subsequent text segmentation and RAG retrieval processes;

[0029] This application determines the distribution concentration of each topic in each long text document by analyzing the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic, as well as the discrete degree of all elements in the topic strength sequence. The beneficial effect is that the distribution concentration can reflect the distribution pattern of the topic in the long text document, which helps to ensure that the segmented short texts are coherent in terms of the topic, thereby improving the quality of text processing, as well as the accuracy and efficiency of retrieval;

[0030] This application determines the distribution of topic expression for each topic in each long text document based on the distribution concentration of each topic in each long text document, combined with the distribution strength. This has the beneficial effect of quantifying the relative importance of each topic in the long text document, helping to avoid segmenting important topics into different short texts, thereby improving the quality of text segmentation and enhancing retrieval accuracy and efficiency.

[0031] This application arranges the topic expression distribution degrees of all types of topics in each long text document in descending order, and then uses the topics corresponding to a preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopts a sliding window algorithm based on the topic strength sequence of each text segmentation topic, obtains all window sequences of each text segmentation topic, and determines the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half elements and the second half elements in each window sequence, so as to obtain short texts under all types of text segmentation topics. Its beneficial effect is that the topic mutation degree reflects the difference in average distribution between the front and back parts of the elements in the window sequence. By determining the topic mutation degree of each window sequence of each text segmentation topic, short texts under all types of text segmentation topics are obtained, which helps to ensure the coherence of topic expression in short texts;

[0032] This application determines the text topic concentration of each short text based on the total number of natural language paragraphs in each short text and the total number of identical natural language paragraphs between each short text and the other short texts, and optimizes the RAG retrieval process in combination with a large language model. The beneficial effect is that the short text with the more concentrated text topic is selected as the final segmentation result, which can improve the optimization effect of the RAG retrieval process.

[0033] This application analyzes the distribution of topic expressions of different topics in long text documents to obtain short text data with more concentrated and coherent meaning, thereby improving the quality of text segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions and advantages of the embodiments of the present application or the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 A flowchart of a method for optimizing a RAG retrieval process based on a fine-tuned embedding model according to an embodiment of the present application;

[0036] Figure 2 A schematic diagram of the topic expression distribution extraction process provided in one embodiment of the present application;

[0037] Figure 3 A schematic diagram of the subject mutation degree extraction process provided in one embodiment of the present application;

[0038] Figure 4 A schematic diagram of the RAG search process provided for one embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to further illustrate the technical means and effects adopted by this application to achieve the predetermined invention purpose, the following, in conjunction with the accompanying drawings and preferred embodiments, describes in detail the specific implementation method, structure, features and effects of a RAG retrieval process optimization method and system based on a fine-tuned embedded model proposed in this application. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0040] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0041] The following describes in detail a specific solution of a RAG retrieval process optimization method and system based on a fine-tuned embedding model provided by the present application with reference to the accompanying drawings.

[0042] See also Figure 1 , which shows a flowchart of a RAG retrieval process optimization method based on a fine-tuning embedding model provided by an embodiment of the present application, the method comprising the following steps:

[0043] Step S1: Obtain each long text document, use a topic model to obtain the probability of various topics in each natural language paragraph in each long text document, and combine the total number of words in each natural language paragraph with the said probability to determine the topic strength of various topics in each natural language paragraph.

[0044] For RAG search, the first step is to build a corresponding text database, which is obtained by performing natural language processing on a large amount of long text. RAG search usually uses text segmentation to complete natural language processing on long text. In this embodiment, each long text document is obtained from the long text database.

[0045] When constructing a text database for RAG retrieval, long texts need to be segmented into multiple short texts. During text segmentation, it is necessary to ensure that each short text can coherently express a theme. That is, each short text must contain the beginning and end of a text theme to ensure smooth expression. Therefore, natural language paragraphs are used as segmentation markers. That is, any natural language paragraph in the long text corresponds to a complete natural language paragraph in the segmented text. This can reduce the incoherence of text expression caused by segmenting text from the middle of a natural language paragraph.

[0046] Furthermore, the expression of a text theme is not isolated from other text themes. For example, a long text has five paragraphs, with the first three paragraphs expressing Theme One and the last three paragraphs expressing Theme Two. In this case, the third paragraph expresses both Theme One and Theme Two. Therefore, a single natural language paragraph can express multiple themes, and two different themes can overlap in certain natural language paragraphs. Traditional methods use overlapping blocks to segment text, i.e., the first three paragraphs form a short text, and the last three paragraphs form a short text. The overlap between the two short texts is called overlapping blocks.

[0047] Therefore, when a natural language paragraph expresses multiple topics, this embodiment uses the LDA Dirichlet allocation model to extract keywords from each long text document to extract the complex features of the natural language paragraph when expressing the topic, specifically:

[0048] Each natural language paragraph in each long text document is used as the input of the LDA Dirichlet allocation model, and the probabilities of various topics in each natural language paragraph are output.

[0049] The LDA Dirichlet model is a well-known technology in the field of natural language processing. The LDA model can estimate the probability that each paragraph in a document belongs to a different topic. The specific process of obtaining the probabilities of various topics in each natural language paragraph will not be described in detail.

[0050] Since the probabilities of various topics in each natural language paragraph output by the LDA Dirichlet model are normalized parameters, that is, the sum of the probabilities of all topics is 1, it represents the proportion of topic expressions in the natural language paragraph. However, the probability of a topic does not take into account the impact of the length of the natural language paragraph on topic expression. In other words, longer natural language paragraphs have more words and express more information, and are more important in reflecting the text topic information. Therefore, this embodiment multiplies the probabilities of various topics in each natural language paragraph by the total number of words in the natural language paragraph to record the topic strength of each topic in each natural language paragraph.

[0051] Step S2: Based on the topic strengths of multiple topics in all natural language paragraphs in each long text document, determine the topic strength sequence of each topic in each long text document, and based on the average distribution of all elements in the topic strength sequence, determine the distribution strength of each topic in each long text document; based on the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic, and the discrete degree of all elements in the topic strength sequence, determine the distribution concentration of each topic in each long text document, and in combination with the distribution strength, determine the topic expression distribution degree of each topic in each long text document.

[0052] For long texts, they have multiple topics, and the RAG retrieval method needs to segment the text according to the topic. This embodiment obtains the topic information in the long text by calculating the topic strength, but the text topics that need to be focused on and segmented in the long text need further screening. Therefore, this embodiment extracts the topic expression distribution of the long text document based on the topic distribution in the entire long text, and selects appropriate topics for text segmentation based on the overall topic distribution of the long text. Specifically:

[0053] S201: Determine a topic strength sequence for each topic in each long text document based on the topic strengths of multiple topics expressed in all natural language paragraphs in each long text document; determine a distribution strength for each topic in each long text document based on the average distribution of all elements in the topic strength sequence for each topic.

[0054] Each long text document contains multiple natural language paragraphs, each of which corresponds to one or more topics. It is uncertain whether any two natural language paragraphs have the same topic, and it is also uncertain how many topics are involved in each natural language paragraph.

[0055] Therefore, this embodiment determines the topic strength sequence of each topic in each long text document based on the topic strengths of multiple topics expressed in all natural language paragraphs in each long text document; and determines the distribution strength of each topic in each long text document based on the average distribution of all elements in the topic strength sequence of each topic, so as to quantify the topic distribution of each topic and determine the importance of each topic in text segmentation. Specifically,

[0056] Among the multiple topics of all natural language paragraphs in each long text document, the topic strengths of all natural language paragraphs under the same topic are combined into a topic strength sequence of the topic;

[0057] Furthermore, the mean of all elements in the topic strength sequence of each topic in each long text document is calculated and recorded as the distribution strength of each topic in each long text document;

[0058] According to the distribution strength of each topic in each long text document, it can be understood that the larger the mean value of all elements in the topic strength sequence, the greater the expression strength of the corresponding topic in the entire long text document, and the more likely it is that it is the main topic that the text wants to express. When performing text segmentation, this topic should be referred to. Conversely, the smaller the mean value of all elements in the topic strength sequence, the smaller the expression strength of the corresponding topic in the entire long text document, and this topic can be ignored when performing text segmentation.

[0059] S202: Determine the distribution concentration of each topic in each long text document based on the discreteness of all elements in the first-order difference sequence of the topic strength sequence of each topic and the discreteness of all elements in the topic strength sequence.

[0060] The expression of the same topic in different natural language paragraphs may vary significantly. This difference may manifest in two ways: first, the topic is significantly concentrated in a specific area of ​​a long text document, indicating that the topic is more densely expressed in this area; second, the expression intensity of the topic fluctuates greatly throughout the long text document, meaning that it appears with different frequencies and intensities in different parts of the text. Therefore, by analyzing the degree of dispersion of all elements in the first-order difference sequence of the topic intensity sequence for each topic, as well as the degree of dispersion of all elements in the topic intensity sequence, we determine the distribution concentration of each topic within each long text document, quantify the distribution of each topic in the long text, and ensure that each segmented short text is thematically coherent when performing text segmentation, thereby improving the quality of text processing and the accuracy and efficiency of retrieval. Specifically:

[0061] The distribution concentration Md of the jth topic in the long text document i i,j The expression is: Where σ1 i,j Represents the degree of dispersion of all elements in the topic strength sequence of the jth topic in the long text document i; σ2 i,j Represents the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence of the j-th topic in the long text document i; α represents a preset constant greater than 0, which is used to prevent the denominator from being 0, wherein the value of α is set manually. In this embodiment, the value of α is 0.01. Under the premise of ensuring that the denominator is not 0 and does not excessively affect the calculation results, the implementer can also set it by himself based on the specific situation. This embodiment does not impose any special restrictions.

[0062] It should be noted that there are many methods for measuring the degree of dispersion of a set of data. In this embodiment, the degree of dispersion of all elements in the topic strength sequence is measured by calculating the standard deviation of all elements in the topic strength sequence of each topic in each long text document. The degree of dispersion of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence of each topic in each long text document is measured by calculating the standard deviation of the absolute values. The implementer may also use methods such as variance or dispersion coefficient to measure the degree of dispersion of data. This embodiment does not impose any special restrictions on the selection of methods for measuring the degree of dispersion of a set of data.

[0063] The method for obtaining the first-order difference sequence is a well-known technology, and the specific acquisition process will not be described in detail.

[0064] Furthermore, according to the distribution concentration of each topic in each long text document, it can be understood that the greater the degree of discreteness of all elements in the topic strength sequence, the more uneven the distribution of the topic in the text, the more likely it is to be concentrated in certain paragraphs, or the fluctuation is large, and the smaller the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence, the less frequent the fluctuation of the topic strength sequence; therefore, combining the two degrees of discreteness, it can be seen that if the degree of discreteness of all elements in the topic strength sequence is greater and the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence is smaller, the distribution concentration is greater, indicating that the corresponding topic is mainly concentrated in a part of continuous paragraphs in the long text document; conversely, if the degree of discreteness of all elements in the topic strength sequence is smaller and the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic strength sequence is greater, the distribution concentration is smaller, indicating that the corresponding topic fluctuates greatly in the long text document and is more dispersed.

[0065] S203: Determine the topic expression distribution degree of each topic in each long text document based on the distribution concentration and distribution strength of each topic in each long text document.

[0066] Distribution strength and distribution concentration describe the characteristics of topics in a text from different perspectives. Distribution strength reflects the overall expression strength of a topic in a text, while distribution concentration reflects whether the distribution pattern of a topic in a text is concentrated or dispersed. Combining the distribution strength and distribution concentration, we obtain the topic expression distribution degree to quantify the relative importance of each topic in a long text document, avoid segmenting important topics into different short texts, improve the quality of text segmentation, and thus enhance retrieval accuracy and efficiency. Specifically:

[0067] The topic expression distribution of each topic in each long text document is the product of the distribution concentration and the distribution intensity of each topic.

[0068] Furthermore, according to the distribution degree of topic expression of each topic in each long text document, it can be understood that the greater the distribution intensity, the more widely and evenly the corresponding topic is distributed in the long text document, and the greater the topic expression distribution degree; and the greater the distribution concentration, the more concentrated the distribution of the corresponding topic in the text, and it may appear frequently in certain paragraphs, and the greater the topic expression distribution degree; conversely, the smaller the distribution intensity, the less significant the expression of the corresponding topic in the long text document, and the smaller the topic expression distribution degree; and the smaller the distribution concentration, the more dispersed the corresponding topic is in the text, and there is no obvious high-intensity area, and the topic expression distribution degree will decrease, resulting in the topic not being particularly emphasized during text segmentation.

[0069] Preferably, the schematic diagram of the topic expression distribution extraction process provided in this embodiment is as follows: Figure 2 shown.

[0070] Step S3: Arrange the topic expression distribution degrees of all topics in each long text document in descending order, and then use the topics corresponding to the first preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopt a sliding window algorithm based on the topic strength sequence of each text segmentation topic to obtain all window sequences of each text segmentation topic, and determine the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half and the second half of the elements in each window sequence, so as to obtain short texts under all text segmentation topics.

[0071] The long text documents are segmented based on the topic expression distribution in step S2, specifically: the topic expression distribution of all topics in each long text document is arranged in descending order and then a preset number of topics are selected as the text segmentation topics based on each long text document.

[0072] It should be noted that the value of the preset number is set manually. In this embodiment, the value of the preset number is set to 50. The implementer can also set it by himself according to the specific situation. This embodiment does not impose any special restrictions.

[0073] In particular, if the number of all topics is less than a preset number, all topics are regarded as text segmentation topics of the long text document.

[0074] A text segmentation topic may be expressed at different locations in a long text document. When performing text segmentation, the areas where the topic is concentrated should be divided into a short text. Therefore, in this embodiment, a sliding window algorithm is used to obtain all window sequences for each text segmentation topic. Based on the difference in the average distribution between the first half and the second half of the elements in each window sequence, the topic mutation degree of each window sequence for each text segmentation topic is determined to obtain short texts under all text segmentation topics. Specifically, the following is the result:

[0075] The sliding window algorithm is used to divide the topic intensity sequence of each text segmentation topic into multiple window sequences, and all window sequences of each text segmentation topic are obtained. The sliding window size is set to A, the sliding step is set to B, and the sliding window starts from the first element in the topic intensity sequence and slides to the last element. Each time it slides, all elements in the window form a window sequence.

[0076] It should be noted that the values ​​of the sliding window size A and the sliding step size B are both manually set. In this embodiment, the value of the sliding window size A is 10, and the value of the sliding step size B is 1. The implementer can also set them according to the specific situation. This embodiment does not impose any special restrictions.

[0077] Furthermore, the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence of each text segmentation topic is used as the topic mutation degree of each window sequence of each text segmentation topic;

[0078] It should be noted that there are many methods for measuring the difference between data. In this embodiment, the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence is measured by calculating the absolute value of the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence. The implementer may also use other methods for measuring the difference between data, such as ratio. This embodiment does not impose any special restrictions on the selection of the method for measuring the difference between data.

[0079] Preferably, the schematic diagram of the subject mutation degree extraction process provided in this embodiment is as follows Figure 3 shown.

[0080] Furthermore, the window sequences corresponding to the first preset number of topic mutation degrees in all window sequences of each text segmentation topic are arranged in descending order of the topic mutation degrees, and are used as the sequences to be analyzed. A dividing point is set between the last element value in the first half of the elements and the first element value in the second half of the elements in each sequence to be analyzed. Using all the dividing points of each text segmentation topic, the natural language paragraphs between two adjacent dividing points in the long text document are combined into short texts under each text segmentation topic.

[0081] It should be noted that the value of the preset number is set manually. The specific setting process of the preset number in this embodiment is as follows:

[0082] Wherein, F-1 represents the value of the preset number; P represents the total number of natural language paragraphs corresponding to all elements in the window sequence; Q represents a preset value. In this embodiment, the value of Q is 5, indicating that when segmenting the text, 5 paragraphs tend to constitute a short text. The implementer can also set it according to the specific situation. This embodiment does not impose any special restrictions. Represents the ceiling function.

[0083] It should be understood that the window sequence is obtained based on the segmented topic strength sequence. The topic strength sequence of each topic contains the topic strength of all natural language paragraphs expressing the corresponding topic. Therefore, each element in the window sequence is actually a topic strength, and each topic strength corresponds to a natural language paragraph. Therefore, the above P represents the total number of natural language paragraphs corresponding to all elements in the window sequence.

[0084] This embodiment selects some segmentation methods from these segmentation methods as the final paragraph segmentation strategy. Text segmentation is a common technology in the field of natural language processing, and its specific principles will not be described in detail.

[0085] Step S4: determining the text topic concentration of each short text based on the total number of natural language paragraphs in each short text and the total number of natural language paragraphs that are common between each short text and the other short texts.

[0086] The better the short texts are segmented from long texts, the more concentrated the text topics in the short texts are, and the topics expressed in the short texts are less likely to appear in adjacent paragraphs of the short texts. Therefore, to improve the quality of text segmentation, the text topic concentration of each short text is determined based on the total number of natural language paragraphs in each short text and the total number of natural language paragraphs that each short text shares with other short texts. Specifically,

[0087] The topic concentration β of the m-th short text m The expression is: Where Ph m,k represents the total number of identical natural language paragraphs between the mth short text and the kth short text; Ph m Indicates the total number of paragraphs in the mth short text; Ph k represents the total number of natural language paragraphs of the k-th short text; K represents the number of short texts under all types of text segmentation topics.

[0088] According to the topic mutation degree of each window sequence of each text segmentation topic, it can be understood that if the total number of identical natural paragraphs between two short texts is larger, the mean number of paragraphs of the two short texts is smaller, indicating that these short texts contain fewer paragraphs, the distribution is more concentrated, and are more likely to discuss a coherent topic; conversely, if the total number of identical natural paragraphs between two short texts is smaller, the mean number of paragraphs of the two short texts is larger, indicating that these short texts contain more paragraphs, the distribution is more dispersed, and are less likely to discuss a coherent topic.

[0089] Furthermore, since there may be repeated texts in the short texts under all text segmentation topics, among the short texts under all text segmentation topics, the short texts containing the same natural language paragraphs are regarded as the same type of short texts, and the cumulative sum of the topic concentrations of all short texts contained in various short texts is regarded as the text topic concentration of various short texts.

[0090] It should be noted that for repeated texts in short texts, although the corresponding short texts are the same short texts, they are obtained under different topic divisions. Therefore, the topic concentration of the repeated short texts is represented by the sum of the topic concentrations rather than the mean, so as to avoid ignoring the topic concentration information from different topics.

[0091] Step S5: Based on the text topic concentration of various short texts and combined with the large language model, the RAG retrieval process is optimized.

[0092] (1) Based on the text topic concentration obtained in step S4, the long text document is divided. Specifically, the short text corresponding to the first preset text topic concentration among the short texts of each long text document is arranged in descending order as the final segmented text;

[0093] It should be noted that the value of the preset first value is artificially set. In this embodiment, the value of the preset first value is G. Based on the value obtained in step S4, The implementer can also set it according to the specific situation, and this embodiment does not impose any special restrictions.

[0094] (2) Further, domain data is obtained, and the large language model and domain data are used as inputs of the adapter fine-tuning method, and the fine-tuned model is output;

[0095] It should be noted that there are many commonly used large language models. In this embodiment, the Bidirectional Encoder Representations from Transformers (BERT) model is used as the input of the adapter fine-tuning method. Implementers can also use other large language models, and this embodiment does not impose any special restrictions.

[0096] Among them, the BERT model and adapter fine-tuning method are both well-known technologies, and their specific principles are not repeated here.

[0097] Domain data generally refers to a dataset related to a specific task or field. These datasets may include industry-specific text data, such as medical literature, legal documents, etc.; or task-specific data, such as product reviews, news reports, etc. The domain data in this embodiment mainly refers to data in the field of medical question and answer.

[0098] (3) Further, the user question is obtained, and the user question and the final segmented texts of all long text documents are used as inputs of the summary extraction algorithm, and the user question summary and the summary of each final segmented text are output;

[0099] It should be noted that there are many commonly used summary extraction algorithms. This embodiment adopts the TD-IDF term frequency-inverse document frequency algorithm. The implementer can also use the TextRank text ranking algorithm to obtain the text summary. This embodiment does not impose any special restrictions on the selection of the summary extraction algorithm.

[0100] Among them, the TD-IDF term frequency-inverse document frequency algorithm is a well-known technology in the field of natural language processing, and the specific process of obtaining the text summary is not repeated here.

[0101] (4) Calculate the similarity between the user question summary and the summaries of all final segmented texts, sort the similarities in descending order, and use the final segmented texts corresponding to the second preset similarity as the input of the fine-tuning model in the RAG retrieval process, and output the question-answering results of the fine-tuning model. It should be noted that the summary is in the form of a vector.

[0102] It should be understood that there are many methods for calculating the similarity between vectors. In this embodiment, the cosine similarity between the user question summary and the summaries of all the final segmented texts is calculated to measure the similarity between the user question summary and the summaries of all the final segmented texts. The implementer may also use other methods for measuring the similarity between vectors, such as the inverse of the Euclidean distance. This embodiment does not impose any special restrictions on the selection of methods for measuring the similarity between vectors.

[0103] The calculation process of cosine similarity is a well-known technology, and its specific calculation steps are not repeated here.

[0104] In addition, it should be understood that the value of the preset second numerical value is manually set. In this embodiment, the value of the preset second numerical value is 3. The implementer can also set it by himself based on the specific situation. This embodiment does not impose any special restrictions.

[0105] So far, this embodiment calculates the distribution intensity according to the intensity of the occurrence of the topic in the long text, calculates the distribution concentration according to the distribution concentration law of the intensity of different topics in the long text, and obtains the topic expression distribution of the topic in the long text document based on the distribution intensity and distribution concentration. Topics with large distribution intensity and concentrated distribution are regarded as topics with good segmentation effect, and these topics are selected to participate in text segmentation; further, the long text document is segmented based on the dividing point to obtain multiple short texts; and based on the text topic concentration, the short text with more concentrated text topics is selected as the final text segmentation result, and the text segmentation result is added to the RAG retrieval process, which improves the quality of text segmentation, optimizes the RAG retrieval process, and improves the accuracy and efficiency of RAG retrieval.

[0106] Preferably, the RAG search process diagram provided in this embodiment is as follows Figure 4 shown.

[0107] Based on the same inventive concept as the above method, an embodiment of the present application also provides a RAG retrieval process optimization system based on a fine-tuning embedded model, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of any one of the above-mentioned RAG retrieval process optimization methods based on a fine-tuning embedded model.

[0108] It should be noted that the order in which the embodiments of the present application are presented is for illustrative purposes only and does not necessarily represent the superiority or inferiority of the embodiments. Furthermore, the foregoing descriptions of specific embodiments of this specification are provided. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order or sequential sequence shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0109] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments.

[0110] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A RAG retrieval process optimization method based on fine-tuning the embedding model, characterized in that: The method comprises the following steps: Obtaining each long text document, using a topic model to obtain the probability of various topics in each natural language paragraph in each long text document, and combining the total number of words in each natural language paragraph with the probability to determine the topic strength of the various topics in each natural language paragraph; Based on the topic strength of each topic in all natural language paragraphs in each long text document, a topic strength sequence of each topic in each long text document is determined, and based on the average distribution of all elements in the topic strength sequence, the distribution strength of each topic in each long text document is determined; based on the discrete degree of differences between all adjacent elements in the topic strength sequence of each topic and the discrete degree of all elements in the topic strength sequence, the distribution concentration of each topic in each long text document is determined, and combined with the distribution strength, the topic expression distribution degree of each topic in each long text document is determined; Arrange the topic expression distribution degrees of all topics in each long text document in descending order, and then use the topics corresponding to the first preset number of topic expression distribution degrees as the text segmentation topics based on each long text document; adopt a sliding window algorithm based on the topic strength sequence of each text segmentation topic to obtain all window sequences of each text segmentation topic, and determine the topic mutation degree of each window sequence of each text segmentation topic based on the difference in average distribution between the first half and the second half of the elements in each window sequence, so as to obtain short texts under all text segmentation topics; Based on the total number of natural language paragraphs in each short text and the total number of common natural language paragraphs between each short text and the other short texts, the topic concentration of each short text is determined to obtain the text topic concentration of various short texts, and the RAG retrieval process is optimized in combination with the large language model.

2. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The method for determining the topic strength sequence of each topic in each long text document is as follows: Among the multiple topics of all natural language paragraphs in each long text document, the topic strengths of all natural language paragraphs under the same topic are combined into a topic strength sequence of the topic.

3. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The distribution intensity of each topic in each long text document is the average of all topic intensities of each topic in each long text document.

4. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The distribution concentration of each topic in each long text document is expressed as follows: Where Md i,j Represents the distribution concentration of the jth topic in the long text document i; σ1 i,j Represents the degree of dispersion of all elements in the topic strength sequence of the jth topic in the long text document i; σ2 i,j It represents the degree of discreteness of the absolute values ​​of all elements in the first-order difference sequence of the topic intensity sequence of the j-th topic in the long text document i; α represents a constant preset to be greater than 0.

5. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The topic expression distribution degree of each topic in each long text document is the product of the distribution intensity and the distribution concentration of each topic in each long text document.

6. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The topic mutation degree of each window sequence of each text segmentation topic is the difference between the mean of the first half of the elements and the mean of the second half of the elements in each window sequence of each text segmentation topic.

7. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The short texts under all kinds of text segmentation topics are obtained, including: Arrange the topic mutation degrees of all window sequences of each text segmentation topic in descending order, and then use the window sequences corresponding to the first preset number of topic mutation degrees as the sequences to be analyzed. Set a dividing point between the last element value in the first half of the elements and the first element value in the second half of the elements in each sequence to be analyzed. Utilize all the dividing points of each text segmentation topic, and in the long text document, group the natural language paragraphs between two adjacent dividing points into short texts under each text segmentation topic.

8. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: Determining the topic concentration of each short text to obtain the text topic concentration of various short texts includes: The topic concentration β of the m-th short text m The expression is: Where Ph m,k represents the total number of identical natural language paragraphs between the mth short text and the kth short text; Ph m The total number of natural language paragraphs in the mth short text; Ph k represents the total number of natural language paragraphs of the k-th short text; K represents the number of short texts under all text segmentation topics; Among the short texts under all kinds of text segmentation topics, the short texts containing the same natural language paragraphs are regarded as the same kind of short texts, and the cumulative sum of the topic concentrations of all short texts contained in various short texts is regarded as the text topic concentration of various short texts.

9. The RAG retrieval process optimization method based on fine-tuning the embedding model according to claim 1, characterized in that: The optimization of the RAG search process includes: Arrange the text topic concentrations of all short texts in descending order and select the short text corresponding to the first preset text topic concentration as the final segmented text; Obtain domain data and use the large language model and domain data as input to the adapter fine-tuning method, outputting a fine-tuned model; Obtain user questions, and use the user questions and the final segmented texts of all long text documents as input to the summary extraction algorithm, and output the user question summary and the summary of each final segmented text; Calculate the similarity between the user question summary and the summaries of all final segmented texts, sort the similarities in descending order, and then preset the final segmented text corresponding to the second numerical similarity as the input of the fine-tuning model in the RAG retrieval process, and output the question and answer results of the fine-tuning model.

10. A RAG retrieval process optimization system based on a fine-tuned embedding model, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the RAG retrieval process optimization method based on fine-tuning the embedding model as described in any one of claims 1 to 9 are implemented.

Images