Multi-level retrieval enhancement method for large language model

Abstract

The invention discloses a multi-level retrieval enhancement method for a large language model, particularly relates to the technical field of multi-level retrieval, and aims to more accurately understand a query intention of a user by performing initial analysis on a query input by the user and generating a core query statement and a primary retrieval document set. Performing clustering analysis on the primary retrieval document set to identify a redundant risk document set, a reserved document set and an information loss risk document set, obtaining redundant detection information and credibility evaluation information of the redundant risk document set, performing primary screening on the redundant risk document set, effectively filtering redundant retrieval documents with relatively low credibility, and improving the reliability of the retrieval documents. A supplementary query statement is generated according to an information loss risk document set, secondary retrieval is performed in a knowledge base, multi-angle requirements and potential fuzzy intentions of a user are more comprehensively covered, a document priority recommendation index is generated through comprehensive calculation of a document quality coefficient and a document freshness coefficient, and the user experience is improved. And the documents with high quality and high timeliness are preferentially displayed to the user, so that the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of multi-level retrieval technology, and more specifically, to a large language model multi-level retrieval enhancement method. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models have made significant progress in the field of natural language processing and are widely used in text generation, question-answering systems, intelligent search engines and other scenarios. However, existing retrieval systems based on large language models still face many challenges when processing complex queries. Traditional single-query retrieval methods usually rely on a single query entered by the user to retrieve relevant documents. This method is prone to insufficient information coverage and is difficult to fully capture the user's query intent, especially when faced with multi-angle or ambiguous requirements. In addition, since the results of information retrieval directly depend on the quality of the initial query, if the initial query fails to accurately reflect the user's intent, the retrieval results will be incomplete or inaccurate. Summary of the invention

[0003] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a large language model multi-level retrieval enhancement method to solve the problems raised in the above-mentioned background technology.

[0004] To achieve the above object, the present invention provides the following technical solutions: A large language model multi-level retrieval enhancement method, characterized by comprising the following steps: Step S1, performing an initial analysis on the query input by the user, generating a core query statement based on the initial analysis result, and performing a primary search in the knowledge base based on the core query statement to obtain a primary search document set; Step S2, performing cluster analysis on the primary search document set to identify redundant risk document sets, retained document sets, and information missing risk document sets; Step S3, obtaining redundant detection information and credibility assessment information of the redundant risk document set, performing a primary screening on the redundant risk document set, and obtaining a redundant document screening set; Step S4, generating a supplementary query statement based on the information missing risk document set, and performing a secondary search in the knowledge base based on the supplementary query statement to obtain an information supplementary document set; Step S5, merge the retained document set, the redundant document screening set, and the information supplement document set to obtain a final document set, obtain the document retrieval quality information and document freshness information of the final document set, generate a document priority recommendation index, sort the documents in the final document set from large to small according to the document priority recommendation index, obtain a document sorting table, and display the documents to the user in sequence according to the document sorting order.

[0005] In a preferred embodiment, in step S2, cluster analysis is performed on the primary search document set to identify redundant risk document sets, retained document sets, and information missing risk document sets, as follows: Step S21, preprocessing the primary search document set, including removing noise, standardizing the text format, and converting each preprocessed primary search document into a vector representation; Step S22, using the elbow method to determine the number K of initial cluster centroids; Step S23, according to the determined number K of initial clustering centroids, randomly selecting K primary retrieval documents from the primary retrieval document set as initial clustering centroids; Step S24, calculating the Euclidean distance between each primary search document in the primary search document set and the centroid of the initial cluster, and assigning each primary search document to the cluster closest to it; Step S25, calculating the average vector of each cluster and taking it as the new cluster centroid; Step S26, repeating steps S24 and S25 to iteratively update the clusters until the centroid of the clusters no longer changes, thereby obtaining a final cluster set; Step S27: preprocess the core query statement, convert the preprocessed core query statement into a vector representation, and use the cosine similarity method to calculate the first cosine similarity between each primary search document in the cluster and the core query statement. , the expression is as follows , where represents the vector representation of the primary retrieved documents in the cluster, Represents the vector representation of the core query statement, express and The dot product operation, represents the Euclidean norm of the primary retrieval document vector, Represents the Euclidean norm of the core query statement vector; In the cluster, count the number of occurrences C1 where the first cosine similarity between the primary search document and the core query statement is greater than 0 and less than or equal to 1; count the number of occurrences C2 where the first cosine similarity between the primary search document and the core query statement is less than 0 and greater than or equal to -1, and calculate the similarity ratio , the expression is as follows ; Step S28, comparing the similarity ratio with a preset similarity ratio stage threshold, marking the clusters, identifying the redundant risk document set, the retained document set, and the information missing risk document set, the similarity ratio stage threshold includes a similarity ratio first threshold and a similarity ratio second threshold, and the similarity ratio first threshold is less than the similarity ratio second threshold; If the similarity ratio is greater than the second similarity ratio threshold, the cluster is marked as a redundant risk cluster, and the primary search document in the redundant risk cluster is added to the redundant risk document set; If the similarity ratio is less than or equal to the second similarity ratio threshold and greater than or equal to the first similarity ratio threshold, the cluster is marked as a normal cluster, and the primary search documents in the normal cluster are added to the reserved document set; If the similarity ratio is less than the first similarity ratio threshold, the cluster is marked as an information missing risk cluster, and the primary search documents in the information missing risk cluster are added to the information missing risk document set.

[0006] In a preferred embodiment, in step S3, redundant detection information and credibility assessment information of the redundant risk document set are obtained, and the redundant risk document set is initially screened to obtain a redundant document screening set, as follows: The redundant detection information includes a second cosine similarity, and the credibility evaluation information includes a credibility evaluation coefficient; The cosine similarity method is used to calculate the second cosine similarity between two primary retrieval documents in the redundant risk document set, and the second cosine similarity is compared with a preset redundancy threshold. If the second cosine similarity is greater than the redundancy threshold, the credibility evaluation coefficients of the two primary retrieval documents are respectively obtained, and the credibility evaluation coefficients of the two primary retrieval documents are compared. The primary retrieval document with the smallest credibility evaluation coefficient is marked as a document to be deleted. If the credibility evaluation coefficients of the two primary retrieval documents are equal, one of the primary retrieval documents is randomly marked as a document to be deleted; The number of times each primary retrieval document in the redundant risk document set is marked as a document to be deleted is counted, the number of times the document is marked as a document to be deleted is compared with a preset marking threshold, and the primary retrieval documents whose number is less than the marking threshold are added to the redundant document screening set.

[0007] In a preferred embodiment, the logic for obtaining the credibility evaluation coefficient is as follows: Get the percentage of self-citations in primary search documents , the expression is as follows , where Indicates the number of times the primary search document cites the author's previously published documents. Indicates the number of all documents cited in the primary search document; Get the number of citations of primary search documents , Number of review rounds , Number of rejections ; Normalize the percentage of self-citations, the number of citations, the number of review rounds, and the number of rejections to calculate the credibility assessment coefficient , the expression is as follows .

[0008] In a preferred embodiment, the document retrieval quality information includes a document retrieval quality coefficient, and the document freshness information includes a document freshness coefficient.

[0009] In a preferred embodiment, the logic for obtaining the document retrieval quality coefficient is as follows: Preprocess the retrieved documents and user input queries in the final document set, including word segmentation, stop word removal, and stemming; Calculating document retrieval quality coefficient , the expression is as follows , where Represents the total number of retrieved documents in the final document set, Represents the i-th word in the user input query, i={1,2,...,n}, n is a positive integer, express The frequency of the word in the retrieved document D, Indicates the document length of the retrieved document D, represents the average length of the retrieved documents in the final document set, and is a constant factor.

[0010] In a preferred embodiment, the logic for obtaining the document freshness coefficient is as follows: Get the release time and user query time of the retrieved document, and calculate the time difference between the release time and the user query time of the retrieved document ; Get the content update time of each retrieved document and calculate the time difference between adjacent content updates , j represents the sequence number of the content update event trigger; Calculate the average time difference between adjacent content updates , the expression is as follows Where j = {1, 2, ..., J}; Calculate the standard deviation of the time difference between adjacent content updates , the expression is as follows ; Calculate document freshness coefficient , the expression is as follows .

[0011] In a preferred embodiment, the obtained document retrieval quality coefficient and document freshness coefficient are normalized, and a retrieval document screening model is constructed based on the normalized document retrieval quality coefficient and document freshness coefficient to generate a document priority recommendation index. The model is based on the following formula , where Respectively represent the preset proportional coefficients of the document retrieval quality coefficient and the document freshness coefficient, and All are greater than 0; According to the document priority recommendation index, the documents in the final document set are sorted from large to small to obtain a document sorting table, and the documents are displayed to the user in sequence according to the document sorting order.

[0012] Technical effects and advantages of the present invention: 1. The present invention more accurately understands the user's query intention by initially parsing the query input by the user and generating core query statements and primary retrieval document sets, performs cluster analysis on the primary retrieval document set to identify redundant risk document sets, retained document sets and information missing risk document sets, obtains redundant detection information and credibility assessment information of the redundant risk document sets, performs initial screening on the redundant risk document sets to obtain a redundant document screening set, effectively filters out redundant and low-credibility retrieval documents, improves the quality and relevance of the final document set, generates supplementary query statements based on the information missing risk document set, performs secondary retrieval in the knowledge base based on the supplementary query statements, obtains an information supplementary document set, and more comprehensively covers the user's multi-angle needs and potential ambiguous intentions.

[0013] 2. The present invention generates a document priority recommendation index through comprehensive calculation of the document quality coefficient and the document freshness coefficient, ensuring that in the final document set, high-quality and timely documents are displayed to users first, effectively improving the user experience, allowing users to obtain high-quality information that meets their needs faster and more accurately when searching, avoiding the problem of incomplete results caused by insufficient information coverage in traditional single query retrieval methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings; Figure 1 Schematic diagram of the structure of the method of the embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0016] Example: Figure 1 The present invention provides a large language model multi-level retrieval enhancement method, comprising the following steps: Step S1, performing an initial analysis on the query input by the user, generating a core query statement based on the initial analysis result, and performing a primary search in the knowledge base based on the core query statement to obtain a primary search document set; Step S2, performing cluster analysis on the primary search document set to identify redundant risk document sets, retained document sets, and information missing risk document sets; Step S3, obtaining redundant detection information and credibility assessment information of the redundant risk document set, performing a primary screening on the redundant risk document set, and obtaining a redundant document screening set; Step S4, generating a supplementary query statement based on the information missing risk document set, and performing a secondary search in the knowledge base based on the supplementary query statement to obtain an information supplementary document set; Step S5, merging the retained document set, the redundant document screening set, and the information supplement document set to obtain a final document set, obtaining document retrieval quality information and document freshness information of the final document set, generating a document priority recommendation index, sorting the documents in the final document set from large to small according to the document priority recommendation index, obtaining a document sorting table, and displaying the documents to the user in order according to the document sorting order; In step S1, the query input by the user is initially parsed, and a core query statement is generated according to the initial parsing result. A primary search is performed in the knowledge base according to the core query statement to obtain a primary search document set, which is as follows: Perform word segmentation on the query input by the user to extract important keywords and phrases. Common word segmentation tools include SpaCy, NLTK, etc.

[0017] Perform part-of-speech tagging on the words after segmentation to identify words with different parts of speech, such as nouns, verbs, and adjectives, which helps identify the key components of the input query; Input the annotated words into the topic model to extract the query topics in the input query. The query topics include query topic words and query topic phrases. Common topic models include Latent Dirichlet Allocation, LDA, etc. The extracted query subject words and query subject phrases are used as core query statements; It should be noted that the extracted query subject words and query subject phrases may include multiple groups, so the core query statement includes one or more core query statements; Use the generated core query statement to perform a primary search in the knowledge base to obtain the primary search document set related to the core query statement. You can choose different retrieval models according to the actual application scenario, such as Boolean retrieval, vector space model, BM25, or deep learning-based retrieval models (such as BERT-based retriever). In step S2, cluster analysis is performed on the primary search document set to identify redundant risk document sets, retained document sets, and information missing risk document sets, as follows: Step S21, preprocessing the primary search document set, including removing noise, standardizing the text format, and converting each preprocessed primary search document into a vector representation; It should be noted that the primary search document can be converted into a vector representation using methods such as TF-IDF, word embedding (such as Word2Vec, BERT), etc. to generate the vector representation of the primary search document. Each dimension in the vector can be a corresponding word in the primary search document. Step S22, using the elbow method to determine the number K of initial cluster centroids; It should be noted that the elbow method is used to determine the initial number of cluster centroids K, starting from K=1, and gradually increasing the K value. For each K value, the K-means clustering algorithm is executed, and the SSE value after each clustering is recorded. The SSE value refers to the sum of the squares of the distances from each primary search document to its cluster centroid. A curve graph is drawn with the K value as the horizontal axis and the SSE value as the vertical axis. In the curve graph, the position where the SSE value decreases significantly slower as the K value increases is found. This point usually presents an "elbow" shape. The slope of the curve at different K values ​​is calculated, and the slope of the curve at different K values ​​is compared with the curve elbow slope threshold range. The K value corresponding to the curve slope that first appears within the curve elbow slope threshold range is taken as the final initial number of cluster centroids K; Step S23, according to the determined number K of initial clustering centroids, randomly selecting K primary retrieval documents from the primary retrieval document set as initial clustering centroids; Step S24, calculating the Euclidean distance between each primary search document in the primary search document set and the centroid of the initial cluster, and assigning each primary search document to the cluster closest to it; Step S25, calculating the average vector of each cluster and taking it as the new cluster centroid; Step S26, repeating steps S24 and S25 to iteratively update the clusters until the centroid of the clusters no longer changes, thereby obtaining a final cluster set; The final clustering cluster set includes multiple clustering clusters; Step S27: preprocess the core query statement, convert the preprocessed core query statement into a vector representation, and use the cosine similarity method to calculate the first cosine similarity between each primary search document in the cluster and the core query statement. , the expression is as follows , where represents the vector representation of the primary retrieved documents in the cluster, Represents the vector representation of the core query statement, express and The dot product operation, represents the Euclidean norm of the primary retrieval document vector, Represents the Euclidean norm of the core query statement vector; The first cosine similarity value obtained using the cosine similarity method ranges from -1 to 1, and its specific meanings are as follows: 1: Indicates that the directions of the two vectors are exactly the same, that is, the angle between them is 0 degrees, which means that the two vectors are very similar; 0: indicates that the two vectors are perpendicular to each other, that is, the angle between them is 90 degrees, which means that the two vectors have no similarity; -1: indicates that the directions of the two vectors are completely opposite, that is, the angle between them is 180 degrees, which means that the two vectors are completely dissimilar; It should be noted that the preprocessing operation for the core query statement is consistent with the preprocessing operation for the primary search document set; In the cluster, count the number of occurrences C1 where the first cosine similarity between the primary search document and the core query statement is greater than 0 and less than or equal to 1; count the number of occurrences C2 where the first cosine similarity between the primary search document and the core query statement is less than 0 and greater than or equal to -1, and calculate the similarity ratio , the expression is as follows ; The similarity ratio reflects the overall similarity between all primary retrieval documents in the cluster and the core query statement. When the similarity ratio is extremely high, it indicates that there are a large number of primary retrieval documents in the cluster that are similar to the core query statement. In this case, there may be a high degree of content duplication between documents within the cluster, which may generate redundant information. When the similarity ratio is extremely low, it indicates that there are a small number of primary retrieval documents in the cluster that are similar to the core query statement. This means that the primary retrieval documents in the cluster may not fully meet the information needs reflected by the core query statement, and there is a risk of missing information. Step S28, comparing the similarity ratio with a preset similarity ratio stage threshold, marking the clusters, identifying the redundant risk document set, the retained document set, and the information missing risk document set, the similarity ratio stage threshold includes a similarity ratio first threshold and a similarity ratio second threshold, and the similarity ratio first threshold is less than the similarity ratio second threshold; If the similarity ratio is greater than the second similarity ratio threshold, the cluster is marked as a redundant risk cluster, and the primary search document in the redundant risk cluster is added to the redundant risk document set; If the similarity ratio is less than or equal to the second similarity ratio threshold and greater than or equal to the first similarity ratio threshold, the cluster is marked as a normal cluster, and the primary search documents in the normal cluster are added to the reserved document set; If the similarity ratio is less than the first similarity ratio threshold, the cluster is marked as an information missing risk cluster, and the primary search documents in the information missing risk cluster are added to the information missing risk document set; In step S3, redundant detection information and credibility assessment information of the redundant risk document set are obtained, and the redundant risk document set is initially screened to obtain a redundant document screening set, as follows: The redundant detection information includes a second cosine similarity, and the credibility evaluation information includes a credibility evaluation coefficient; The cosine similarity method is used to calculate the second cosine similarity between two primary retrieval documents in the redundant risk document set, and the second cosine similarity is compared with a preset redundancy threshold. If the second cosine similarity is greater than the redundancy threshold, the credibility evaluation coefficients of the two primary retrieval documents are respectively obtained, and the credibility evaluation coefficients of the two primary retrieval documents are compared. The primary retrieval document with the smallest credibility evaluation coefficient is marked as a document to be deleted. If the credibility evaluation coefficients of the two primary retrieval documents are equal, one of the primary retrieval documents is randomly marked as a document to be deleted; Count the number of times each primary search document in the redundant risk document set is marked as a document to be deleted, compare the number of times marked as a document to be deleted with a preset marking threshold, and add the primary search documents whose number is less than the marking threshold to the redundant document screening set; It should be noted that the calculation method of the second cosine similarity between two primary retrieval documents in the redundant risk document set using the cosine similarity method is consistent with the calculation method of the first cosine similarity, wherein the two primary retrieval documents are any two primary retrieval documents in the redundant risk document set; Traversing the primary search documents in the redundant risk document set, and marking the primary search documents in the redundant risk document set using the above method. Since it is necessary to traverse all the primary search documents in the redundant risk document set, the number of times that the same primary search document is marked as a document to be deleted includes zero to multiple times; The credibility assessment coefficient is used to measure the information reliability and authority of the primary search documents in the redundant risk document set, which helps to identify which primary search documents have higher credibility, thereby providing an objective reference basis in the screening process and determining the primary search documents that should be retained or deleted; The logic for obtaining the credibility evaluation coefficient is as follows: Get the percentage of self-citations in primary search documents , the expression is as follows , where Indicates the number of times the primary search document cites the author's previously published documents. Indicates the number of all documents cited in the primary search document; Get the number of citations of primary search documents , Number of review rounds , Number of rejections ; Normalize the percentage of self-citations, the number of citations, the number of review rounds, and the number of rejections to calculate the credibility assessment coefficient , the expression is as follows ; After the author submits the document to a journal or conference, the editor will send it to peer reviewers for review. The reviewers will provide feedback and suggestions, which may include modification, supplementation or deletion of the document content. The author will make revisions based on the reviewers' feedback and then resubmit the revised document to the review organization. This process is called a review round. Multiple review rounds usually indicate that the document has undergone a more rigorous review process. The reviewers have conducted multiple detailed inspections and revisions on the content of the document. The final approved document may be more accurate, complete and rigorous in content. Step S4, generating a supplementary query statement based on the information missing risk document set, and performing a secondary search in the knowledge base based on the supplementary query statement to obtain an information supplementary document set; The primary search documents in the information missing risk document set are segmented to extract important keywords and phrases.

[0018] Perform part-of-speech tagging on the words after word segmentation to identify words of different parts of speech such as nouns, verbs, and adjectives; Input the annotated words into the topic model to extract the document topics of the primary retrieval documents in the information-missing document set. The document topics include document topic words and document topic phrases. Compare the extracted document topics with the query topics, identify the topic information that is not covered in the information missing risk document set, and determine the topic areas that are not fully covered. These areas may include specific topics, time, geographic location, target users, etc. Generate multiple supplementary query statements based on the identified uncovered subject areas using a multi-query retriever. These supplementary query statements will expand the original query content from different perspectives to ensure that all aspects of the user's query are more fully covered in the secondary search. For example, if there is a lack of "low budget" related information under the tourism topic, you can generate supplementary queries such as "best low-budget travel destinations"; Use the generated supplementary query statement to conduct a secondary search in the knowledge base to obtain a set of supplementary information documents; In step S5, the retained document set, the redundant document screening set, and the information supplement document set are merged to obtain a final document set, and the document retrieval quality information and document freshness information of the final document set are obtained to generate a document priority recommendation index. The documents in the final document set are sorted from large to small according to the document priority recommendation index to obtain a document sorting table, and the documents are displayed to the user in order according to the document sorting order. The document retrieval quality information includes a document retrieval quality coefficient, and the document freshness information includes a document freshness coefficient; The document retrieval quality coefficient is used to measure the matching degree between the retrieved documents in the final document set and the user's input query. A higher document retrieval quality coefficient indicates that the matching degree between the retrieved documents in the final document set and the user's input query is higher, which reflects that the retrieval quality of the document is higher, and the corresponding probability of being recommended first is greater; The logic for obtaining the document retrieval quality coefficient is as follows: Preprocess the retrieved documents and user input queries in the final document set, including word segmentation, stop word removal, and stemming; Calculating document retrieval quality coefficient , the expression is as follows , where Represents the total number of retrieved documents in the final document set, Represents the i-th word in the user input query, i={1,2,...,n}, n is a positive integer, express The word frequency in the search document D, where the word frequency refers to the number of times the words in the user input query appear in the search document, Indicates the document length of the retrieved document D, represents the average length of the retrieved documents in the final document set, and is a constant factor, set according to actual conditions; M is a constant factor used to control the effect of word frequency on the document retrieval quality coefficient, which determines the saturation of word frequency in calculating the document retrieval quality coefficient. L is a constant factor used to adjust the effect of the document length of the retrieved document on word frequency, which controls the normalization effect of the retrieved document length in the calculation. The document freshness coefficient is used to measure the timeliness or update degree of the retrieved documents in the final document set. The higher the document freshness coefficient, the newer the content of the retrieved document, reflecting its greater relevance and value at the current time point. In the process of document sorting and recommendation, documents with higher freshness coefficients are more likely to be displayed to users first because they usually contain more timely and realistic information. The logic for obtaining the document freshness coefficient is as follows: Get the release time and user query time of the retrieved document, and calculate the time difference between the release time and the user query time of the retrieved document ; Get the content update time of each retrieved document and calculate the time difference between adjacent content updates , j represents the sequence number of the content update event trigger; Calculate the average time difference between adjacent content updates , the expression is as follows Where j = {1, 2, ..., J}; Calculate the standard deviation of the time difference between adjacent content updates , the expression is as follows ; Calculate document freshness coefficient , the expression is as follows ; Normalize the obtained document retrieval quality coefficient and document freshness coefficient, build a retrieval document screening model based on the normalized document retrieval quality coefficient and document freshness coefficient, and generate a document priority recommendation index The model is based on the following formula , where Respectively represent the preset proportional coefficients of the document retrieval quality coefficient and the document freshness coefficient, and All are greater than 0; It should be noted that , It is set according to the actual situation. For example, the expert empowerment method is adopted, that is, experts in related fields are invited to determine the preset proportion coefficients of various indicators through professional opinion surveys and comprehensive evaluations. In addition, various methods such as hierarchical analysis method and fuzzy comprehensive evaluation method can also be considered to determine the preset proportion coefficients to ensure the objectivity and scientificity of the preset proportion coefficients, which will not be elaborated here; From the above calculation expression, it can be seen that the greater the document retrieval quality coefficient and the greater the document freshness coefficient, the greater the document priority recommendation index, indicating that the greater the probability of the retrieved document being recommended first. On the contrary, the smaller the document retrieval quality coefficient and the smaller the document freshness coefficient, the smaller the document priority recommendation index, indicating that the probability of the retrieved document being recommended first is smaller. Sort the documents in the final document set from large to small according to the document priority recommendation index to obtain a document sorting table, and display the documents to the user in the order of document sorting; The present invention understands the user's query intention more accurately by initially parsing the query input by the user and generating core query statements and primary search document sets, performs cluster analysis on the primary search document set to identify redundant risk document sets, retained document sets and information missing risk document sets, obtains redundant detection information and credibility assessment information of the redundant risk document sets, performs initial screening on the redundant risk document sets to obtain redundant document screening sets, effectively filters redundant and less credible search documents, improves the quality and relevance of the final document set, generates supplementary query statements according to the information missing risk document set, performs secondary retrieval in the knowledge base according to the supplementary query statements, obtains information supplementary document sets, and more comprehensively covers the user's multi-angle needs and potential fuzzy intentions; The present invention generates a document priority recommendation index through comprehensive calculation of the document quality coefficient and the document freshness coefficient, ensuring that in the final document set, documents with high quality and strong timeliness are displayed to users first, effectively improving the user experience, so that users can obtain high-quality information that meets their needs faster and more accurately when searching, avoiding the problem of incomplete results caused by insufficient information coverage in traditional single query retrieval methods; The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0019] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0020] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A large language model multi-level retrieval enhancement method, characterized by: The steps include: Step S1, performing an initial analysis on the query input by the user, generating a core query statement based on the initial analysis result, and performing a primary search in the knowledge base based on the core query statement to obtain a primary search document set; Step S2, performing cluster analysis on the primary search document set to identify redundant risk document sets, retained document sets, and information missing risk document sets; Step S3, obtaining redundant detection information and credibility assessment information of the redundant risk document set, performing a primary screening on the redundant risk document set, and obtaining a redundant document screening set; Step S4, generating a supplementary query statement based on the information missing risk document set, and performing a secondary search in the knowledge base based on the supplementary query statement to obtain an information supplementary document set; Step S5, merging the retained document set, the redundant document screening set, and the information supplement document set to obtain a final document set, obtaining document retrieval quality information and document freshness information of the final document set, generating a document priority recommendation index, sorting the documents in the final document set from large to small according to the document priority recommendation index, obtaining a document sorting table, and displaying the documents to the user in order according to the document sorting order; The document retrieval quality information includes a document retrieval quality coefficient, and the document freshness information includes a document freshness coefficient; The logic for obtaining the document retrieval quality coefficient is as follows: Preprocess the retrieved documents and user input queries in the final document set, including word segmentation, stop word removal, and stemming; Calculating document retrieval quality coefficient , the expression is as follows , where Represents the total number of retrieved documents in the final document set, Represents the i-th word in the user input query, i={1,2,...,n}, n is a positive integer, express The frequency of the word in the retrieved document D, Indicates the document length of the retrieved document D, represents the average length of the retrieved documents in the final document set, and is a constant factor; The logic for obtaining the document freshness coefficient is as follows: Get the release time and user query time of the retrieved document, and calculate the time difference between the release time and the user query time of the retrieved document ; Get the content update time of each retrieved document and calculate the time difference between adjacent content updates , j represents the sequence number of the content update event trigger; Calculate the average time difference between adjacent content updates , the expression is as follows Where j = {1, 2, ..., J}; Calculate the standard deviation of the time difference between adjacent content updates , the expression is as follows ; Calculate document freshness coefficient , the expression is as follows .

2. The large language model multi-level retrieval enhancement method according to claim 1, characterized in that: In step S2, cluster analysis is performed on the primary search document set to identify redundant risk document sets, retained document sets, and information missing risk document sets, as follows: Step S21, preprocessing the primary search document set, including removing noise, standardizing the text format, and converting each preprocessed primary search document into a vector representation; Step S22, using the elbow method to determine the number K of initial cluster centroids; Step S23, according to the determined number K of initial clustering centroids, randomly selecting K primary retrieval documents from the primary retrieval document set as initial clustering centroids; Step S24, calculating the Euclidean distance between each primary search document in the primary search document set and the centroid of the initial cluster, and assigning each primary search document to the cluster closest to it; Step S25, calculating the average vector of each cluster and taking it as the new cluster centroid; Step S26, repeating steps S24 and S25 to iteratively update the clusters until the centroid of the clusters no longer changes, thereby obtaining a final cluster set; Step S27: preprocess the core query statement, convert the preprocessed core query statement into a vector representation, and use the cosine similarity method to calculate the first cosine similarity between each primary search document in the cluster and the core query statement. , the expression is as follows , where represents the vector representation of the primary retrieved document in the cluster, Represents the vector representation of the core query statement, express and The dot product operation, represents the Euclidean norm of the primary retrieval document vector, Represents the Euclidean norm of the core query statement vector; In the cluster, count the number of occurrences C1 where the first cosine similarity between the primary search document and the core query statement is greater than 0 and less than or equal to 1; count the number of occurrences C2 where the first cosine similarity between the primary search document and the core query statement is less than 0 and greater than or equal to -1, and calculate the similarity ratio , the expression is as follows ; Step S28, comparing the similarity ratio with a preset similarity ratio stage threshold, marking the clusters, identifying the redundant risk document set, the retained document set, and the information missing risk document set, the similarity ratio stage threshold includes a similarity ratio first threshold and a similarity ratio second threshold, and the similarity ratio first threshold is less than the similarity ratio second threshold; If the similarity ratio is greater than the second similarity ratio threshold, the cluster is marked as a redundant risk cluster, and the primary search document in the redundant risk cluster is added to the redundant risk document set; If the similarity ratio is less than or equal to the second similarity ratio threshold and greater than or equal to the first similarity ratio threshold, the cluster is marked as a normal cluster, and the primary search documents in the normal cluster are added to the reserved document set; If the similarity ratio is less than the first similarity ratio threshold, the cluster is marked as an information missing risk cluster, and the primary search documents in the information missing risk cluster are added to the information missing risk document set.

3. The large language model multi-level retrieval enhancement method according to claim 2, characterized in that: In step S3, redundant detection information and credibility assessment information of the redundant risk document set are obtained, and the redundant risk document set is initially screened to obtain a redundant document screening set, as follows: The redundant detection information includes a second cosine similarity, and the credibility evaluation information includes a credibility evaluation coefficient; The cosine similarity method is used to calculate the second cosine similarity between two primary retrieval documents in the redundant risk document set, and the second cosine similarity is compared with a preset redundancy threshold. If the second cosine similarity is greater than the redundancy threshold, the credibility evaluation coefficients of the two primary retrieval documents are respectively obtained, and the credibility evaluation coefficients of the two primary retrieval documents are compared. The primary retrieval document with the smallest credibility evaluation coefficient is marked as a document to be deleted. If the credibility evaluation coefficients of the two primary retrieval documents are equal, one of the primary retrieval documents is randomly marked as a document to be deleted; The number of times each primary retrieval document in the redundant risk document set is marked as a document to be deleted is counted, the number of times the document is marked as a document to be deleted is compared with a preset marking threshold, and the primary retrieval documents whose number is less than the marking threshold are added to the redundant document screening set.

4. The large language model multi-level retrieval enhancement method according to claim 3, characterized in that: The logic for obtaining the credibility evaluation coefficient is as follows: Get the percentage of self-citations in primary search documents , the expression is as follows , where Indicates the number of times the primary search document cites the author's previously published documents. Indicates the number of all documents cited in the primary search document; Get the number of citations of primary search documents , Number of review rounds , Number of rejections ; Normalize the percentage of self-citations, the number of citations, the number of review rounds, and the number of rejections to calculate the credibility assessment coefficient , the expression is as follows .

5. The method for enhancing multi-level retrieval using a large language model according to claim 4, characterized in that: Normalize the obtained document retrieval quality coefficient and document freshness coefficient, build a retrieval document screening model based on the normalized document retrieval quality coefficient and document freshness coefficient, and generate a document priority recommendation index The model is based on the following formula , where Respectively represent the preset proportional coefficients of the document retrieval quality coefficient and the document freshness coefficient, and All are greater than 0; According to the document priority recommendation index, the documents in the final document set are sorted from large to small to obtain a document sorting table, and the documents are displayed to the user in sequence according to the document sorting order.

Images