Document retrieval method based on multi-field information and outlier detection

Abstract

The application relates to a document retrieval method based on multi-field information and outlier detection. The method comprises the following steps: acquiring document query information used for retrieving a document, and acquiring field similarity between the document query information and different field information of each candidate document; if at least one high outlier exists in current field similarity between the document query information and any field information of a current candidate document, obtaining document similarity between the document query information and the current candidate document according to each high outlier; if no high outlier exists in the current field similarity, obtaining the document similarity between the document query information and the current candidate document according to at least one current field similarity; and obtaining a document retrieval result corresponding to the document query information according to the document similarity between the document query information and each candidate document. The method can improve the relevance of the document retrieval result.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document retrieval method based on multi-field information and outlier detection. Background Technology

[0002] With the development of large-scale pre-trained language models and vectorization techniques, Retrieval-Augmented Generation (RAG) systems have become an important architecture for handling knowledge-intensive tasks. The core of a RAG system is to retrieve the most relevant document chunks to the user's query from a knowledge base using vector search, and then provide these chunks as context to a large model to generate the final answer.

[0003] During retrieval, the RAG system searches in parallel across the vector indexes of different fields of a document (such as title, body text details, abstract, etc.), returning a list of documents sorted based on their own similarity scores for each field. However, the distribution range of vector index scores varies across different document fields, making the search result scores incomparable and distorting the generated document ranking results. Consequently, it fails to accurately retrieve documents highly relevant to the user's query.

[0004] Therefore, traditional technologies for enhancing retrieval suffer from the problem of low relevance between document retrieval results and user queries. Summary of the Invention

[0005] Therefore, it is necessary to provide a document retrieval method, apparatus, computer equipment, readable storage medium, and computer program product that can improve the relevance of document retrieval results to user queries based on multi-field information and outlier detection, in order to address the above-mentioned technical problems.

[0006] Firstly, this application provides a document retrieval method based on multi-field information and outlier detection, the method comprising the following steps:

[0007] Obtain document query information for retrieving documents, and obtain the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base;

[0008] If there is at least one high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained according to each of the high outlier values; the current candidate document is any one of the candidate documents; the high outlier value is an outlier value that is greater than the outlier value boundary;

[0009] If there are no high outliers in the current field similarity between the document query information and any of the field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on at least one of the current field similarities.

[0010] Based on the document similarity between the document query information and each of the candidate documents, the document retrieval results corresponding to the document query information are obtained.

[0011] In one embodiment, after obtaining the field similarity between the document query information and the different field information of each candidate document, the method further includes:

[0012] Based on the similarity of each of the current fields, obtain the outlier boundary corresponding to the current candidate document;

[0013] The current field similarity that exceeds the outlier boundary is identified as a high outlier.

[0014] In one embodiment, obtaining the outlier boundary corresponding to the current candidate document includes:

[0015] Obtain the first quartile and the third quartile corresponding to the similarity of each of the current fields;

[0016] The interquartile range is obtained based on the difference between the third quartile and the first quartile.

[0017] The outlier boundary is obtained by using the third quartile, the interquartile range, and a preset adjustment parameter.

[0018] The step of identifying high outliers from the current field similarity based on the current field similarity and the outlier boundary includes:

[0019] The current field similarity that exceeds the outlier boundary is identified as a high outlier.

[0020] In one embodiment, obtaining the outlier boundary corresponding to the current candidate document includes:

[0021] Obtain the mean and standard deviation of the similarity of each of the current fields;

[0022] The outlier boundary is obtained using the mean, the standard deviation, and a pre-set standard score threshold.

[0023] In one embodiment, obtaining the document similarity between the document query information and the current candidate document based on at least one of the current field similarities includes:

[0024] The average of the similarities of each of the current fields is taken as the document similarity.

[0025] or

[0026] The median of the similarity scores for each of the current fields is used as the document similarity score.

[0027] or

[0028] The average of the similarities of each target field is taken as the document similarity; the target field similarity is the top preset number of current field similarities with the highest similarity among all the current field similarities.

[0029] In one embodiment, the field information includes title information, body content information, and summary information; the step of obtaining document query information for retrieving documents and obtaining the field similarity between the document query information and different field information of each candidate document includes:

[0030] Obtain the user query, vectorize the user query, and obtain the document query information;

[0031] Obtain a first similarity between the document query information and the title information of each candidate document, a second similarity between the document query information and the main text information of each candidate document, and a third similarity between the document query information and the summary information of each candidate document;

[0032] Each of the first similarity, each of the second similarity, and each of the third similarity are normalized, and the normalized first similarity, each of the second similarity, and each of the third similarity are used as the field similarity between the document query information and the different field information of each candidate document.

[0033] Secondly, this application provides a document retrieval device based on multi-field information and outlier detection, the device comprising:

[0034] The field similarity acquisition module is used to acquire document query information for retrieving documents, and to acquire the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base;

[0035] The first document similarity acquisition module is used to obtain the document similarity between the document query information and the current candidate document based on each of the high outlier values ​​if there is at least one high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document; the current candidate document is any one of the candidate documents.

[0036] The second document similarity acquisition module is used to obtain the document similarity between the document query information and the current candidate document based on at least one of the current field similarities if there is no high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document.

[0037] The document retrieval result generation module is used to obtain the document retrieval result corresponding to the document query information based on the document similarity between the document query information and each of the candidate documents.

[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the document retrieval method based on multi-field information and outlier detection described above.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0040] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the above-described method.

[0041] The aforementioned document retrieval method, apparatus, computer equipment, readable storage medium, and computer program product based on multi-field information and outlier detection acquire document query information for document retrieval and obtain field similarity between the document query information and different field information of each candidate document. If at least one high outlier value exists in the current field similarity between the document query information and any field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on each high outlier value. A high outlier value is an outlier value greater than the outlier value boundary. If no high outlier value exists in the current field similarity between the document query information and any field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on at least one current field similarity value. By calculating the document similarity between the query information and each candidate document, and obtaining the document retrieval results corresponding to the document query information, it is possible to obtain the similarity scores of the multi-field vector retrieval results of each candidate document. The similarity scores of different fields are first normalized to convert similarity scores from different fields with different dimensions into uniform and comparable field similarities. Then, outlier detection is used to fuse the field similarities of each document in each field, so as to obtain a document similarity that highly reflects the relevance between the document query information and each candidate document. In this way, documents that are highly relevant to the user query can be accurately identified, thereby improving the relevance of the document retrieval results and solving the problem of low relevance between document retrieval results and user queries in traditional retrieval enhancement generation technology. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 This is an application environment diagram of a document retrieval method based on multi-field information and outlier detection in one embodiment;

[0044] Figure 2 This is a flowchart illustrating a document retrieval method based on multi-field information and outlier detection in one embodiment.

[0045] Figure 3 This is a flowchart illustrating another document retrieval method based on multi-field information and outlier detection in one embodiment;

[0046] Figure 4This is a structural block diagram of a document retrieval device based on multi-field information and outlier detection in one embodiment;

[0047] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0049] With the development of large-scale pre-trained language models and vectorization techniques, Retrieval-Augmented Generation (RAG) systems have become an important architecture for handling knowledge-intensive tasks. In a RAG system, the core step is to retrieve the most relevant document chunks (chunks) to the user's query from the knowledge base through vector search and provide them as context to the generative model. To improve retrieval accuracy, traditional techniques typically vectorize and index different parts of a single document (such as title, body text, abstract, etc.). During retrieval, the system searches in parallel across these vector indexes for different fields, returning a list of documents sorted based on their own similarity scores for each field.

[0050] While traditional techniques enrich retrieval sources through multi-field searches, they suffer from significant technical drawbacks when merging search results from different fields: 1. Incomparable scores: The vector indexes of different fields naturally exhibit varying score distribution ranges. For example, the cosine similarity score of the title field may generally be concentrated in the high range of [0.8, 1.0], while the score of the details (body text) field may be widely distributed in the range of [0.4, 0.9]. These scores lack comparability, and simple merging and addition can distort the ranking results. 2. Suboptimal ranking results: Due to the incomparability of scores, simple merging strategies (such as directly taking the highest score from each field) cannot accurately reflect the overall relevance of the document. This may result in a document with an extremely high score in one field but irrelevant scores in other fields ranking higher than a document with good relevance in all fields, thus reducing the probability of retrieving high-quality contextual material. 3. Rigid strategies and lack of adaptability: Existing methods typically employ fixed strategies (such as hard-coded weights) for merging, failing to dynamically adjust based on the score distribution characteristics of each query's results. This lack of flexibility makes it difficult to maintain optimal performance across various query scenarios.

[0051] Therefore, there is an urgent need for a normalization method that can effectively solve the problem of incomparability of multi-field vector search result scores and adaptively fuse results, so as to improve the relevance of document retrieval results to user queries and thus enhance the overall performance of the RAG system's retrieval process.

[0052] Therefore, this application provides a document retrieval method based on multi-field information and outlier detection. By normalizing and fusing multi-field vector search results for RAG systems, it can effectively solve the problem of incomparable scores between different fields, and generate the optimal final ranking list through an adaptive fusion strategy.

[0053] The document retrieval method based on multi-field information and outlier detection provided in this application can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Server 104 can obtain document query information for document retrieval and obtain the field similarity between the document query information and different field information of each candidate document; candidate documents are documents stored in a preset knowledge base. If there is at least one high outlier value in the current field similarity between the document query information and any field information of the current candidate document, server 104 can obtain the document similarity between the document query information and the current candidate document based on each high outlier value; the current candidate document is any one of the candidate documents; a high outlier value is an outlier value greater than the outlier value boundary. If there is no high outlier value in the current field similarity between the document query information and any field information of the current candidate document, server 104 can obtain the document similarity between the document query information and the current candidate document based on at least one current field similarity. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and IoT devices. Server 104 obtains document retrieval results corresponding to the document query information based on the document similarity between the document query information and each candidate document. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In one exemplary embodiment, such as Figure 2 As shown, a document retrieval method based on multi-field information and outlier detection is provided. This embodiment applies this method to... Figure 1 The method is illustrated using server 104 as an example. In this embodiment, the method includes the following steps S202 to S208. Wherein:

[0055] Step S202: Obtain document query information for retrieving documents, and obtain the field similarity between the document query information and different field information of each candidate document.

[0056] Candidate documents can be documents stored in a preset knowledge base.

[0057] Among them, document query information can be processed and used by users to retrieve documents.

[0058] The field information can be various parts of a document that can be used for retrieval, such as the title, body content, and abstract.

[0059] In practical applications, server 104 can process unstructured user queries to obtain document query information for document retrieval. Based on the field information of the document query information in a preset knowledge base, it can perform a similarity search to retrieve the field information of candidate documents related to the user query, obtain the similarity score of each field information of each candidate document, and then normalize the similarity score to obtain the field similarity between the document query information and the different field information of each candidate document.

[0060] Step S204: If there is at least one high outlier value in the current field similarity between the document query information and any field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on each high outlier value.

[0061] The current candidate document can be any one of the candidate documents; the current field similarity can be the field similarity of the field information in the current candidate document.

[0062] Optionally, a high outlier can be an outlier that is greater than the outlier boundary.

[0063] In practical applications, server 104 can perform outlier detection on the field similarity of each candidate document based on the outlier strategy in order to determine the document similarity of each candidate document.

[0064] For any current candidate document, if there is at least one high outlier in the current field similarity between the document query information and any field information of the current candidate document, an outlier value greater than the outlier value boundary can be selected as the document similarity between the document query information and the current candidate document.

[0065] Step S206: If there are no high outliers in the current field similarity between the document query information and any field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on at least one current field similarity.

[0066] In the specific implementation, for the current candidate document, if there are no high outliers in the current field similarity between the document query information and any field information of the current candidate document, the server 104 can obtain the document similarity between the document query information and the current candidate document based on the field similarity.

[0067] Step S208: Based on the document similarity between the document query information and each candidate document, obtain the document retrieval results corresponding to the document query information.

[0068] In the specific implementation, after outlier detection on all candidate documents, server 104 can determine the document similarity between the document query information and each candidate document, and then sort the document similarity corresponding to all candidate documents to obtain the document retrieval results corresponding to the document query information.

[0069] The document retrieval method described above, based on multi-field information and outlier detection, obtains document query information for document retrieval and acquires field similarity between the document query information and different fields of each candidate document. If at least one high outlier exists in the current field similarity between the document query information and any field of the current candidate document, the document similarity between the document query information and the current candidate document is obtained based on each high outlier. A high outlier is an outlier value greater than the outlier boundary. If no high outlier exists in the current field similarity between the document query information and any field of the current candidate document, the document similarity between the document query information and the current candidate document is obtained based on at least one current field similarity. By calculating document similarity, document retrieval results corresponding to document query information can be obtained. This involves acquiring the similarity scores of multi-field vector retrieval results for each candidate document. The similarity scores for different fields are first normalized to convert similarity scores from different fields with different dimensions into a unified and comparable field similarity. Then, outlier detection is used to fuse the field similarities of each document across various fields, resulting in a document similarity score that highly reflects the relevance between the document query information and each candidate document. This allows for the accurate identification of documents highly relevant to the user query, thereby improving the relevance of document retrieval results and solving the problem of low relevance between document retrieval results and user queries in traditional retrieval enhancement generation techniques.

[0070] In an exemplary embodiment, after obtaining the field similarity between the document query information and the different field information of each candidate document, the method further includes: obtaining the outlier boundary corresponding to the current candidate document based on the current field similarity; and determining the current field similarity greater than the outlier boundary as a high outlier.

[0071] For example, the same candidate document may be found in the search results of multiple fields. Therefore, the similarity of each field of each candidate document can be grouped according to the document identifier of each candidate document. Thus, for any current candidate document, the field similarity between the document query information and each field information in the current candidate document can be obtained.

[0072] In practical applications, server 104 can calculate the corresponding descriptive statistical indicators based on the field similarity of each current candidate document; then, based on the descriptive statistical indicators and pre-set adjustment parameters, it can calculate the outlier boundary corresponding to the current candidate document; and can identify high outliers from the current field similarity by defining current field similarity values ​​greater than the outlier boundary. For example, the descriptive statistical indicators can be quartiles, mean, standard deviation, median, etc.

[0073] The technical solution of this embodiment obtains the outlier boundary corresponding to the current candidate document and identifies high outliers from the current field similarity. It can realize an adaptive fusion strategy based on outliers. It can adjust according to the specific distribution of field similarity of the search results returned by each query, rather than relying on fixed parameters of different fields, so as to capture the extreme relevance of the document on a certain field. In this way, it can more accurately identify documents that are highly related to the document query information corresponding to the user query.

[0074] In an exemplary embodiment, obtaining the outlier boundary corresponding to the current candidate document includes: obtaining the first quartile and the third quartile corresponding to the similarity of each current field; obtaining the interquartile range based on the difference between the third quartile and the first quartile; and obtaining the outlier boundary using the third quartile, the interquartile range, and a pre-set adjustment parameter.

[0075] Based on the similarity of each current field and the outlier boundary, identify high outliers from the current field similarity, including: identifying current field similarities greater than the outlier boundary as high outliers.

[0076] In the specific implementation, server 104 can calculate the first quartile corresponding to the field similarity of each current candidate document. Third and quartiles Then, based on the difference between the third quartile and the first quartile, the interquartile range is obtained. ,Right now Then use the third quartile Interquartile range and preset adjustment parameters This yields the outlier boundary, i.e. Values ​​greater than the outlier boundary can be considered. The corresponding current field similarity is identified as a high outlier, and the maximum value among the high outliers is used as the document similarity. Optionally, pre-set adjustment parameters... It can be 1.5.

[0077] The technical solution of this embodiment uses the interquartile range method to determine the high outlier values ​​of the similarity of each field in each candidate document, and uses the maximum value among the high outlier values ​​as the document similarity, thereby effectively detecting outliers in non-normally distributed field similarity datasets, and providing a reliable detection method for accurately identifying highly relevant search documents.

[0078] In an exemplary embodiment, obtaining the outlier boundary corresponding to the current candidate document includes: obtaining the mean and standard deviation of the similarity of each current field; and obtaining the outlier boundary using the mean, standard deviation, and a pre-set standard score threshold.

[0079] In the specific implementation, server 104 can calculate the mean value corresponding to the field similarity of each current candidate document. and standard deviation Then use the mean Standard deviation and preset adjustment parameters Obtain the outlier boundary. It can be greater than the outlier boundary. The corresponding current field similarity is identified as a high outlier, and the maximum value among the high outliers is used as the document similarity. Optionally, pre-set adjustment parameters... That is, the standard score threshold. It can be 2.5 or 3.

[0080] For example, server 104 can also use a preset standard score threshold as the outlier boundary; first, obtain the standard score corresponding to the similarity of each current field. If the standard score corresponding to the current field similarity is greater than the standard score threshold, the current field similarity is considered a high outlier. Optionally, the standard score threshold... It can be 2.5 or 3.

[0081] The technical solution of this embodiment determines the high outlier values ​​of the similarity of each field in each candidate document by using the standard score method, and takes the maximum value among the high outlier values ​​as the document similarity, thereby effectively detecting outlier values ​​in normally distributed or approximately normally distributed field similarity datasets, and providing a reliable detection method for accurately identifying highly relevant search documents.

[0082] In an exemplary embodiment, obtaining the document similarity between the document query information and the current candidate document based on at least one current field similarity includes: taking the average of the current field similarities as the document similarity; or taking the median of the current field similarities as the document similarity; or taking the average of the target field similarities as the document similarity.

[0083] The target field similarity is the number of current field similarities with the highest similarity among all current field similarities. For example, it can be the Top-K score.

[0084] In a specific implementation, if there are no high outliers in the current field similarity between the document query information and any field information of the current candidate document, the server 104 may use any one of the average value, median value of each current field similarity, and the average value of the top preset number of current field similarities with the highest similarity among each current field similarity as the document similarity between the document query information and the current candidate document.

[0085] The technical solution of this embodiment uses the average, median, and average Top-K score of the similarity of each current field as the document similarity between the document query information and the current candidate document. Based on this, a backoff strategy can be implemented when no outliers are detected. This ensures that even when high outliers do not exist, i.e. when the data distribution of the similarity of each field of the current candidate document is relatively concentrated, highly relevant search documents can still be accurately identified.

[0086] In an exemplary embodiment, obtaining document query information for retrieving documents and obtaining field similarity between the document query information and different field information of each candidate document includes: obtaining a user query, vectorizing the user query to obtain document query information; obtaining a first similarity between the document query information and the title information of each candidate document, a second similarity between the document query information and the body content information of each candidate document, and a third similarity between the document query information and the summary information of each candidate document; normalizing each first similarity, each second similarity, and each third similarity, and using the normalized first similarity, each second similarity, and each third similarity as the field similarity between the document query information and different field information of each candidate document.

[0087] It should be noted that the field information includes title information, body content information, and summary information. This information can be the result of vectorizing the documents in the knowledge base, and it includes vector indexes for various fields such as the title, body content, and summary of the document.

[0088] Among them, document query information can be user queries that have been vectorized and can be used to retrieve documents.

[0089] In the specific implementation, server 104 can obtain user queries, vectorize the user queries to obtain user queries that have been vectorized and indexed, and can be used to retrieve documents, i.e., document query information; then, based on the document query information and the field information of each candidate document, it performs similarity retrieval on the title information, body content information, and summary information of each candidate document in the knowledge base, respectively, to obtain the first similarity score, second similarity score, and third similarity score between the document query information and the title information, body content information, and summary information of each candidate document, and obtain a list of each similarity score; then, for the similarity score list of each field information, it normalizes each first similarity score, each second similarity score, and each third similarity score, and uses the normalized first similarity score, each second similarity score, and each third similarity score as the field similarity between the document query information and the different field information of each candidate document.

[0090] For example, the normalization (also known as standardization) of each similarity can be performed using the Min-Max normalization method (also known as linear normalization), that is, normalized score = (original similarity score - minimum value of the current similarity score list) / (maximum value of the current similarity score list - minimum value of the current similarity score list).

[0091] In this application embodiment, the normalization method is not limited, as long as it can unify the field similarity of different fields to a comparable scale. For example, Z-Score normalization (standard score method, also known as zero-mean normalization) can be used, i.e., normalized score = (original similarity score - mean of the current similarity score list) / standard deviation of the current similarity score list; decimal scaling normalization can also be used, i.e., normalized score = original similarity score / ,in It can be to move the original similarity score. The integer power of the decimal point position.

[0092] The technical solution in this embodiment vectorizes the user query and retrieves the field similarity of each candidate document from the knowledge base, thereby enabling precise retrieval of documents related to the user query through vector search. Furthermore, by normalizing the field similarity scores, similarity scores from different fields and scales are converted into a unified and comparable standard score, ensuring the uniformity and comparability of different field similarities. This allows for good adaptability and stability for queries with different characteristics, laying an accurate foundation for subsequent fusion of different field similarities among candidate documents.

[0093] In an exemplary embodiment, the document retrieval result corresponding to the document query information is obtained based on the document similarity between the document query information and each candidate document, including: sorting the document similarity of each candidate document and generating sorted document retrieval results.

[0094] For example, document sorting results can be used as input to the generation component of a retrieval enhancement generation system.

[0095] The technical solution of this embodiment, by sorting the document similarity of each candidate document after the "normalization and fusion" process, can identify documents that are highly relevant to the document query information, optimize the document retrieval results corresponding to the document query information, and thus provide the generation component of the RAG (Retrieval Enhanced Generation) system with higher quality and more accurate basic document materials, improve the accuracy, relevance and reliability of the final generated answer, and improve the user experience.

[0096] In a specific embodiment, such as Figure 3 As shown in the embodiment of this application, a document retrieval method based on multi-field information and outlier detection is provided, including:

[0097] S1: Multi-field parallel vector search: Receives user queries, vectorizes them, and performs similarity searches in the title vector index, detail vector index, and summary vector index of the knowledge base. Each index returns an ordered list containing document IDs and their original similarity scores.

[0098] S2: Score Normalization Processing: For each field's returned score list, independent normalization processing is performed to convert it into a standard score conforming to a predetermined distribution. This application embodiment preferably uses the minimum-maximum normalization method based on the current query results. The calculation formula is: Normalized Score = (Original Score - Current List Minimum) / (Current List Maximum - Current List Minimum). This step uniformly maps scores from different fields to the [0, 1] interval, solving the core problem of score incomparability.

[0099] S3: Adaptive Fusion Based on Outlier Detection: For the same document, if it appears in the search results of multiple fields, its final score is fused from the normalized scores of these fields. The key to this application's embodiment lies in employing an outlier-based strategy during fusion, specifically: a) For each document, collect its normalized scores across all hit fields; b) Use the interquartile range (IQR) method to detect high outliers in these scores. Calculate the first quartile (Q1) and third quartile (Q3) of these scores, defining the outlier boundary as Q3 + k * IQR (k is an adjustable parameter, defaulting to 1.5); c) If high outliers exist, take the maximum value of all high outliers as the document's final score; otherwise, revert to taking the average of all normalized scores. This strategy prioritizes capturing extreme relevance exhibited by a document in a single field while also considering cases where multiple fields are correlated.

[0100] S4: Generate the final sorted list: Sort all documents in descending order according to the final score calculated in step S3, generate the final search results list, and output it to the generation component of the RAG system.

[0101] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0102] Based on the same inventive concept, this application also provides a document retrieval device based on multi-field information and outlier detection for implementing the document retrieval method based on multi-field information and outlier detection described above. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more embodiments of the document retrieval device based on multi-field information and outlier detection provided below can be found in the limitations of the document retrieval method based on multi-field information and outlier detection described above, and will not be repeated here.

[0103] In an exemplary embodiment, as shown in FIG4, a document retrieval device based on multi-field information and outlier detection is provided, including:

[0104] The field similarity acquisition module 410 is used to acquire document query information for retrieving documents, and to acquire the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base;

[0105] The first document similarity acquisition module 420 is used to obtain the document similarity between the document query information and the current candidate document based on each high outlier value if there is at least one high outlier value in the current field similarity between the document query information and any field information of the current candidate document; the current candidate document is any one of the candidate documents.

[0106] The second document similarity acquisition module 430 is used to obtain the document similarity between the document query information and the current candidate document based on at least one current field similarity if there is no high outlier value in the current field similarity between the document query information and any field information of the current candidate document.

[0107] The document retrieval result generation module 440 is used to obtain the document retrieval results corresponding to the document query information based on the document similarity between the document query information and each candidate document.

[0108] In one embodiment, the first document similarity acquisition module 420 is further configured to acquire the outlier boundary corresponding to the current candidate document based on the similarity of each current field; and to determine the current field similarity greater than the outlier boundary as a high outlier.

[0109] In one embodiment, the first document similarity acquisition module 420 is further configured to acquire the first quartile and the third quartile corresponding to each current field similarity; obtain the interquartile range based on the difference between the third quartile and the first quartile; obtain the outlier boundary using the third quartile, the interquartile range, and a pre-set adjustment parameter; and identify high outliers from the current field similarities based on each current field similarity and the outlier boundary, including: determining current field similarities greater than the outlier boundary as high outliers.

[0110] In one embodiment, the first document similarity acquisition module 420 is further configured to acquire the mean and standard deviation corresponding to the similarity of each current field; and to obtain the outlier boundary using the mean, standard deviation and a pre-set standard score threshold.

[0111] In one embodiment, the second document similarity acquisition module 430 is further configured to use the average value of the similarity of each current field as the document similarity; or use the median of the similarity of each current field as the document similarity; or use the average value of the similarity of each target field as the document similarity; the target field similarity is the first preset number of current field similarities with the highest similarity among the current field similarities.

[0112] In one embodiment, the field similarity acquisition module 410 is further configured to acquire user queries, vectorize user queries to obtain document query information; acquire a first similarity between the document query information and the title information of each candidate document, a second similarity between the document query information and the body content information of each candidate document, and a third similarity between the document query information and the summary information of each candidate document; normalize each first similarity, each second similarity, and each third similarity, and use the normalized first similarity, each second similarity, and each third similarity as the field similarity between the document query information and different field information of each candidate document.

[0113] The modules in the aforementioned document retrieval device based on multi-field information and outlier detection can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0114] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 5. The computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor of the computer device provides computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the computer device stores document retrieval data. The I / O interfaces of the computer device are used for exchanging information between the processor and external devices. The communication interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a document retrieval method based on multi-field information and outlier detection.

[0115] Those skilled in the art will understand that the structure shown in Figure 5 is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0116] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the various embodiments of the document retrieval method described above.

[0117] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the document retrieval method embodiments described above.

[0118] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps described in various embodiments of the computational document retrieval method.

[0119] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0120] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence processors, etc., and are not limited to these.

[0121] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0122] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A document retrieval method based on multi-field information and outlier detection, characterized in that, The method includes: Obtain document query information for retrieving documents, and obtain the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base; If there is at least one high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained according to each of the high outlier values; the current candidate document is any one of the candidate documents; the high outlier value is an outlier value that is greater than the outlier value boundary; If there are no high outliers in the current field similarity between the document query information and any of the field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on at least one of the current field similarities. Based on the document similarity between the document query information and each of the candidate documents, the document retrieval results corresponding to the document query information are obtained.

2. The method according to claim 1, characterized in that, After obtaining the field similarity between the document query information and the different field information of each candidate document, the method further includes: Based on the similarity of each of the current fields, obtain the outlier boundary corresponding to the current candidate document; The current field similarity that exceeds the outlier boundary is identified as a high outlier.

3. The method according to claim 2, characterized in that, The step of obtaining the outlier boundary corresponding to the current candidate document based on the similarity of each of the current fields includes: Obtain the first quartile and the third quartile corresponding to the similarity of each of the current fields; The interquartile range is obtained based on the difference between the third quartile and the first quartile. The outlier boundary is obtained by using the third quartile, the interquartile range, and a pre-set adjustment parameter.

4. The method according to claim 2, characterized in that, The step of obtaining the outlier boundary corresponding to the current candidate document based on the similarity of each of the current fields includes: Obtain the mean and standard deviation of the similarity of each of the current fields; The outlier boundary is obtained using the mean, the standard deviation, and a pre-set standard score threshold.

5. The method according to claim 1, characterized in that, The step of obtaining the document similarity between the document query information and the current candidate document based on at least one of the current field similarities includes: The average of the similarities of each of the current fields is taken as the document similarity. or The median of the similarities of each of the current fields is used as the document similarity. or The average of the similarities of each target field is taken as the document similarity; the target field similarity is the top preset number of current field similarities with the highest similarity among all the current field similarities.

6. The method according to any one of claims 1 to 5, characterized in that, The field information includes title information, body content information, and summary information; obtaining document query information for retrieving documents, and obtaining the field similarity between the document query information and different field information of each candidate document, includes: Obtain the user query, vectorize the user query, and obtain the document query information; Obtain a first similarity between the document query information and the title information of each candidate document, a second similarity between the document query information and the main text information of each candidate document, and a third similarity between the document query information and the summary information of each candidate document; Each of the first similarity, each of the second similarity, and each of the third similarity are normalized, and the normalized first similarity, each of the second similarity, and each of the third similarity are used as the field similarity between the document query information and the different field information of each candidate document.

7. A document retrieval device based on multi-field information and outlier detection, characterized in that, The device includes: The field similarity acquisition module is used to acquire document query information for retrieving documents, and to acquire the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base; The first document similarity acquisition module is used to obtain the document similarity between the document query information and the current candidate document based on each of the high outlier values ​​if there is at least one high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document; the current candidate document is any one of the candidate documents. The second document similarity acquisition module is used to obtain the document similarity between the document query information and the current candidate document based on at least one of the current field similarities if there is no high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document. The document retrieval result generation module is used to obtain the document retrieval result corresponding to the document query information based on the document similarity between the document query information and each of the candidate documents.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images