Document retrieval method based on multi-field information and outlier detection

Abstract

The invention relates to a document retrieval method based on multi-field information and outlier detection. The method comprises the following steps: acquiring document query information for retrieving documents, and acquiring field similarity between the document query information and different field information of each candidate document; if at least one high outlier exists in the current field similarity between the document query information and any one piece of field information of the current candidate document, obtaining the document similarity between the document query information and the current candidate document according to each high outlier; if the high outlier does not exist in the current field similarity, obtaining the document similarity between the document query information and the current candidate document according to at least one current field similarity; and obtaining a document retrieval result corresponding to the document query information according to the document similarity between the document query information and each candidate document. By adopting the method, the correlation of document retrieval results can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, in particular to a document retrieval method based on multi-field information and outlier detection. BACKGROUND

[0002] With the development of large-scale pre-training language models and vectorization technology, retrieval-augmented generation (RAG) systems have become an important architecture for handling knowledge-intensive tasks. In the RAG system, the core is to retrieve the most relevant document chunks from the knowledge base through vector search and provide them as context to the large model to generate the final answer.

[0003] During retrieval, the RAG system will search in the vector index of different fields (such as title, detailed text, abstract, etc.) of the document in parallel, and each field will return a document list sorted based on its own similarity score. However, the distribution range of the vector index scores of different document fields is different, which causes the search result scores of different fields to be incomparable, making the generated document ranking result distorted, and thus the highly relevant documents to the user query cannot be accurately retrieved.

[0004] Therefore, the traditional technology has the problem of low relevance between the document retrieval result and the user query in the retrieval-augmented generation technology. SUMMARY

[0005] Therefore, it is necessary to provide a document retrieval method based on multi-field information and outlier detection to improve the relevance between the document retrieval result and the user query in the retrieval-augmented generation technology.

[0006] In a first aspect, the present application provides a document retrieval method based on multi-field information and outlier detection, which comprises the following steps:

[0007] Obtaining document query information for retrieving documents, and obtaining field similarity between the document query information and different field information of each candidate document; the candidate document is a document stored in a preset knowledge base;

[0008] If there is at least one high outlier in the current field similarity between the document query information and any field information of the current candidate document, obtaining the document similarity between the document query information and the current candidate document according to each high outlier; the current candidate document is any one of the candidate documents; the high outlier is an outlier greater than the outlier boundary;

[0009] if there is no high outlier in the current field similarities between the document query information and any of the field information of the current candidate document, obtaining a document similarity between the document query information and the current candidate document according to at least one of the current field similarities;

[0010] obtaining a document retrieval result corresponding to the document query information according to the document similarities between the document query information and each of the candidate documents.

[0011] In one of the embodiments, after the field similarities between the document query information and different field information of each candidate document are obtained, the method further comprises:

[0012] obtaining an outlier boundary corresponding to the current candidate document according to each of the current field similarities;

[0013] determining the current field similarity greater than the outlier boundary as a high outlier.

[0014] In one of the embodiments, the obtaining of the outlier boundary corresponding to the current candidate document comprises:

[0015] obtaining a first quartile and a third quartile corresponding to each of the current field similarities;

[0016] obtaining a quartile range according to a difference between the third quartile and the first quartile;

[0017] obtaining the outlier boundary by using the third quartile, the quartile range and a preset adjustment parameter;

[0018] The identifying of the high outlier from the current field similarities according to each of the current field similarities and the outlier boundary comprises:

[0019] determining the current field similarity greater than the outlier boundary as a high outlier.

[0020] In one of the embodiments, the obtaining of the outlier boundary corresponding to the current candidate document comprises:

[0021] obtaining a mean value and a standard deviation corresponding to each of the current field similarities;

[0022] obtaining the outlier boundary by using the mean value, the standard deviation and a preset standard score threshold.

[0023] In one of the embodiments, the obtaining of the document similarity between the document query information and the current candidate document according to at least one of the current field similarities comprises:

[0024] the average of each of the current field similarities is taken as the document similarity;

[0025] or

[0026] the median of each of the current field similarities is taken as the document similarity;

[0027] or

[0028] the average of each of the target field similarities is taken as the document similarity; the target field similarity is a preset number of current field similarities with the largest similarity among each of the current field similarities.

[0029] In one embodiment, the field information includes title information, body content information, and abstract information; the document query information used for retrieving the document is obtained, and a field similarity between the document query information and different field information of each candidate document is obtained, including:

[0030] a user query is obtained, and the user query is vectorized to obtain the document query information;

[0031] a first similarity between the document query information and the title information of each candidate document, a second similarity between the document query information and the body content information of each candidate document, and a third similarity between the document query information and the abstract information of each candidate document are obtained;

[0032] each of the first similarity, each of the second similarity, and each of the third similarity is normalized, and each of the normalized first similarity, each of the normalized second similarity, and each of the normalized third similarity is taken as a field similarity between the document query information and different field information of each candidate document.

[0033] In a second aspect, the present application provides a document retrieval device based on multi-field information and outlier detection, the device comprising:

[0034] a field similarity obtaining module, configured to obtain document query information used for retrieving a document, and obtain a field similarity between the document query information and different field information of each candidate document; the candidate document is a document stored in a preset knowledge base;

[0035] a first document similarity obtaining module, configured to, if there is at least one high outlier in a current field similarity between the document query information and any of field information of a current candidate document, obtain a document similarity between the document query information and the current candidate document according to each of the high outliers; the current candidate document is any one of the candidate documents;

[0036] a second document similarity obtaining module, configured to, if there is no high outlier in the current field similarity between the document query information and any field information of the current candidate document, obtain a document similarity between the document query information and the current candidate document according to at least one of the current field similarities;

[0037] a document retrieval result generating module, configured to obtain a document retrieval result corresponding to the document query information according to the document similarities between the document query information and each of the candidate documents.

[0038] In a third aspect, the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the document retrieval method based on multi-field information and outlier detection when executing the computer program.

[0039] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the steps of the method when executed by a processor.

[0040] In a fifth aspect, the present application provides a computer program product, comprising a computer program, and the computer program implements the steps of the method when executed by a processor.

[0041] The document retrieval method, device, computer device, readable storage medium and computer program product based on multi-field information and outlier detection, by obtaining document query information used for retrieving a document, and obtaining field similarities between the document query information and different field information of each candidate document, if there is at least one high outlier in the current field similarities between the document query information and any field information of a current candidate document, obtaining a document similarity between the document query information and the current candidate document according to each high outlier, the high outlier is an outlier greater than an outlier boundary, if there is no high outlier in the current field similarities between the document query information and any field information of the current candidate document, obtaining a document similarity between the document query information and the current candidate document according to at least one current field similarity, obtaining a document retrieval result corresponding to the document query information according to the document similarities between the document query information and each candidate document, the similarity scores of the multi-field vector retrieval results of each candidate document can be obtained, the similarity scores of different fields are normalized first, so as to convert the similarity scores with different dimensions from different fields into unified comparable field similarities, and then the field similarities are fused by outlier detection, so as to fuse the field similarities of each document in each field, and obtain a document similarity highly reflecting the relevance of the document query information and each candidate document, so that the document highly relevant to the user query can be accurately identified, and the relevance of the document retrieval result is improved, and the problem of low relevance between the document retrieval result and the user query in the traditional technology in the retrieval enhancement generation technology is solved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the description of the embodiments of the present application or the related art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other related drawings can be obtained by those skilled in the art without creative labor.

[0043] Figure 1 An application environment diagram of a document retrieval method based on multi-field information and outlier detection in an embodiment;

[0044] Figure 2 A flowchart of a document retrieval method based on multi-field information and outlier detection in an embodiment;

[0045] Figure 3 A flowchart of another document retrieval method based on multi-field information and outlier detection in an embodiment;

[0046] Figure 4A structural block diagram of a document retrieval device based on multi-field information and outlier detection in an embodiment;

[0047] Figure 5 An internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0049] With the development of large-scale pre-training language models and vectorization technology, retrieval-augmented generation (RAG) systems have become an important architecture for handling knowledge-intensive tasks. In the RAG system, the core step is to retrieve the most relevant document chunks from the knowledge base through vector search and provide them as context to the generation model. In order to improve retrieval accuracy, traditional techniques usually vectorize and index different parts of a single document (such as title, detailed text, abstract, etc.) separately. During retrieval, the system will search in the vector indexes of these different fields in parallel, and each field will return a document list sorted based on its own similarity score.

[0050] Although traditional techniques enrich the retrieval sources through multi-field retrieval, there are significant technical defects when merging the search results of different fields: 1. Score incomparability problem: the score distribution ranges of different field vector indexes naturally differ. For example, the cosine similarity scores of the title field may generally concentrate in the high score interval [0.8, 1.0], while the scores of the detailed (text) field may widely distribute in the interval [0.4, 0.9]. These scores of different fields lack comparability, and simple merging and adding will cause distortion of the ranking results. 2. Sub-optimal ranking result problem: due to the score incomparability, simple merging strategies (such as directly taking the highest score of each field) cannot accurately reflect the overall relevance of the document, which may cause a document with extremely high score in one field but irrelevant in other fields to rank higher than all documents with good relevance in all fields, thereby reducing the probability of retrieving high-quality context materials. 3. Strategy rigidity, lack of adaptive ability problem: existing methods usually use fixed strategies (such as hard-coded weights) for merging, which cannot dynamically adjust according to the score distribution characteristics of the returned results of each query, have poor flexibility, and are difficult to maintain optimal performance in various query scenarios.

[0051] Therefore, there is an urgent need for a normalization method that can effectively solve the problem of non-comparability of multi-field vector search result scores and adaptively perform result fusion to improve the relevance of document retrieval results to user queries, thereby improving the overall effect of the retrieval link of the RAG system.

[0052] Therefore, the document retrieval method based on multi-field information and outlier detection provided by the application can effectively solve the problem of non-comparability of scores between different fields through normalization and fusion of multi-field vector search results of the RAG system, and generate an optimal final ranking list through an adaptive fusion strategy.

[0053] The document retrieval method based on multi-field information and outlier detection provided by the embodiments of the application can be applied to an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The server 104 can obtain document query information used for retrieving a document, and obtain field similarities between the document query information and different field information of each candidate document. The candidate document is a document stored in a preset knowledge base. If there is at least one high outlier value in the current field similarities between the document query information and any field information of a current candidate document, the server 104 can obtain a document similarity between the document query information and the current candidate document according to each high outlier value. The current candidate document is any one of the candidate documents. The high outlier value is an outlier value greater than an outlier boundary. If there is no high outlier value in the current field similarities between the document query information and any field information of a current candidate document, the server 104 can obtain a document similarity between the document query information and the current candidate document according to at least one current field similarity. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers and Internet of Things devices. The server 104 obtains a document retrieval result corresponding to the document query information according to the document similarities between the document query information and each candidate document. The server 104 can be a physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0054] In an exemplary embodiment, as shown in Figure 2 A document retrieval method based on multi-field information and outlier detection is provided. The embodiment takes the method as an example applied to the server 104 in Figure 1 The method includes the following steps S202 to S208. Wherein:

[0055] In step S202, the document query information used for retrieving the document is obtained, and the field similarity between the document query information and the field information of each candidate document is obtained.

[0056] The candidate document can be a document stored in a preset knowledge base.

[0057] The document query information can be a user query processed to be used for retrieving the document.

[0058] The field information can be each part in the document that can be used for retrieval, such as a title, a content, an abstract, and the like.

[0059] In actual application, the server 104 can process the unstructured user query to obtain the document query information used for retrieving the document, and perform similarity search according to the field information in the preset knowledge base, so as to retrieve the field information of the candidate document related to the user query, obtain the similarity scores of the field information of each candidate document, and then normalize the similarity scores to obtain the field similarity between the document query information and the field information of each candidate document.

[0060] In step S204, if there is at least one high outlier value in the current field similarity between the document query information and any field information of the current candidate document, the document similarity between the document query information and the current candidate document is obtained according to the high outlier value.

[0061] The current candidate document can be any one of the candidate documents, and the current field similarity can be the field similarity of the field information of the current candidate document.

[0062] Optionally, the high outlier value can be an outlier value greater than an outlier boundary.

[0063] In actual application, the server 104 can perform outlier detection on the field similarity of each candidate document based on an outlier strategy to determine the document similarity of each candidate document.

[0064] For any current candidate document, if there is at least one high outlier value in the current field similarity between the document query information and any field information of the current candidate document, an outlier value greater than an outlier boundary can be selected as the document similarity between the document query information and the current candidate document.

[0065] In step S206, if there is no high outlier value in the current field similarity between the document query information and any field information of the current candidate document, the document similarity between the document query information and the current candidate document is obtained according to at least one current field similarity.

[0066] In a specific implementation, for the current candidate document, in the case that there is no high outlier in the current field similarity between the document query information and any field information of the current candidate document, the server 104 can obtain the document similarity between the document query information and the current candidate document according to the field similarity.

[0067] Step S208, obtaining the document retrieval result corresponding to the document query information according to the document similarity between the document query information and each candidate document.

[0068] In a specific implementation, after the outlier detection on all candidate documents, the server 104 can determine the document similarity between the document query information and each candidate document, and then sort the document similarity corresponding to all candidate documents to obtain the document retrieval result corresponding to the document query information.

[0069] The above document retrieval method based on multi-field information and outlier detection, by obtaining document query information used for retrieving documents, and obtaining field similarity between the document query information and different field information of each candidate document; if there is at least one high outlier in the current field similarity between the document query information and any field information of the current candidate document, obtaining the document similarity between the document query information and the current candidate document according to each high outlier; the high outlier is an outlier greater than the outlier boundary; if there is no high outlier in the current field similarity between the document query information and any field information of the current candidate document, obtaining the document similarity between the document query information and the current candidate document according to at least one current field similarity; obtaining the document retrieval result corresponding to the document query information according to the document similarity between the document query information and each candidate document, can realize obtaining the similarity score of the multi-field vector retrieval result of each candidate document, normalizing the similarity scores of different fields first, so as to convert the similarity scores with different dimensions from different fields into unified comparable field similarity, and then fusing the field similarity through outlier detection, so as to fuse the field similarity of each document in each field, obtain the document similarity highly reflecting the relevance between the document query information and each candidate document, so as to accurately identify the documents highly relevant to the user query, thereby improving the relevance of the document retrieval result, and solving the problem of low relevance between the document retrieval result and the user query in the traditional technology in the retrieval enhancement generation technology.

[0070] In an exemplary embodiment, after obtaining the field similarity between the document query information and different field information of each candidate document, it further includes: obtaining the outlier boundary corresponding to the current candidate document according to each current field similarity; determining the current field similarity greater than the outlier boundary as a high outlier.

[0071] Exemplarily, the same candidate document can hit in the search results of multiple field information, and therefore the field similarities of the candidate documents can be grouped according to the document identifiers of the candidate documents, so that for any current candidate document, the field similarities of the document query information and the field information in the current candidate document can be obtained.

[0072] In actual application, the server 104 can calculate the descriptive statistical index corresponding to each current candidate document according to the field similarities of the current candidate documents, calculate the outlier boundary corresponding to the current candidate document according to the descriptive statistical index and the preset adjustment parameter, and determine the current field similarity greater than the outlier boundary as a high outlier to identify the high outlier from the current field similarity. Exemplarily, the descriptive statistical index can be quartile, mean, standard deviation, median, etc.

[0073] The technical scheme of the embodiment can realize the adaptive fusion strategy based on outliers by obtaining the outlier boundary corresponding to the current candidate document and identifying the high outlier from the current field similarity, can adjust according to the specific distribution of the field similarity of the search result returned each time, instead of relying on the fixed parameters of different fields, so as to capture the extreme relevance of the document in a certain field, and thus the document highly relevant to the document query information corresponding to the user query can be more accurately identified.

[0074] In an exemplary embodiment, obtaining the outlier boundary corresponding to the current candidate document includes: obtaining the first quartile and the third quartile corresponding to each current field similarity; obtaining the interquartile range according to the difference between the third quartile and the first quartile; and obtaining the outlier boundary by using the third quartile, the interquartile range, and the preset adjustment parameter.

[0075] Identifying the high outlier from the current field similarity according to the current field similarity and the outlier boundary includes: determining the current field similarity greater than the outlier boundary as the high outlier.

[0076] In specific implementation, the server 104 can calculate the first quartile and the third quartile corresponding to each current field similarity according to the field similarity of each current candidate document. Then, the interquartile range is obtained according to the difference between the third quartile and the first quartile. Then, the outlier boundary is obtained by using the third quartile, the interquartile range, and the preset adjustment parameter. The current field similarity greater than the outlier boundary can be determined as the high outlier.​​​​​ The maximum value in the high outlier value is taken as the document similarity. Optionally, the preset adjustment parameter may be 1.5.

[0077] The technical solution of the embodiment determines the high outlier value of each field similarity of each candidate document by the quartile range method, takes the maximum value in the high outlier value as the document similarity, and thus effectively detects the outlier value in the field similarity data set of non-normal distribution, thereby providing a reliable detection means for accurately identifying the highly relevant search documents.

[0078] In an exemplary embodiment, obtaining the outlier value boundary corresponding to the current candidate document includes: obtaining the mean value and the standard deviation corresponding to each current field similarity; and obtaining the outlier value boundary by using the mean value, the standard deviation, and a preset standard score threshold.

[0079] In a specific implementation, the server 104 can calculate the mean value and the standard deviation corresponding to each current field similarity according to the field similarity of each current candidate document. and a preset adjustment parameter to obtain the outlier value boundary . The current field similarity corresponding to the outlier value boundary may be determined as a high outlier value, and thus the maximum value in the high outlier value is taken as the document similarity. Optionally, the preset adjustment parameter , i.e., the standard score threshold may be 2.5 or 3.

[0080] Exemplarily, the server 104 can also take the preset standard score threshold as the outlier value boundary; first obtain the standard score corresponding to each current field similarity ; and in a case where the standard score corresponding to the current field similarity is greater than the standard score threshold, the current field similarity is taken as a high outlier value. Optionally, the standard score threshold may be 2.5 or 3.

[0081] The technical solution of the embodiment determines the high outlier value of each field similarity of each candidate document by the standard score method, takes the maximum value in the high outlier value as the document similarity, and thus effectively detects the outlier value in the field similarity data set of normal distribution or approximate normal distribution, thereby providing a reliable detection means for accurately identifying the highly relevant search documents.

[0082] ​In an example embodiment, the document similarity between the document query information and the current candidate document according to the at least one current field similarity includes: taking an average of the current field similarities as the document similarity; or taking a median of the current field similarities as the document similarity; or taking an average of the target field similarities as the document similarity.

[0083] The target field similarity is a preset number of current field similarities with the largest similarity among the current field similarities, and is exemplarily a Top-K score.

[0084] In a specific implementation, in a case where there is no high outlier in the current field similarity between the document query information and any field information of the current candidate document, the server 104 can take any one of the average of the current field similarities, the median of the current field similarities, and the average of the preset number of current field similarities with the largest similarity among the current field similarities as the document similarity between the document query information and the current candidate document.

[0085] The technical solution of the embodiment can ensure that the highly relevant search document can still be accurately identified when there is no high outlier, i.e., the data distribution of the field similarity of the current candidate document is relatively concentrated, by taking the average of the current field similarities, the median of the current field similarities, and the average of the Top-K score as the document similarity between the document query information and the current candidate document.

[0086] In an example embodiment, the document query information for searching the document is obtained, and the field similarity between the document query information and different field information of each candidate document is obtained, including: obtaining a user query, and vectorizing the user query to obtain the document query information; obtaining a first similarity between the document query information and title information of each candidate document, a second similarity between the document query information and content information of each candidate document, and a third similarity between the document query information and abstract information of each candidate document; and performing normalization processing on each first similarity, each second similarity, and each third similarity, respectively, and taking the normalized each first similarity, each second similarity, and each third similarity as the field similarity between the document query information and different field information of each candidate document.

[0087] It should be noted that the field information includes the title information, the content information, and the abstract information, and can be information that establishes a vector index for each part of the field, such as the title, the detailed content of the text, and the abstract, of the document in the knowledge base after vectorization processing, including a title vector index, a content vector index, and an abstract vector index.

[0088] The document query information can be a user query that is processed by vectorization and can be used to retrieve documents.

[0089] In a specific implementation, the server 104 can obtain a user query, vectorize the user query, and obtain a user query that is processed by vectorization and has a vector index and can be used to retrieve documents, i.e., document query information. Then, according to the document query information and the field information of each candidate document, the document query information and the title information, the content information, and the abstract information of each candidate document are respectively subjected to similarity retrieval in the knowledge base, and the first similarity score, the second similarity score, and the third similarity score between the document query information and the title information, the content information, and the abstract information of each candidate document are respectively obtained, and a list of similarity scores is obtained. Then, for the list of similarity scores of each field information, the first similarity score, the second similarity score, and the third similarity score are respectively subjected to normalization processing, and the normalized first similarity score, the normalized second similarity score, and the normalized third similarity score are used as the field similarity between the document query information and the different field information of each candidate document.

[0090] Exemplarily, the normalization processing (also normalization, standardization) of the similarity scores can utilize the Min-Max normalization method (also linear normalization), i.e., normalized score = (original similarity score - minimum value of the current list of similarity scores) / (maximum value of the current list of similarity scores - minimum value of the current list of similarity scores).

[0091] In the embodiments of the present application, the normalization method is not limited, as long as it can achieve the method of unifying the field similarity of different fields to a comparable scale. For example, the Z-Score normalization method (standard score method, also known as zero-mean normalization method) can also be used, i.e., normalized score = (original similarity score - mean value of the current list of similarity scores) / standard deviation of the current list of similarity scores; and the decimal scaling normalization can also be used, i.e., normalized score = original similarity score / 10 , wherein may be an integer power of moving the original similarity score by decimal point positions.

[0092] The technical scheme of the embodiment can accurately retrieve the document related to the user query through vector search by vectorizing the user query and retrieving the field similarity of each candidate document in the knowledge base. Moreover, the similarity scores from different fields and different dimensions are converted into unified and comparable standard scores through normalization processing of the field similarity, ensuring the uniformity and comparability of the similarity of different fields, making the query of different characteristics maintain good adaptability and stability, and laying an accurate foundation for subsequent fusion of the similarity of different fields of each candidate document.

[0093] In an exemplary embodiment, the document retrieval result corresponding to the document query information is obtained according to the document similarity between the document query information and each candidate document, including: sorting the document similarity of each candidate document to generate a sorted document retrieval result.

[0094] Exemplarily, the document sorting result can be input into the generation component of the retrieval enhancement generation system.

[0095] The technical scheme of the embodiment can identify the document highly related to the document query information by sorting the document similarity of each candidate document after the “normalization and fusion” processing, optimize the document retrieval result corresponding to the document query information, thereby providing the generation component of the RAG (retrieval augmentation generation) system with more high-quality and more accurate basic document materials, improving the accuracy, relevance and reliability of the finally generated answer, and improving the user experience.

[0096] In specific embodiments, as shown in Figure 3 The document retrieval method based on multi-field information and outlier detection provided by the embodiment of the application includes:

[0097] S1: Multi-field parallel vector search: receiving a user query, vectorizing the user query, and performing similarity search in the title vector index, the detail vector index and the abstract vector index of the knowledge base respectively, each index returning an ordered list containing a document ID and its original similarity score.

[0098] S2: Score normalization processing: for the score list returned by each field, independent normalization processing is performed to convert it into a standard score conforming to a predetermined distribution. The minimum-maximum normalization method based on the query result is preferably adopted in the embodiment of the application, and the calculation formula is: normalized score = (original score - current list minimum value) / (current list maximum value - current list minimum value). This step uniformly maps the scores of different fields to the interval [0, 1], solving the core problem of incomparable scores.

[0099] S3: Adaptive fusion based on outlier detection: For the same document, if it appears in the search results of multiple fields, its final score is fused by the normalized scores of these fields. The key of the embodiment of the present application is to use the outlier-based strategy in fusion, specifically: a) For each document, collect its normalized scores on all hit fields; b) Use the interquartile range (IQR) method to detect high outliers in these scores. Calculate the first quartile (Q1) and the third quartile (Q3) of these scores, and define the outlier boundary as Q3 + k * IQR (k is an adjustable parameter, and the default value is 1.5); c) If there are high outliers, take the maximum value of all high outliers as the final score of the document; if not, fall back to take the average of all normalized scores. This strategy can preferentially capture the extreme relevance of the document in a certain field, while taking into account the case where multiple fields are relevant.

[0100] S4: Generating the final ranking list: Sort all documents in descending order according to the final scores calculated in step S3 to generate the final search result list, and output it to the generation component of the RAG system.

[0101] It should be understood that although each step in the flowchart involved in each embodiment as described above is displayed in sequence according to the arrow, these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowchart involved in each embodiment as described above can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0102] Based on the same inventive concept, the embodiments of the present application also provide a document retrieval device based on multi-field information and outlier detection for implementing the above-mentioned document retrieval method based on multi-field information and outlier detection. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more document retrieval device embodiments based on multi-field information and outlier detection provided below can refer to the limitations of the document retrieval method based on multi-field information and outlier detection in the above text, and will not be repeated here.

[0103] In an exemplary embodiment, as shown in FIG. 4, a document retrieval device based on multi-field information and outlier detection is provided, comprising:

[0104] The field similarity obtaining module 410 is configured to obtain document query information used for retrieving the document, and obtain field similarities between the document query information and different field information of each candidate document; the candidate document is a document stored in a preset knowledge base.

[0105] The first document similarity obtaining module 420 is configured to, if there is at least one high outlier value in the current field similarities between the document query information and any field information of a current candidate document, obtain a document similarity between the document query information and the current candidate document according to the high outlier values; the current candidate document is any one of the candidate documents.

[0106] The second document similarity obtaining module 430 is configured to, if there is no high outlier value in the current field similarities between the document query information and any field information of a current candidate document, obtain a document similarity between the document query information and the current candidate document according to at least one current field similarity.

[0107] The document retrieval result generating module 440 is configured to obtain a document retrieval result corresponding to the document query information according to the document similarities between the document query information and the candidate documents.

[0108] In one of the embodiments, the first document similarity obtaining module 420 is further configured to obtain an outlier value boundary corresponding to the current candidate document according to the current field similarities, and determine the current field similarity greater than the outlier value boundary as the high outlier value.

[0109] In one of the embodiments, the first document similarity obtaining module 420 is further configured to obtain a first quartile and a third quartile corresponding to each current field similarity, obtain a quartile range according to a difference between the third quartile and the first quartile, obtain the outlier value boundary by using the third quartile, the quartile range and a preset adjustment parameter, and identify the high outlier value from the current field similarities according to the current field similarities and the outlier value boundary, including determining the current field similarity greater than the outlier value boundary as the high outlier value.

[0110] In one of the embodiments, the first document similarity obtaining module 420 is further configured to obtain a mean value and a standard deviation corresponding to each current field similarity, and obtain the outlier value boundary by using the mean value, the standard deviation and a preset standard score threshold.

[0111] In one of the embodiments, the second document similarity obtaining module 430 is further configured to take the mean value of the current field similarities as the document similarity, or take the median of the current field similarities as the document similarity, or take the mean value of target field similarities as the document similarity; the target field similarity is a current field similarity with a maximum similarity in the current field similarities.

[0112] In one of the embodiments, the field similarity obtaining module 410 is further configured to obtain the user query, vectorize the user query to obtain document query information, obtain a first similarity between the document query information and title information of each candidate document, a second similarity between the document query information and content information of each candidate document, and a third similarity between the document query information and abstract information of each candidate document, and normalize each first similarity, each second similarity, and each third similarity, and take the normalized each first similarity, each second similarity, and each third similarity as the field similarity between the document query information and different field information of each candidate document.

[0113] The modules in the document retrieval device based on multi-field information and outlier detection described above can be realized by software, hardware, and combinations thereof, in whole or in part. The modules described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to the modules.

[0114] In one exemplary embodiment, a computer device is provided, which can be a server, and an internal structure diagram of the computer device can be as shown in FIG. 5. The computer device includes a processor, a memory, an input / output interface (I / O), and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The database of the computer device is configured to store document retrieval data. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement a document retrieval method based on multi-field information and outlier detection.

[0115] Those skilled in the art can understand that the structure shown in FIG. 5 is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.

[0116] In one embodiment, a computer device is provided, comprising a memory and a processor, the memory stores a computer program, and the processor implements the steps in each embodiment of the above-mentioned document retrieval method when executing the computer program.

[0117] In one embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps in each embodiment of the above-mentioned document retrieval method.

[0118] In one embodiment, a computer program product is provided, comprising a computer program, and the computer program is executed by a processor to implement the steps in each embodiment of the above-mentioned document retrieval method.

[0119] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0120] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magneto resistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence processor, etc., without being limited thereto.

[0121] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0122] The above-described embodiments are merely illustrative of several embodiments of the present application, and the description is relatively specific and detailed, but should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A document retrieval method based on multi-field information and outlier detection, characterized in that, The method includes: Obtain document query information for retrieving documents, and obtain the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base; If there is at least one high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained according to each of the high outlier values; the current candidate document is any one of the candidate documents; the high outlier value is an outlier value that is greater than the outlier value boundary; If there are no high outliers in the current field similarity between the document query information and any of the field information of the current candidate document, then the document similarity between the document query information and the current candidate document is obtained based on at least one of the current field similarities. Based on the document similarity between the document query information and each of the candidate documents, the document retrieval results corresponding to the document query information are obtained.

2. The method according to claim 1, characterized in that, After obtaining the field similarity between the document query information and the different field information of each candidate document, the method further includes: Based on the similarity of each of the current fields, obtain the outlier boundary corresponding to the current candidate document; The current field similarity that exceeds the outlier boundary is identified as a high outlier.

3. The method according to claim 2, characterized in that, The step of obtaining the outlier boundary corresponding to the current candidate document based on the similarity of each of the current fields includes: Obtain the first quartile and the third quartile corresponding to the similarity of each of the current fields; The interquartile range is obtained based on the difference between the third quartile and the first quartile. The outlier boundary is obtained by using the third quartile, the interquartile range, and a pre-set adjustment parameter.

4. The method according to claim 2, characterized in that, The step of obtaining the outlier boundary corresponding to the current candidate document based on the similarity of each of the current fields includes: Obtain the mean and standard deviation of the similarity of each of the current fields; The outlier boundary is obtained using the mean, the standard deviation, and a pre-set standard score threshold.

5. The method according to claim 1, characterized in that, The step of obtaining the document similarity between the document query information and the current candidate document based on at least one of the current field similarities includes: The average of the similarities of each of the current fields is taken as the document similarity. or The median of the similarity scores for each of the current fields is used as the document similarity score. or The average of the similarities of each target field is taken as the document similarity; the target field similarity is the top preset number of current field similarities with the highest similarity among all the current field similarities.

6. The method according to any one of claims 1 to 5, characterized in that, The field information includes title information, body content information, and summary information; obtaining document query information for retrieving documents, and obtaining the field similarity between the document query information and different field information of each candidate document, includes: Obtain the user query, vectorize the user query, and obtain the document query information; Obtain a first similarity between the document query information and the title information of each candidate document, a second similarity between the document query information and the main text information of each candidate document, and a third similarity between the document query information and the summary information of each candidate document; Each of the first similarity, each of the second similarity, and each of the third similarity are normalized, and the normalized first similarity, each of the second similarity, and each of the third similarity are used as the field similarity between the document query information and the different field information of each candidate document.

7. A document retrieval device based on multi-field information and outlier detection, characterized in that, The device includes: The field similarity acquisition module is used to acquire document query information for retrieving documents, and to acquire the field similarity between the document query information and different field information of each candidate document; the candidate documents are documents stored in a preset knowledge base; The first document similarity acquisition module is used to obtain the document similarity between the document query information and the current candidate document based on each of the high outlier values ​​if there is at least one high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document; the current candidate document is any one of the candidate documents. The second document similarity acquisition module is used to obtain the document similarity between the document query information and the current candidate document based on at least one of the current field similarities if there is no high outlier value in the current field similarity between the document query information and any of the field information of the current candidate document. The document retrieval result generation module is used to obtain the document retrieval result corresponding to the document query information based on the document similarity between the document query information and each of the candidate documents.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images