Information query method and related device

Abstract

The application discloses an information query method and related device, and relates to the field of information processing. In response to a query text input by a user, at least one document obtained is combined with the query text respectively to obtain at least one to-be-detected data pair. A pre-trained correlation detection model is called to process each to-be-detected data pair to obtain correlation information of each to-be-detected data pair, wherein the correlation information is information representing whether there is correlation between the document and the query text in the to-be-detected data pair. The to-be-detected data pair in which it is determined that the document and the query text are correlated is a target data pair, and a reply result corresponding to the query text is generated according to the document in the target data pair and output. According to whether the document and the query request are correlated, the application takes the correlation as a standard for selecting a reply, can effectively filter out irrelevant documents, generates a reply result according to relevant documents, improves the accuracy of the reply result, and achieves the purpose of meeting the user's query requirement.

Description

Technical Field

[0001] This application relates to the field of information processing technology, and in particular to an information retrieval method and related apparatus. Background Technology

[0002] Information retrieval refers to the process of finding information highly relevant to a query request from a large amount of raw information, and then returning query results that best satisfy the user's query intent based on this highly relevant raw information. The relevance between the raw information and the query request refers to the degree of correlation between the information corresponding to the query request and the raw information. Taking the application of information retrieval in the automotive field as an example, a user inputs fault information into a question-and-answer model for vehicle configuration. The model evaluates the relevance between the stored documents recording fault cases and the fault information, sorts them according to the degree of relevance between each document and the fault information, and generates a response result corresponding to the fault information based on the sorted document sequence. This allows users to quickly locate the cause of the vehicle fault and find solutions.

[0003] However, in practical applications, the accuracy of the query results obtained from information query requests is low, and the query results are often irrelevant to the query request. There is an urgent need for an information query method with higher accuracy. Summary of the Invention

[0004] In view of the above problems, this application provides an information retrieval method and related apparatus to improve the accuracy of information retrieval. The specific solution is as follows:

[0005] The first aspect of this application provides an information retrieval method, including:

[0006] In response to receiving query text input by the user, retrieve at least one document;

[0007] The at least one document is combined with the query text to obtain at least one data pair to be detected;

[0008] A pre-trained relevance detection model is invoked to process each of the data pairs to be detected, thereby obtaining relevance information for each data pair to be detected. The relevance information is information that characterizes whether there is a relevance between the document and the query text in the data pair to be detected. The relevance detection model is trained using data pair samples as feature values ​​and the relevance information corresponding to the data pair samples as target values. The data pair samples include at least one data pair containing a document sample and a query text sample.

[0009] The data pairs to be detected that satisfy the preset conditions are identified as target data pairs, where the preset conditions are that the document and the query text are related.

[0010] Based on the documents in the target data pair, generate and output the response results corresponding to the query text.

[0011] In one possible implementation, the process of pre-training the correlation detection model includes:

[0012] Obtain data pairs corresponding to multiple query text samples, and the relevance information of the data pairs. Each data pair includes one positive data pair and multiple negative data pairs. The document samples in the positive data pair are related to the query text samples, while the document samples in the negative data pairs are not related to the query text samples. The relevance information includes a first score indicating the presence of relevance and a second score indicating the absence of relevance.

[0013] Construct an initial correlation detection model, which includes at least an encoder and a linear classification layer;

[0014] Based on the data pair samples corresponding to each query text sample and the relevance information of the corresponding annotations of the data pair samples, the initial relevance detection model is trained to obtain the relevance detection model.

[0015] In one possible implementation, training the initial relevance detection model based on the data pair samples corresponding to each query text sample and the relevance information of the annotations corresponding to the data pair samples to obtain the relevance detection model includes:

[0016] Based on the data pair samples corresponding to each query text sample, the initial relevance detection model is trained to obtain the training result corresponding to each data pair sample;

[0017] Based on the training results and correlation information corresponding to each data pair sample, the loss value of the preset loss function is determined;

[0018] Determine whether the loss value reaches a preset threshold;

[0019] If the loss value does not reach the preset threshold, the initial relevance detection model is trained based on the data pair samples corresponding to each query text sample to obtain the training result corresponding to each data pair sample until the loss value reaches the preset threshold.

[0020] When the loss value reaches the preset threshold, the initial correlation detection model is determined to be trained successfully, and the correlation detection model is obtained.

[0021] In one possible implementation, the process by which the correlation detection model processes the data pair to be detected includes:

[0022] The encoder of the relevance detection model is invoked to encode the data pair to be detected, thereby obtaining the word vectors corresponding to the data to be detected;

[0023] The linear classification layer of the relevance detection model is invoked to classify the word vectors according to the degree of relevance between the document represented by the word vectors and the query text, and the classification result is obtained. The classification result is used to characterize whether there is a relevance between the document corresponding to the word vectors and the query text.

[0024] Based on the classification results, the relevance information corresponding to the word vectors is output.

[0025] In one possible implementation, generating and outputting a response result corresponding to the query text based on the document in the target data pair includes:

[0026] Based on the relevance information of each target data pair, the documents in the target data pair are sorted to obtain a document sequence sorted according to their relevance to the query text;

[0027] The generative large model is invoked to learn the document content in the document sequence, generate and output the response results corresponding to the query text.

[0028] In one possible implementation, prior to the invocation of the generative large model, the following is also included:

[0029] Identify whether the document in the target data pair meets the respondable conditions, the respondable conditions including at least: the timeliness of the document meets the preset timeliness requirements, and the source of the document is a preset data source;

[0030] If there are documents in the target data pair that do not meet the respondability criteria, the documents that do not meet the respondability criteria are filtered out from the document sequence to obtain a document sequence for replying.

[0031] A second aspect of this application provides an information query device, comprising:

[0032] The document retrieval unit is used to retrieve at least one document in response to receiving query text input by the user;

[0033] A data combination unit is used to combine the at least one document with the query text to obtain at least one data pair to be detected;

[0034] The relevance detection unit is used to call a pre-trained relevance detection model to process each of the data pairs to be detected, and obtain the relevance information of each data pair to be detected. The relevance information is information that characterizes whether there is a relevance between the document and the query text in the data pair to be detected. The relevance detection model is trained with data pair samples as feature values ​​and the relevance information corresponding to the data pair samples as target values. The data pair samples include at least one data pair containing a document sample and a query text sample.

[0035] A data pair confirmation unit is used to determine that the data pairs to be detected that satisfy a preset condition are target data pairs, wherein the preset condition is that the document and the query text are related.

[0036] The response generation unit is used to generate and output a response result corresponding to the query text based on the document in the target data pair.

[0037] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the information query method described in the first aspect or any implementation thereof.

[0038] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:

[0039] The memory is used to store computer programs;

[0040] The processor is used to execute the computer program so that the electronic device can implement the information query method of the first aspect or any implementation thereof.

[0041] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the information query method described in the first aspect or any implementation thereof.

[0042] By employing the aforementioned technical solution, the information query method provided in this application uses a combination of a document and query text as a data pair to be detected for relevance testing. It independently detects each data pair using a relevance detection model, obtaining relevance information to determine whether the document and query text are related or not. This reduces the impact of relative relationships between documents on the detection results and improves the accuracy of relevance detection. Furthermore, based on the relevance information of each data pair, documents that are relevant to the query text are used as the basis for the response, generating and outputting response results, thus improving the accuracy of the response results. Therefore, this application uses the relevance between a document and the query request as the criterion for selecting responses, effectively filtering out irrelevant documents and generating response results based on relevant documents, thereby improving the accuracy of the response results and achieving the goal of meeting user query needs. Attached Figure Description

[0043] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0044] Figure 1 A flowchart illustrating an information retrieval method provided in this application;

[0045] Figure 2 Example diagram of the training correlation detection model provided in this application;

[0046] Figure 3 This application provides a system architecture diagram for implementing an information retrieval method;

[0047] Figure 4 A schematic diagram of the structure of an information query device provided in this application;

[0048] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0049] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0050] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0051] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0052] This application can be applied to the field of information processing. The following section will use a large model as an example to introduce several application scenarios that can be implemented in products.

[0053] First, we introduce one possible application scenario of this application. The question-answering model combines RAG (Retrieval-Augmented Generation) technology to retrieve the most relevant text data from a large text database for the user's input question text or query text, and then uses this text data to help generate the response text for the query.

[0054] In existing implementations, retrieval models can be built using pairwise or listwise methods. They are then trained using a contrastive learning loss function to learn how to rank the probabilities of positive and negative data. This allows the retrieval model to output a score for each document (doc) given the input query, and then rank the documents based on the scores.

[0055] However, considering that relevance ranking applies to all documents in the database, and the retrieval model's database stores a large number of documents, the above information query method would place a significant load on the retrieval model during the response generation process. Therefore, based on the ranked document sequence, documents with low relevance can be filtered out, and responses can be generated only based on the sequence of highly relevant documents.

[0056] However, because the score obtained by this retrieval model is a relative score, it cannot establish a clear threshold or rule to distinguish between relevant and irrelevant data. For example, a question-answering model needs to output a response based on a user's query. Currently, the database stores two documents. After receiving a query, the relevance of each document to the query is identified. Suppose that document A and document B have a relevance of 10% and 8% to the query, respectively, both of which are low in relevance and could even be considered irrelevant. However, to find the basis for generating a response, the question-answering model will compare document A and document B. Since document A has a higher relevance than document B, it will be considered more relevant by the question-answering model. Therefore, document A receives an evaluation result representing high relevance, i.e., a high score, while document B receives a low score representing weak relevance or irrelevance. The low-scoring document is filtered out, and the response is generated based on document A. This results in the final response being generated based on document A, which is irrelevant to the query, and fails to meet the user's query intent.

[0057] Therefore, existing information retrieval methods only consider the relative relationship between documents when evaluating, scoring, and ranking documents for relevance, ignoring the degree of relevance between documents and the query request. This results in the selection of documents as the basis for the response being inaccurate, thus affecting the accuracy of the obtained response results.

[0058] To address the aforementioned problems, embodiments of this application provide an information query method. The information query method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0059] Reference Figure 1 , Figure 1 This is a flowchart illustrating an information query method provided in an embodiment of this application, as shown below. Figure 1 As shown in the embodiment of this application, an information query method may include steps S110 to S150, which are described in detail below.

[0060] Step S110: In response to receiving the query text input by the user, at least one document is obtained.

[0061] In the above scenario example, the information query method can be applied to the question answering model, so users can input query text into the question answering model through voice input, text input, etc. Optionally, the information query method can also be applied to knowledge question answering systems, document generation and summarization, search engines, AI assistants and other servers to meet users' query needs.

[0062] After receiving the query text input by the user, at least one document is obtained to serve as the basis for generating a response. In this embodiment, there are multiple ways to obtain the document. In one possible implementation, all pre-stored documents are retrieved from the knowledge base or database of the question-answering model as the document basis for subsequent relevance queries.

[0063] Optionally, the similarity between each document in the database and the query text is identified, and at least one document with a similarity higher than a preset similarity threshold is obtained. It is understandable that if all document data in the database is used as the object of subsequent relevance detection and ranking, most of which are irrelevant to the query text, it would unnecessarily increase the load on the question-answering model server and reduce query efficiency. Therefore, this embodiment of the application performs a coarse-grained initial screening of the documents in the database, and uses the documents that may be related to the query text obtained from the initial screening as the objects of subsequent relevance detection.

[0064] The initial screening process utilizes natural language processing (NLP) technology to convert the query text and document into vector representations and calculate the similarity between the two vectors. The relationship between this similarity and a preset similarity threshold is used to determine if there is a correlation between the document and the query text. If the similarity is greater than the preset threshold, the document is considered to be related to the query text and is included in the subsequent relevance detection process. Based on this method, each document in the database is screened to obtain at least one document for relevance detection in subsequent steps S120-S130.

[0065] Optionally, the initial screening can also be achieved by identifying whether the content of each document in the library contains key fields from the query text. If a document contains key fields from the query text, it is used as the object for subsequent relevance detection. This application embodiment does not exclusively limit the coarse-grained initial screening process.

[0066] Step S120: Combine at least one document with the query text to obtain at least one data pair to be detected.

[0067] Before performing relevance detection, this application embodiment obtains multiple "query text-document" pairs of data to be detected by combining each document with the query text. This allows for independent relevance detection of documents and query text, avoiding the influence of the relative relationship between documents on the detection results.

[0068] Step S130: Call the pre-trained correlation detection model to process each pair of data to be detected and obtain the correlation information of each pair of data to be detected.

[0069] Relevance information is information that characterizes whether there is a correlation between the document and the query text in the data pair to be detected. The relevance detection model is trained using data pair samples as feature values ​​and the relevance information corresponding to the data pair samples as target values. Data pair samples include: at least one data pair containing a document sample and a query text sample.

[0070] In this embodiment, the relevance detection model is trained on the samples based on the data, so that the trained relevance detection model can output relevance information that characterizes whether there is a relevance between the documents in the data pair and the query text, based on the input data pair. This relevance information is used to qualitatively determine the relevance between the documents in the data pair and the query text.

[0071] Optionally, the data pair sample contains multiple data pairs. Based on whether there is a correlation between the document sample and the query text sample in the data pair, correlation information is labeled for each data pair. This is used to train a correlation detection model, so that the correlation detection model can output correlation information representing whether there is a correlation or not between the document and the query text in the data to be detected, according to the degree of correlation between the two.

[0072] Optionally, the data sample contains multiple data pairs, each with a different degree of relevance between the document and the query text. Assuming the question-answering model requires that only documents with a relevance of 90% or higher be considered relevant, then relevant information is labeled according to the degree of relevance between the document and the query text in each data pair. Based on this training, the relevance detection model performs a binary classification-like process during relevance detection of the data to be tested. For example, if the degree of relevance between the document and the query text in the data pair is not as expected, such as only 60%, then the relevance detection model outputs relevance information indicating that there is no relevance between the document and the query text in the data pair.

[0073] Based on this, the relevance detection model, by analyzing the relevance information output for each pair of data to be detected, can directly distinguish whether each document is related to the query text. Compared to existing technologies that determine relevance based on the relative relationship between documents, this embodiment returns to the relationship between documents and the query text, detecting each document independently and reducing the influence between documents. Furthermore, in an optional implementation, the relevance detection model can also perform a preliminary relevance screening based on the relationship between the degree of relevance between the document and the query text and the expected degree, and then further qualitatively determine whether a relevance exists, improving the flexibility and accuracy of relevance detection.

[0074] Step S140: Determine the data pairs to be detected that meet the preset conditions for relevance information as target data pairs. The preset conditions are that the document and the query text are related.

[0075] Step S150: Based on the documents in the target data pair, generate and output the response results corresponding to the query text.

[0076] Based on step S130, the system can clearly distinguish between documents that are relevant to the query text and those that are not, without being affected by the relative relationships between documents, according to the relevance information of each pair of data to be detected. Furthermore, documents irrelevant to the query text are filtered out, and only documents relevant to the query text are used as the basis for the response, generating a response result for the query text and outputting it to meet the user's query needs.

[0077] Furthermore, in the question-answering model scenario described above, another situation exists: Document A and Document B have a relevance of 90% and 80% to the query, respectively, both being highly relevant. However, to obtain highly relevant documents, the answering model compares Document A and Document B when ranking relevance. Since Document A is more relevant than Document B, Document A receives a high score indicating high relevance, while Document B receives a low score indicating weak or no relevance, and the answer is generated based on Document A. This results in the filtering out of Document B, which is also highly relevant, making the answering model's evidence insufficient and the resulting answer inaccurate.

[0078] However, since this application's embodiments use the relevance information of each pair of data to be detected as the filtering basis, where the relevance information characterizes the qualitative result of whether there is a relevance between the documents in the data pair and the query text, rather than filtering by comparing the degree of relationship between the data pairs, this application can use all documents qualitatively deemed to be relevant as the basis for the response, generating a more comprehensive and complete response.

[0079] In summary, the information query method provided in this application uses a combination of a document and query text as a pair of data to be detected for relevance testing. It independently detects each pair of data using a relevance detection model, obtaining relevance information to determine whether the document and query text are related or not. This reduces the impact of relative relationships between documents on the detection results and improves the accuracy of relevance detection. Furthermore, based on the relevance information of each pair of data to be detected, documents that are relevant to the query text are selected as the basis for the response, generating and outputting response results, thus improving the accuracy of the response results. Therefore, this application uses the relevance between a document and the query request as the criterion for selecting responses, effectively filtering out irrelevant documents and generating response results based on relevant documents, thereby improving the accuracy of the response results and achieving the goal of meeting user query needs.

[0080] Next, other possible implementations of the information query method provided in this application will be described in detail through the following embodiments.

[0081] In one possible implementation, the process of pre-training the relevance detection model includes: acquiring data pair samples corresponding to multiple query text samples, and relevance information of the corresponding annotations of the data pair samples. The data pair samples include: one positive data pair and multiple negative data pairs. The document samples in the positive data pair are related to the query text samples, and the document samples in the negative data pairs are not related to the query text samples. The relevance information includes: a first score representing the existence of relevance and a second score representing the absence of relevance; constructing an initial relevance detection model, which includes at least: an encoder and a linear classification layer; and training the initial relevance detection model based on the data pair samples corresponding to each query text sample and the relevance information of the corresponding annotations of the data pair samples to obtain the relevance detection model.

[0082] First, a dataset for training the model is established. Based on the domain in which the information query method is applied, relevant document data and question-and-answer data are collected. For example, in the automotive field, rich document data and question-and-answer data related to automobiles and general questions are collected. Query text (`query`) is extracted from the question-and-answer data, and each query is associated with a document (`doc`) to obtain a data pair corresponding to each query. Further, manual annotation or a detailed text prompt is used to guide GPT4 in labeling the relevance of each data pair. For example, if the `doc` and `query` in a data pair do not meet the relevance criteria specified in the prompt, the relevant information in that data pair can be labeled as irrelevant. Similarly, if the `doc` and `query` in a data pair meet the relevance criteria specified in the prompt, the relevant information in that data pair can be labeled as relevant.

[0083] Optionally, if this application embodiment aims to obtain a score that characterizes relevance for each document, as in the prior art, then for data pairs that are related, a first score representing relevance, such as 1, can be assigned; for data pairs that are not related, a second score representing non-relevance, such as 0, can be assigned. Based on this, the relevance information of the data pairs and corresponding labeled data pairs is used to train the relevance detection model, so that the trained relevance detection model achieves the following effect: for input data pairs that are related to the target data, a first score is output; for input data pairs that are not related to the target data, a second score is output, clearly distinguishing whether the document is related to the query text from the output score.

[0084] Furthermore, an initial correlation detection model is constructed to evaluate the effectiveness of existing open-source models. The lightweight, fast-response, and stable-ranking model BGE-Reranker-base can be selected as the base model to construct an initial correlation detection model that includes at least an encoder and a linear classification layer.

[0085] Based on the dataset established above, the initial correlation detection model is trained. Optionally, the dataset is divided into a training set and a validation set. The initial correlation detection model is trained on samples using multiple data points in the training set. During training, the initial correlation detection model processes the data in the validation set to obtain the output results corresponding to the validation set. When the accuracy between the output results and the correlation information between the data and the samples reaches a certain threshold, the initial correlation detection model is considered to have completed training.

[0086] In another possible implementation, an initial relevance detection model is trained based on the data pairs corresponding to each query text sample and the relevance information of the corresponding annotations, to obtain a relevance detection model. This includes: training the initial relevance detection model based on the data pairs corresponding to each query text sample to obtain the training result for each data pair; determining the loss value of a preset loss function based on the training result and relevance information for each data pair; determining whether the loss value reaches a preset threshold; if the loss value does not reach the preset threshold, training the initial relevance detection model based on the data pairs corresponding to each query text sample to obtain the training result for each data pair until the loss value reaches the preset threshold; if the loss value reaches the preset threshold, determining that the initial relevance detection model training is complete, thus obtaining the relevance detection model.

[0087] The preset loss function can be the binary cross-entropy loss function (BCE loss), a type of cross-entropy loss for binary classification. It measures the difference between the probability distribution predicted by the model and the actual label in a binary classification problem. By minimizing this difference, the model parameters are optimized, making the model's output closer to the actual label and improving the model's classification performance. In this embodiment, BCE loss is introduced during the training of the relevance detection model. The purpose is to encourage the model to output a score closer to the first score for positive data pairs with relevance and a score closer to the second score for negative data pairs without relevance. If the first score is 1 and the second score is 2, the relevance detection model uses the relevance between the document and the query text in the data pair to determine the data's comparability, facilitating the setting of preset conditions. For example, data pairs with a relevance score less than 2 are considered to have a relevance between the document and the query text.

[0088] Reference Figure 2 The example diagram provided in this application illustrates the training process of the correlation detection model. Figure 2 Taking the training process of a data pair corresponding to a query text as an example, the query corresponds to one positive example document pos1 and three negative example documents neg1, neg2, and neg3. The query is combined with each document to form a query-doc data pair.

[0089] Each data pair is input into the relevance detection model in the format "[CLS]query[SEP][SEP]doc[SEP]". Specifically, the relevance detection model's processing of the input data pairs / data pairs to be detected, both during training and after training, can include: calling the encoder of the relevance detection model to encode the data pairs to be detected, obtaining word vectors corresponding to the data pairs; calling the linear classification layer of the relevance detection model to classify the word vectors according to the degree of relevance between the documents represented by the word vectors and the query text, obtaining classification results, which are used to characterize whether there is a relevance between the documents corresponding to the word vectors and the query text; and outputting the relevance information corresponding to the word vectors based on the classification results.

[0090] Optional, Figure 2 In this example, the relevance detection model uses a cross-encoder. The document and query are concatenated to form a data pair, which is then input into the cross-encoder for encoding, resulting in word vector embeddings that represent the data features of the query-document data pair. Further, these word vector embeddings are input into a linear classification layer. The linear classification layer classifies the data pairs corresponding to the word vectors based on the relevance relationship between the document and the query text reflected in the word vector embeddings. It determines whether there is a relevance between the document and the query text. If a relevance exists, the linear classification layer scores the data pairs, with the score approaching the first score; if no relevance exists, the linear classification layer scores the data pairs, with the score approaching the second score.

[0091] Based on the above process, positive and negative data pairs are scored separately to obtain the relevance information of each data pair, i.e., the relevance score. Figure 2 The positive and negative data pairs are respectively represented by pos1-score, neg1-score, neg2-score, and neg3-score, which are used as the training result of one data pair. Referring to the BCEloss formula in equation (1), the optimization of the correlation detection model by BCEloss is explained.

[0092] BCEloss=-[(y*log(p))+((1-y)*log(1-p))]........................ (1)

[0093] Where y is the relevance information of the input data to the label, i.e., the first score or the second score; p is the probability value obtained by transforming the score output by the relevance detection model through the Sigmoid function.

[0094] If the first score is 1 and the second score is 0, when y = 1, and the input data pair is a positive example, the loss value BCEloss is -log(p); when y = 0, and the input data pair is a positive example, the loss value BCEloss is -log(1-p). The goal of training the correlation detection model is to minimize the loss value BCEloss. When y = 1, the loss value BCEloss can only reach its minimum value of 0 if the model output p = 1, and when y = 0, the loss value BCEloss can only reach its minimum value of 0 if the model output p = 0.

[0095] Based on this, the loss value BCEloss is calculated according to each training result and compared with a threshold. If the threshold is set to 0, it can be determined whether the training result is the same as the labeled relative information. If the loss value BCEloss is not equal to 0, it proves that the training result is not the same as the labeled relative information. The model parameters of the relevance detection model are adjusted, and training continues until the loss value BCEloss equals 0, which proves that the training result is the same as the labeled relative information. In this embodiment, BCEloss is used for model training to encourage the scores of positive data pairs to approach the first score of 1 and the scores of negative data pairs to approach the second score of 0.

[0096] In another possible implementation, the preset loss function also includes a contrastive learning loss function. That is, the correlation detection model is optimized and trained using both BCE loss and the contrastive learning loss function (InfoNCE: Contrastive Estimation of Neural Entropy, InfoNCE loss). The optimization training of the correlation detection model by InfoNCE loss is explained with reference to the InfoNCE loss formula in equation (2) below.

[0097]

[0098] Where L_InfoNCE represents the InfoNCE loss value, q represents the encoding vector of the query text sample, k+ represents the encoding vector of the document in the positive data pair that is related to the query text sample, k_n represents the encoding vector of the document in the nth negative data pair that is not related to the query text sample, N is the total number of negative data pairs; exp(a,b) represents the relevance score of the relevance test model output containing the data pair a and b.

[0099] In this embodiment, in the training results corresponding to a data pair sample, if the relevance score of the negative data pair is equal to the second score, and the relevance score of the positive data pair is equal to the first score, then L_InfoNCE = 0. Otherwise, L_InfoNCE is greater than 0. During the training process, after each training iteration, based on the relevance scores of all data pairs in the training results, the loss value L_InfoNCE is calculated, and it is determined whether it reaches a preset threshold. If the preset threshold is set to 0, and if it is not reached, the model parameters of the relevance detection model are further optimized so that the loss value corresponding to the next training result meets the condition. Based on this, by using one positive data pair and multiple negative data pairs in the data pair sample, the relevance score of the positive data pair is encouraged to be much higher than that of the negative data pairs. This clearly distinguishes between data pairs that are related and those that are not, based on their corresponding relevance scores, facilitating the subsequent setting of thresholds for filtering irrelevant documents.

[0100] Referring to the following formula (3), the correlation detection model is trained by combining BCE loss and InfoNCE loss, and the training ends when the loss value Loss reaches the preset value.

[0101] Loss=w*BCEloss+(1-w)*L_InfoNCE ............................. (3)

[0102] Where w is the weight parameter assigned to BCEloss.

[0103] The training process described above is used to train the relevance detection model. BCE loss and InfoNCE loss are employed to encourage the scores of positive data pairs to approach the first score and the scores of negative data pairs to approach the second score. At the same time, the scores of positive data pairs are encouraged to be much higher than the scores of negative data pairs. This makes the relevance detection model comparable in relevance scores between different queries, which facilitates the setting of thresholds in the filtering conditions.

[0104] In one possible implementation, based on the documents in the target data pair, a response result corresponding to the query text is generated and output, including: sorting the documents in the target data pair according to the relevance information of each target data pair to obtain a document sequence sorted according to the degree of relevance to the query text; calling a generative large model to learn the document content in the document sequence, generating a response result corresponding to the query text and outputting it.

[0105] The documents in the target data pairs are reordered to more accurately prioritize documents that answer the query text. Ideally, all documents in a target data pair should be relevant to the query text, resulting in a top-ranking relevance score. However, considering that relevance detection models cannot be trained to an ideal state, any relevance score close to the top score can be considered relevant, leading to inconsistent relevance scores for target data pairs. Given these inconsistencies, target data pairs closer to the top score indicate a stronger relevance between their documents and the query text. Therefore, based on the relevance score of each target data pair, the documents are sorted from highest to lowest, resulting in a document sequence. This document sequence is then input into a generative model, allowing it to comprehensively generate accurate responses based on document relevance.

[0106] In another possible implementation, before calling the generative large model, the following steps are also included: identifying whether the documents in the target data pair meet the respondability criteria, which at least include: the timeliness of the document meets the preset timeliness requirements, and the source of the document is a preset data source; if there are documents in the target data pair that do not meet the respondability criteria, the documents that do not meet the respondability criteria are filtered out from the document sequence to obtain the document sequence for replying.

[0107] This application takes into account that users may have requirements for the timeliness and authority of the query results. Therefore, after detecting the relevance between the document and the query text, it can further detect whether the relevant documents meet additional requirements such as timeliness and authority, so that the obtained response can better meet the user's query needs.

[0108] Optionally, the timeliness requirement can be determined by identifying the document's update or storage time. For example, if a user requires query results to reference data from 2025, and the relevant document was updated in 2024, it does not meet the timeliness requirement. Authority can also be determined by identifying the document's data source. For instance, if a user requires responses based solely on authoritative academic journals, a pre-defined list of academic journals can be used as the default data source. If the relevant document's data source is a social media page, which is not part of the default data source, then that relevant document does not meet the authority requirement. It is understandable that other possible query requirements exist, which will not be listed here. The purpose of this example is to illustrate that after relevance testing, in addition to determining the documents to be used as the basis for the response based on relevance, other factors can be considered to further filter the relevant documents, making the documents used as the basis for the response more in line with the user's needs and improving the accuracy of the response results.

[0109] Reference Figure 3This application provides a system architecture diagram for implementing an information query method, which serves as an example to illustrate the practical application of the information query method.

[0110] First, an offline relevance detection model is trained to detect data pairs of documents and query texts. Query texts (queries and documents) are collected from the online logs of the question-answering model, and each query is combined with each document to obtain multiple query-document data pairs for each query. Relevance annotations are performed on each query-document data pair using manual annotation or GPT4 annotation. Based on the relevance information of the query-document data pairs and their corresponding annotations, a training dataset is generated. This dataset is then used to train the pre-built initial relevance detection model, and the trained model is self-tested to check its training completion. The specific training process is described above and will not be repeated here. Furthermore, the trained relevance detection model is applied to the fine-ranking module of the online processing.

[0111] In the online application process, the user first enters a query text, triggering the online query process. Further, through data retrieval, a coarse-grained search is performed from a knowledge base or database to retrieve at least one document related to the query, and this document is then coarsely ranked. Next, the fine-ranking module uses a trained relevance detection model to perform relevance detection on the documents obtained from the coarse ranking. Specifically, each document is combined with the query to obtain at least one data pair to be detected. This data pair is then input into the relevance detection model for processing. Based on the relevance information or relevance score of each data pair, documents relevance to the query are identified, and these relevance documents are ranked to obtain the fine-ranked sequence.

[0112] Furthermore, the multi-factor model, incorporating features such as timeliness and authority, filters relevant documents in the refined ranking sequence again. The responsiveness model is responsible for determining whether the documents sent to the LLM are of high quality, timely, and capable of answering the query. Based on this model, each relevant document in the refined ranking sequence is filtered again. Finally, the remaining relevant documents serve as the basis for responses and are sent to the LLM, which then combines the content of these relevant documents to generate the corresponding response to the query.

[0113] In summary, the embodiments of this application utilize GPT4 to acquire training data more quickly and effectively. Furthermore, during training, BCE loss and InfoNCE loss are employed to make the relevance detection results of relevant and irrelevant data pairs comparable. This reduces the impact of relative results in document-to-document comparisons, improving the accuracy of relevance detection. On the other hand, it effectively filters out low-quality, irrelevant documents, which, as a crucial part of RAG (Related Article Analysis), effectively enhances the authenticity and relevance of end-to-end responses.

[0114] The above describes an information query method provided by an embodiment of this application. The following describes the apparatus for performing the above information query method.

[0115] Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of an information query device provided in an embodiment of this application. Figure 4 As shown, the information query device includes:

[0116] The document acquisition unit 100 is used to acquire at least one document in response to receiving query text input by the user;

[0117] The data combination unit 200 is used to combine the at least one document with the query text to obtain at least one data pair to be detected.

[0118] The relevance detection unit 300 is used to call a pre-trained relevance detection model to process each of the data pairs to be detected to obtain relevance information for each data pair to be detected. The relevance information is information that characterizes whether there is a relevance between the document and the query text in the data pair to be detected. The relevance detection model is trained using data pair samples as feature values ​​and the relevance information labeled with the data pair samples as target values. The data pair samples include at least one data pair containing a document sample and a query text sample.

[0119] Data pair confirmation unit 400 is used to determine the data pair to be detected that meets the preset condition of the correlation information as the target data pair, wherein the preset condition is that the document and the query text are related.

[0120] The response generation unit 500 is used to generate and output a response result corresponding to the query text based on the document in the target data pair.

[0121] In summary, this application's embodiments use a combination of a document and query text as a data pair to be detected for relevance testing. By independently detecting each data pair using a relevance detection model, relevance information can be obtained regarding the existence of a correlation between the document and query text within the data pair. This determines whether the relationship between the document and query text is relevant or irrelevant, reducing the impact of relative relationships between documents on the detection results and improving the accuracy of relevance detection. Furthermore, based on the relevance information of each data pair, documents that are relevant to the query text are selected as the basis for the response, generating and outputting response results, thus improving the accuracy of the response results. Therefore, this application uses the relevance between a document and the query request as the criterion for selecting responses, effectively filtering out irrelevant documents and generating response results based on relevant documents, thereby improving the accuracy of the response results and achieving the goal of meeting user query needs.

[0122] In one possible implementation, the functional unit for pre-training the correlation detection model includes:

[0123] A data sample acquisition unit is used to acquire data pair samples corresponding to multiple query text samples, and the relevance information of the data pair samples. The data pair samples include: one positive data pair and multiple negative data pairs. The document samples in the positive data pair are related to the query text samples, and the document samples in the negative data pairs are not related to the query text samples. The relevance information includes: a first score indicating the existence of relevance and a second score indicating the absence of relevance.

[0124] A model building unit is used to build an initial correlation detection model, which includes at least an encoder and a linear classification layer.

[0125] The model training unit is used to train the initial relevance detection model based on the data pair samples corresponding to each query text sample and the relevance information of the labels corresponding to the data pair samples, so as to obtain the relevance detection model.

[0126] In one possible implementation, the model training unit includes:

[0127] The sample training subunit is used to train the initial relevance detection model based on the data pair sample corresponding to each query text sample, and obtain the training result corresponding to each data pair sample;

[0128] The loss determination subunit is used to determine the loss value of a preset loss function based on the training results and the correlation information corresponding to each data pair sample.

[0129] The threshold determination subunit is used to determine whether the loss value reaches a preset threshold.

[0130] The loop execution subunit is used to train the initial relevance detection model based on the data pair samples corresponding to each query text sample, and obtain the training result corresponding to each data pair sample, until the loss value reaches the preset threshold, provided that the loss value has not reached the preset threshold.

[0131] The training completion sub-unit is used to determine that the initial correlation detection model has been trained when the loss value reaches the preset threshold, thereby obtaining the correlation detection model.

[0132] In one possible implementation, the process by which the correlation detection model processes the data pair to be detected includes:

[0133] The encoder of the relevance detection model is invoked to encode the data pair to be detected, thereby obtaining the word vectors corresponding to the data to be detected;

[0134] The linear classification layer of the relevance detection model is invoked to classify the word vectors according to the degree of relevance between the document represented by the word vectors and the query text, and the classification result is obtained. The classification result is used to characterize whether there is a relevance between the document corresponding to the word vectors and the query text.

[0135] Based on the classification results, the relevance information corresponding to the word vectors is output.

[0136] In one possible implementation, the response generation unit 500 includes:

[0137] The document sorting subunit is used to sort the documents in the target data pair according to the relevance information of each target data pair, so as to obtain a document sequence sorted according to the degree of relevance to the query text;

[0138] The response result generation subunit is used to call the generative large model, learn the document content in the document sequence, generate and output the response result corresponding to the query text.

[0139] One possible implementation also includes:

[0140] The condition recognition subunit is used to identify whether the document in the target data pair meets the answerable conditions before the response result generation subunit executes the step of calling the generative big model, learning the document content in the document sequence, generating and outputting the response result corresponding to the query text. The answerable conditions include at least: the timeliness of the document meets the preset timeliness requirements, and the source of the document is a preset data source.

[0141] The response document sequence determination subunit is used to filter out documents that do not meet the respondability conditions from the document sequence when there are documents in the target data pair that do not meet the respondability conditions, thereby obtaining a document sequence for response.

[0142] This application also provides an electronic device in its embodiments. (See reference...) Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0143] like Figure 5 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. When the electronic device is powered on, the RAM 603 also stores various programs and data required for the operation of the electronic device. The processing unit 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0144] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, memory cards, hard drives, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0145] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the information query methods provided in this application.

[0146] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the query methods provided in this application.

[0147] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0148] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0149] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0150] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An information retrieval method, characterized in that, include: In response to receiving query text input by the user, retrieve at least one document; The at least one document is combined with the query text to obtain at least one data pair to be detected; A pre-trained relevance detection model is invoked to process each of the data pairs to be detected, thereby obtaining relevance information for each data pair to be detected. The relevance information is information that characterizes whether there is a relevance between the document and the query text in the data pair to be detected. The relevance detection model is trained using data pair samples as feature values ​​and the relevance information corresponding to the data pair samples as target values. The data pair samples include at least one data pair containing a document sample and a query text sample. The data pairs to be detected that satisfy the preset conditions are identified as target data pairs, where the preset conditions are that the document and the query text are related. Based on the documents in the target data pair, generate and output the response results corresponding to the query text.

2. The information query method according to claim 1, characterized in that, The process of pre-training the correlation detection model includes: Obtain data pairs corresponding to multiple query text samples, and the relevance information of the data pairs. Each data pair includes a positive data pair and multiple negative data pairs. The document samples in the positive data pair are related to the query text samples, while the document samples in the negative data pairs are not related to the query text samples. The relevance information includes a first score indicating the presence of relevance and a second score indicating the absence of relevance. Construct an initial correlation detection model, which includes at least an encoder and a linear classification layer; Based on the data pair samples corresponding to each query text sample and the relevance information of the corresponding annotations of the data pair samples, the initial relevance detection model is trained to obtain the relevance detection model.

3. The information query method according to claim 2, characterized in that, The process of training the initial relevance detection model based on the data pair samples corresponding to each query text sample and the relevance information of the corresponding annotations of the data pair samples to obtain the relevance detection model includes: Based on the data pair samples corresponding to each query text sample, the initial relevance detection model is trained to obtain the training result corresponding to each data pair sample; Based on the training results and correlation information corresponding to each data pair sample, the loss value of the preset loss function is determined; Determine whether the loss value reaches a preset threshold; If the loss value does not reach the preset threshold, the initial relevance detection model is trained based on the data pair samples corresponding to each query text sample to obtain the training result corresponding to each data pair sample until the loss value reaches the preset threshold. When the loss value reaches the preset threshold, the initial correlation detection model is determined to be trained successfully, and the correlation detection model is obtained.

4. The information query method according to any one of claims 1-3, characterized in that, The process by which the correlation detection model processes the data pairs to be detected includes: The encoder of the relevance detection model is invoked to encode the data pair to be detected, thereby obtaining the word vectors corresponding to the data to be detected; The linear classification layer of the relevance detection model is invoked to classify the word vectors according to the degree of relevance between the document represented by the word vectors and the query text, and the classification result is obtained. The classification result is used to characterize whether there is a relevance between the document corresponding to the word vectors and the query text. Based on the classification results, the relevance information corresponding to the word vectors is output.

5. The information query method according to any one of claims 1-4, characterized in that, The step of generating and outputting a response result corresponding to the query text based on the document in the target data pair includes: Based on the relevance information of each target data pair, the documents in the target data pair are sorted to obtain a document sequence sorted according to their relevance to the query text; The generative large model is invoked to learn the document content in the document sequence, generate and output the response results corresponding to the query text.

6. The information query method according to claim 5, characterized in that, Before invoking the generative large model, the following is also included: Identify whether the document in the target data pair meets the respondable conditions, the respondable conditions including at least: the timeliness of the document meets the preset timeliness requirements, and the source of the document is a preset data source; If there are documents in the target data pair that do not meet the respondability criteria, the documents that do not meet the respondability criteria are filtered out from the document sequence to obtain a document sequence for replying.

7. An information query device, characterized in that, include: The document retrieval unit is used to retrieve at least one document in response to receiving query text input by the user; A data combination unit is used to combine the at least one document with the query text to obtain at least one data pair to be detected; The relevance detection unit is used to call a pre-trained relevance detection model to process each of the data pairs to be detected, and obtain the relevance information of each data pair to be detected. The relevance information is information that characterizes whether there is a relevance between the document and the query text in the data pair to be detected. The relevance detection model is trained with data pair samples as feature values ​​and the relevance information corresponding to the data pair samples as target values. The data pair samples include at least one data pair containing a document sample and a query text sample. A data pair confirmation unit is used to determine that the data pairs to be detected that satisfy a preset condition are target data pairs, wherein the preset condition is that the document and the query text are related. The response generation unit is used to generate and output a response result corresponding to the query text based on the document in the target data pair.

8. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the information query method as described in any one of claims 1 to 6.

9. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the information query method as described in any one of claims 1 to 6.

10. A computer storage medium, characterized in that, The storage medium carries one or more computer programs, which, when executed by an electronic device, enable the electronic device to implement the information query method as described in any one of claims 1 to 6.

Images