Confidence evaluation method and device, electronic equipment and storage medium

Abstract

The invention provides a confidence evaluation method and device, electronic equipment and a storage medium. The method comprises the steps that key information and semantic features of a query instruction are accurately captured by query features; the search result features comprehensively represent the core key points and the text features of the document; semantic difference features are determined based on the semantic difference between the query instruction and the search result, text difference features are determined based on the text difference between the query instruction and the search result, and the internal relation between the query instruction and the search result in the semantic and text level is deeply mined; the query features, the search result features, the semantic difference features and the text difference features can comprehensively reflect the matching degree and quality information of the query instruction and the search result, so that confidence coefficient prediction is performed based on the query features, the search result features, the semantic difference features and the text difference features to obtain a confidence coefficient score result; the performance of the RAG retrieval module can be efficiently evaluated, and dependence on external reference data and a complex large language model is got rid of.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a confidence assessment method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology, large language models (LLMs) have achieved remarkable results in natural language processing tasks. However, LLMs suffer from the problem of delayed knowledge updates. Against this backdrop, Retrieval-Augmented Generation (RAG) has emerged and become a hot research topic. RAG aims to combine external knowledge sources with large language models. When generating answers, it first retrieves relevant information from external knowledge sources (such as document repositories and knowledge bases). This information is then used to assist the model in generating more accurate, comprehensive, and relevant answers. This effectively addresses the shortcomings of LLMs in knowledge timeliness, broadens their application scenarios, and holds broad application prospects in areas such as intelligent question answering, information retrieval, and text generation.

[0003] Ragas is a highly representative tool for RAG system evaluation, offering a rich and diverse set of evaluation metrics and striving to build a comprehensive and in-depth evaluation system for RAG systems. However, Ragas suffers from significant drawbacks in its application. Its heavy reliance on large language models for complex semantic understanding and evaluation calculations, coupled with the requirement for pre-determined reference answers for most metrics, makes it difficult to implement in many practical scenarios. This significantly limits its flexibility and breadth of application, highlighting the need for developing new no-reference confidence evaluation schemes. Summary of the Invention

[0004] The present invention provides a confidence assessment method, apparatus, electronic device, and storage medium to address significant deficiencies in the application of Ragas in the prior art. Ragas relies heavily on large language models when performing complex semantic understanding and evaluation calculations, and most indicator calculations require pre-knowledge of reference answers, which greatly limits its flexibility and breadth of application.

[0005] The present invention provides a confidence assessment method, comprising the following steps: Obtaining a query instruction and a search result corresponding to the query instruction, and determining a query feature corresponding to the query instruction and a search result feature corresponding to the search result; Determining a semantic difference feature based on a semantic difference between the query instruction and the search result, and determining a text difference feature based on a text difference between the query instruction and the search result; A confidence prediction is performed based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0006] According to a confidence assessment method provided by the present invention, the step of determining the search result feature corresponding to the search result includes: Performing text quality evaluation on the search results to obtain a text quality score vector; Performing a timeliness evaluation on the search results to obtain a timeliness score vector; Performing an authority evaluation on the search results to obtain an authority score vector; Performing a topic relevance evaluation on the search results to obtain a topic relevance score vector; The search result feature is determined based on the text quality score vector, the timeliness score vector, the authority score vector, and the topic relevance score vector.

[0007] According to a confidence evaluation method provided by the present invention, determining text difference features based on text differences between the query instruction and the search results includes: Performing word segmentation processing on the query instruction and the search result respectively to obtain a query instruction word segmentation result and a search word segmentation result; Perform keyword recognition on the query instruction word segmentation result to obtain query keywords; The frequency and frequency of the query keyword in the search word segmentation results are determined, and the text difference feature is determined based on the frequency and the frequency.

[0008] According to a confidence evaluation method provided by the present invention, determining a semantic difference feature based on the semantic difference between the query instruction and the search result includes: Encoding the query instruction and the search result respectively to obtain a first feature and a second feature, and determining a semantic similarity between the first feature and the second feature; Inputting the query instruction and the search result into an encoder for feature interaction, obtaining basic features output by the encoder, and determining text pair similarity corresponding to text pairs constructed by the query instruction and the search result based on the basic features; The semantic difference feature is determined based on the semantic similarity and the text pair similarity.

[0009] According to a confidence assessment method provided by the present invention, inputting the query instruction and the search result into an encoder for feature interaction to obtain basic features output by the encoder includes: The query instruction and the search result are spliced and input into the encoder for lexical interaction, syntactic interaction and semantic interaction respectively, so as to obtain lexical features, syntactic features and semantic features output by the encoder.

[0010] According to a confidence assessment method provided by the present invention, the step of determining the query feature corresponding to the query instruction includes: Performing dependency syntax analysis on the query instruction to determine a dependency relationship type in the query instruction and a relationship type frequency corresponding to the dependency relationship type; Determining dependency syntactic features based on the dependency relationship type and the relationship type frequency; Based on the recognition model, identifying typos and incorrect sentences in the query instruction, and determining text quality features based on the typos and incorrect sentences; The query feature is determined based on the dependency syntactic feature and the text quality feature.

[0011] According to a confidence assessment method provided by the present invention, the step of training the recognition model includes: Obtaining a first sample text, a labeled typo text of the first sample text, and an initial model; Inputting the first sample text into the initial model to obtain a predicted error text output by the initial model; Determining a first loss based on a difference between the predicted error text and the labeled typo text, and iterating parameters of the initial model based on the first loss to obtain an initial recognition model; Obtaining a second sample text and labeled incorrect sentences of the second sample text; Inputting the second sample text into the initial recognition model to obtain a predicted error sentence output by the initial recognition model; Based on the difference between the predicted error sentence and the label error sentence, a second loss is determined, and based on the second loss, parameters of the initial recognition model are iterated to obtain the recognition model.

[0012] The present invention also provides a confidence assessment device, comprising the following units: an acquiring unit, configured to acquire a query instruction and a search result corresponding to the query instruction, and determine a query feature corresponding to the query instruction and a search result feature corresponding to the search result; a difference determining unit, configured to determine a semantic difference feature based on a semantic difference between the query instruction and the search result, and to determine a text difference feature based on a text difference between the query instruction and the search result; A confidence score determination unit is configured to perform confidence prediction based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements any of the above-described confidence assessment methods when executing the program.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which implements any of the above-mentioned confidence assessment methods when executed by a processor.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the confidence assessment method described above is implemented.

[0016] The confidence assessment method, device, electronic device and storage medium provided by the present invention first determine the query features corresponding to the query instruction to accurately capture the key information and semantic features of the query instruction; secondly, determine the search result features corresponding to the search results to comprehensively characterize the core points and text characteristics of the document; thirdly, based on the semantic differences between the query instruction and the search results, determine the semantic difference features, and based on the text differences between the query instruction and the search results, determine the text difference features, and deeply explore the intrinsic connection between the two at the semantic and text levels. Since the query features, search result features, semantic difference features and text difference features can comprehensively reflect the matching degree and quality information between the query instruction and the search results, confidence prediction is performed based on the query features, search result features, semantic difference features and text difference features to obtain a confidence score result, which can efficiently evaluate the performance of the RAG retrieval module, get rid of the dependence on external reference data and complex large language models, and greatly improve the practicality and flexibility of confidence assessment. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 It is a schematic diagram illustrating evaluation indicators of Ragas searcher components in the prior art.

[0019] Figure 2 This is one of the flow charts of the confidence assessment method provided by the present invention.

[0020] Figure 3 This is the second flow chart of the confidence assessment method provided by the present invention.

[0021] Figure 4 This is the third flow chart of the confidence assessment method provided by the present invention.

[0022] Figure 5 It is a flowchart of the search result feature generation steps provided by the present invention.

[0023] Figure 6 It is a flowchart of the steps of generating the semantic difference feature provided by the present invention.

[0024] Figure 7 It is a flowchart of the steps of generating text difference features provided by the present invention.

[0025] Figure 8 This is the fourth flow chart of the confidence assessment method provided by the present invention.

[0026] Figure 9 It is a flowchart of the steps of determining the query features provided by the present invention.

[0027] Figure 10 It is a flowchart of the training steps of the recognition model provided by the present invention.

[0028] Figure 11 It is a structural diagram of the confidence assessment device provided by the present invention.

[0029] Figure 12 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0030] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0031] The terms "first," "second," and the like in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type.

[0032] Common evaluation methods for RAG search effectiveness in related technologies include reference-answer-based and no-reference evaluation metrics. Reference-answer-based evaluation metrics rely on pre-defined reference answers or annotated data and evaluate the performance of the RAG system by comparing the similarity or consistency between the generated results and the reference answers. No-reference evaluation metrics do not rely on reference answers, but instead measure the performance of the RAG system by evaluating the intrinsic quality of the generated content.

[0033] The evaluation indicators based on the reference answers are as follows: 1) Recall@k: This metric primarily measures the recall of correct, relevant documents among the top k documents in the retrieval results. A high Recall@k value indicates that the retrieval system is better at finding documents relevant to the question, but it does not reflect the ranking quality of the retrieval results or the effectiveness of the retrieved documents in generating the final answer.

[0034] 2) MRR (Mean Reciprocal Rank): This is primarily used to evaluate the ranking of the first correct, relevant document in the search results. A higher MRR indicates a higher ranking of the correct, relevant document in the search results, reflecting the accuracy of the search system's relevance judgment. However, it also fails to fully consider the overall impact of the search results on answer generation.

[0035] 3) F1 score: This combines Precision and Recall to provide a more comprehensive measure of the quality of retrieval results. In RAG evaluation, it considers the proportion of retrieved relevant documents to all retrieved documents, as well as the proportion of retrieved relevant documents to all truly relevant documents. However, these reference-based evaluation metrics rely on manually annotated reference data, a process that is time-consuming, labor-intensive, and potentially subjective.

[0036] The non-reference evaluation indicators are as follows: 1) Diversity metrics: These metrics are used to measure the diversity of retrieval results, for example, by calculating the similarity between documents in the retrieval results. If the retrieval results are too similar, it may mean that the retrieval system has not fully explored the diversity of the knowledge source and may miss some important information.

[0037] 2) Novelty metric: This metric measures whether search results contain up-to-date information, rather than simply reiterating existing knowledge. However, accurately defining novelty is challenging in practical applications, and a single novelty metric alone cannot fully reflect the performance of the RAG search module.

[0038] In summary, the existing RAG search effect evaluation methods have many limitations.

[0039] Ragas is a leading tool for RAG system evaluation, offering a rich and diverse set of metrics. It strives to build a comprehensive and in-depth evaluation framework for RAG systems. In addition to the common Recall@k, MRR, and F1 score metrics, it also includes BLEU (bilingual evaluation understudy), ROUGE (Recall-Oriented Understudy for Gisting Evaluation), and METEOR (Metric for Evaluation of Translation with Explicit Ordering). BLEU is mainly based on the n-gram idea, and measures the accuracy of text generation by calculating the ratio of the maximum number of n-grams that appear in the candidate text and the reference text to the number of n-grams in the candidate text; ROUGE, from the perspective of recall rate, calculates the proportion of the overlapping part of the candidate text and the reference text to the reference text, which is used to evaluate the relevance of the generated text and the reference text; METEOR comprehensively considers factors such as precision, recall rate, and the alignment of the generated text and the reference text, and obtains a comprehensive evaluation score through specific weighted calculations. These indicators evaluate the quality of the answers generated by the RAG system from different aspects.

[0040] In addition, Ragas also includes the following feature evaluation indicators: Figure 1 This is a schematic diagram illustrating the evaluation index of the Ragas searcher component in the prior art, such as Figure 1 As shown in the figure, 1. Context relevancy: evaluates the relevance of the retrieved context to the user's question, calculated by identifying the number of sentences in the retrieved context that are relevant to the question. Ideally, the retrieved context should only contain information that answers the question.

[0041] 2. Answer relevance: This evaluates the relevance of the generated answer to the user's question. This evaluation of answer relevance does not take into account ground truth, but rather penalizes answers that are incomplete or contain redundant details. To calculate this score, the LLM is prompted to generate appropriate questions for the answer multiple times and measures the average cosine similarity between these generated questions and the original question. The basic idea is that if the generated answer accurately solves the original question, the LLM should be able to generate questions from the answer that match the original question.

[0042] 3. Faithfulness: Evaluates the factual consistency of the generated answer with the given context. If all the basic facts (claims) proposed in the answer can be inferred from the given context, the generated answer is considered faithful. To calculate this score, a set of claims are first identified from the generated answer. Then, each of these claims is cross-checked with the given context to determine whether they can be inferred from the given context.

[0043] 4. Context Precision: Evaluates whether all items related to the ground-truth presented in the context are ranked high. Ideally, all relevant document chunks must appear in the top layer.

[0044] 5. Context Recall: This evaluates how closely the retrieved context agrees with the human-provided ground truth answer. To calculate this score, each sentence in the ground truth answer is analyzed to determine whether it can be attributed to the retrieved context. Ideally, all sentences in the ground truth answer should be attributed to the retrieved context.

[0045] In summary, Ragas has significant flaws in its application. It relies heavily on large language models for complex semantic understanding and evaluation calculations, and most metric calculations require prior knowledge of the reference answers. This is difficult to achieve in many practical application scenarios, greatly limiting its flexibility and breadth of application, and highlighting the need to develop new reference-free confidence evaluation schemes.

[0046] Based on the above problems, the present invention provides a confidence assessment method. Figure 2 This is one of the flow charts of the confidence assessment method provided by the present invention. Figure 3 This is the second flow chart of the confidence assessment method provided by the present invention, such as Figure 2 、 Figure 3 As shown, the method includes step 110 , step 120 and step 130 .

[0047] Step 110: Obtain a query instruction and a search result corresponding to the query instruction, and determine a query feature corresponding to the query instruction and a search result feature corresponding to the search result.

[0048] Specifically, you can obtain a query instruction (Query) and the corresponding search results (Doc). Here, a query instruction refers to the text or command submitted by the user to the information retrieval system to express their information needs. The query instruction can include keywords, phrases, or more complex query expressions, instructing the system to find documents or data related to the user's needs. The search results are the relevant information retrieved by the information retrieval system from its indexed document collection or database based on the query instruction submitted by the user.

[0049] Then, the query feature corresponding to the query instruction and the search result feature corresponding to the search result may be determined.

[0050] Here, query features are generated using state-of-the-art (SOTA) natural language processing models. Specifically, query instructions are subjected to dependency syntactic analysis to determine dependency syntactic features. Typos and grammatical errors in the query instructions are also identified to generate text quality features. Finally, query features are determined based on these dependency syntactic and text quality features. For example, the dependency syntactic analysis capabilities of DDParser or LTP (Language Technology Platform) can be used to deeply analyze the grammatical structure of the query instructions, particularly the dependencies between terms. Furthermore, the frequency of occurrence of different sentence structures is counted to construct query features. Query features can also encompass complex intent, typos, grammatical errors, and redundant information.

[0051] Here, the search result features are extremely critical for comprehensively and accurately evaluating the performance of the RAG retrieval module. The feature engineering of the search results in the embodiment of the present invention covers four core dimensions, namely text quality evaluation, timeliness evaluation, authority evaluation, and topic relevance evaluation of the title and content.

[0052] Step 120 : determining a semantic difference feature based on the semantic difference between the query instruction and the search result, and determining a text difference feature based on the text difference between the query instruction and the search result.

[0053] Specifically, semantic difference features can be determined based on the semantic differences between the query instruction and the search results. Here, the query instruction and the search results can be encoded separately using SimCSE (Simple Contrastive Learning of Sentence Embeddings), or ConSERT (Contrastive Framework for Sentence Representation Transfer), or CoSENT (Cosine Sentence Transformer), to obtain a first feature corresponding to the query instruction and a second feature corresponding to the search result. This is not specifically limited in the embodiments of the present invention.

[0054] It can be understood that the deep neural networks in SimCSE, ConSERT, and CoSENT encode the query instructions and search results respectively, obtaining the text embedding corresponding to the query instructions and the text embedding corresponding to the search results. The similarity between the two representations is then calculated to obtain the semantic difference features between the query instructions and the search results.

[0055] It can be understood that the deep neural networks in SimCSE, ConSERT, and CoSENT encode the query instructions and search results separately to obtain the text representation (embedding) corresponding to the query instructions and the text representation corresponding to the search results. The similarity between the two representations is then calculated to obtain the semantic difference features between the query instructions and the search results. This is a representation-based matching method.

[0056] Furthermore, considering that calculating text similarity at the final stage would be overly dependent on the quality of the text representation and would also lose basic but critical text features (such as lexical and syntactic features), we propose to interact with text features as early as possible to capture more fundamental features. Finally, at higher levels (such as the output layer of the Transformer model), the matching score is calculated based on these basic matching features. The query command and search results can be input into the encoders of Conv-KNRM (Convolutional Kernelized Neural Ranking Machine), RE2 (Residual Embedding Encoder), and Keyword-BERT (Keyword-Based Bidirectional Encoder Representations from Transformers) for feature interaction. This generates the basic features output by the encoders, which are then used to determine the similarity of the text pairs.

[0057] Since the input of the above-mentioned models is two texts, the query instruction and the search result, and the output is the similarity value of the text pair, no vector representation of the text is generated. Therefore, this type of matching processing method can be determined as an interaction-based matching method.

[0058] Finally, under the two matching methods of representation-based matching and interaction-based matching, multiple models will output similarity results of multiple query instructions and search results. The above similarity results are combined to construct semantic difference features, which serve as an important basis for the subsequent reference-free confidence evaluation of the RAG retrieval module.

[0059] Furthermore, text difference features can be determined based on the textual differences between the query and search results. Textual differences here refer to the differences between the query and search results at the textual level. For example, textual differences can include the frequency and frequency of the query keyword in the search results (keyword hit rate). Based on this frequency and frequency, textual difference features between the query and search results can be determined. Keyword hit rate can be achieved by performing word segmentation on the query and search results and using the Term Weight algorithm.

[0060] Step 130 : Confidence prediction is performed based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0061] Specifically, after obtaining the query features, search result features, semantic difference features, and text difference features, confidence prediction can be performed based on the query features, search result features, semantic difference features, and text difference features to obtain a confidence score result.

[0062] It should be noted that query features, search result features, semantic difference features, and text difference features can be understood as confidence feature vectors constructed through feature engineering. They will be input into the confidence model to predict the final confidence score results of the query instructions and search results. The embodiment of the present invention follows the principle of Occam's razor, abandons complex and easily overfitting models, and instead uses basic machine learning models such as MLP (Multilayer Perceptron), SVM (Support Vector Machine), XGBoost (eXtreme Gradient Boosting) to complete the confidence prediction task. In actual applications, it is found that the final effect of the confidence model depends to a large extent on whether massive and accurate supervised training data can be constructed.

[0063] It is understandable that the confidence model can output a confidence score between 0 and 1, thereby efficiently evaluating the performance of the RAG retrieval module, getting rid of the dependence on external reference data and complex large language models, and greatly improving the practicality and autonomy of confidence assessment.

[0064] To effectively address the critical issue of search result confidence evaluation without a reference, this embodiment of the present invention specifically designs a set of standardized and executable data annotation standards. This standard divides the confidence of query instructions (query) and search result (doc) pairs into five levels based on multiple aspects such as subject, intent, and qualifiers. See Table 1 below for details: Table 1 It is understandable that in terms of confidence model training, the embodiments of the present invention propose an innovative and efficient solution. First, to ensure the accuracy and consistency of data annotation, a standardized and executable standard operating procedure (SOP) is designed: the search results corresponding to the query instructions are graded from multiple dimensions, covering key aspects such as subject, intent, and qualifiers, and subdivided into five levels. In this process, large language model annotation is introduced to improve annotation efficiency, and combined with a manual sampling and correction mechanism, the advantages of both are fully utilized to construct high-quality training data, laying a solid foundation for confidence model training.

[0065] It should be noted that the query feature (query feature) determination module, search result feature (doc feature) module, query and doc semantic difference feature determination module, and query and doc text difference feature determination module in the embodiments of the present invention are all pluggable and can be independently trained and predicted. This design allows for flexible selection of one or more feature construction modules based on specific usage scenarios and online machine performance, and a smooth integration of the confidence score prediction process. This effectively improves the adaptability and flexibility of the entire confidence assessment method, meeting the diverse RAG retrieval module evaluation requirements.

[0066] The method provided by the embodiment of the present invention first determines the query features corresponding to the query instruction to accurately capture the key information and semantic features of the query instruction; secondly, determines the search result features corresponding to the search results to comprehensively characterize the core points and text characteristics of the document; thirdly, based on the semantic differences between the query instruction and the search results, determines the semantic difference features, and based on the text differences between the query instruction and the search results, determines the text difference features, and deeply explores the intrinsic connection between the two at the semantic and text levels. Since the query features, search result features, semantic difference features and text difference features can comprehensively reflect the matching degree and quality information between the query instruction and the search results, confidence prediction is performed based on the query features, search result features, semantic difference features and text difference features to obtain a confidence score result, which can efficiently evaluate the performance of the RAG retrieval module, get rid of the dependence on external reference data and complex large language models, and greatly improve the practicality and flexibility of confidence evaluation.

[0067] Based on the above embodiments, Figure 4 This is the third flow chart of the confidence assessment method provided by the present invention, such as Figure 4 As shown, the method includes: Step 310: Obtain a query instruction and search results corresponding to the query instruction, and determine a query feature corresponding to the query instruction.

[0068] Specifically, a query is the text or command that a user submits to an information retrieval system to express their information needs. A query can include keywords, phrases, or more complex query expressions, instructing the system to find documents or data relevant to the user's needs. Search results are the relevant information that the information retrieval system retrieves from its indexed document collection or database based on the query submitted by the user.

[0069] In this embodiment of the present invention, query features are extracted using state-of-the-art natural language processing models. Specifically, dependency syntactic analysis techniques are used to determine the dependencies between terms in a query, thereby extracting dependency syntactic features. At the same time, typos and grammatical errors in the query are identified, thereby generating text quality features. Ultimately, query features are constructed by combining dependency syntactic features with text quality features.

[0070] Step 320: perform text quality evaluation on the search results to obtain a text quality score vector; perform timeliness evaluation on the search results to obtain a timeliness score vector; perform authority evaluation on the search results to obtain an authority score vector; perform topic relevance evaluation on the search results to obtain a topic relevance score vector; determine the search result features based on the text quality score vector, the timeliness score vector, the authority score vector, and the topic relevance score vector.

[0071] Specifically, Figure 5 This is a flow chart of the steps for generating search result features provided by the present invention, such as Figure 5 As shown, text quality assessment can be performed on search results to generate a text quality score vector. For example, the LanguageTool open-source model can be used for text quality assessment. The LanguageTool model meticulously checks search results for grammatical correctness, appropriate vocabulary usage, and reasonable sentence structure. By analyzing the number and severity of spelling errors, grammatical errors, and improper punctuation in search results, it provides a quantitative basis for text quality assessment and generates a text quality score vector.

[0072] The search results are then evaluated for timeliness, yielding a timeliness score vector. For example, web crawler technology is used to regularly capture the publication time information of documents on relevant topics on the internet. By comparing the time difference between the search results and the most recently published document, and analyzing the update frequency trends of documents on the topic within a specific time period, the timeliness of the search results can be determined. For example, if knowledge in a certain field is rapidly updated, but the information in the search results is published relatively recently and a large number of new documents have recently emerged in that field, the timeliness of the search results can be determined to be low, and a timeliness score vector can be determined accordingly.

[0073] Furthermore, the authority of the search results can be evaluated to obtain an authority score vector. For example, in terms of authority evaluation, knowledge graph technology is used to correlate and match elements such as the citation source, author information, and publishing organization in the search results with the authoritative data sources in the knowledge graph to determine the authority level of the search results. For example, if the author of the document in the search results is well-known in a specific field and the references cited are mostly from core journals, well-known research institutions, or authoritative industry reports in the field, a higher authority score can be assigned, thereby determining the authority score vector.

[0074] Then, the search results can be evaluated for topic relevance to obtain a topic relevance score vector. For example, in terms of evaluating the topic relevance of the title and content, Sentence-BERT (Sentence Embeddings using Siamese BERT-Networks) or CoSENT is used to convert the title and content into semantic vectors respectively. The topic relevance is then measured by calculating the similarity between the two vectors. The higher the similarity, the closer the topic fits the title and content; conversely, the lower the similarity between the two vectors, the lower the topic fits the title and content.

[0075] Finally, the search result features are determined based on the text quality score vector, timeliness score vector, authority score vector and topic relevance score vector.

[0076] Furthermore, in practical applications, the open-source model can be optimized and trained using real-world corpora accumulated from multiple apps. Ultimately, one or more feature vectors are generated based on the evaluation results of each dimension. These feature vectors comprehensively and accurately characterize the document's characteristics, providing strong support for the no-reference confidence evaluation of the RAG retrieval module.

[0077] Step 330 : determining a semantic difference feature based on the semantic difference between the query instruction and the search result, and determining a text difference feature based on the text difference between the query instruction and the search result.

[0078] Specifically, Figure 6 is a flow chart of the steps for generating the semantic difference feature provided by the present invention, such as Figure 6As shown, a semantic difference feature can be determined based on the semantic difference between the query instruction and the search result. Here, the query instruction and the search result can be encoded separately using SimCSE, ConSERT, or CoSENT, thereby obtaining a first feature corresponding to the query instruction and a second feature corresponding to the search result. This is not specifically limited in the embodiments of the present invention.

[0079] It can be understood that the query instructions and search results are encoded separately through the deep neural networks in SimCSE, ConSERT and CoSENT to obtain the representation of the text corresponding to the query instruction and the representation of the text corresponding to the search result. The similarity of the two representations is then calculated to obtain the semantic similarity between the query instruction and the search result. This method is a representation-based matching method.

[0080] Furthermore, the query instructions and search results can be input into the encoders in Conv-KNRM, RE2, Keyword-BERT and Cross-Encoder respectively for feature interaction to obtain the basic features output by the encoder. Then, based on the basic features, the similarity of the text pairs is determined. This method is an interaction-based matching method.

[0081] Finally, under the two matching methods of representation-based matching and interaction-based matching, multiple models will output similarity results of multiple query instructions and search results. The above similarity results are combined to construct semantic difference features, which serve as an important basis for the subsequent reference-free confidence evaluation of the RAG retrieval module.

[0082] Step 340 : Confidence prediction is performed based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0083] Specifically, after obtaining query features, search result features, semantic difference features, and textual difference features, these features can be used to perform confidence prediction, thereby obtaining a confidence score. Specifically, these features can be regarded as confidence feature vectors formed after feature engineering processing and input into the confidence model to predict the confidence score between the query instruction and the search result.

[0084] In an embodiment of the present invention, following the principle of Occam's razor, complex and easily overfitting models are abandoned, and instead basic machine learning models such as MLP, SVM, and XGBoost are used as confidence prediction models to complete the confidence prediction task. The confidence model can output a confidence score between 0 and 1, which helps to efficiently evaluate the performance of the RAG retrieval module. In this way, it is possible to get rid of the dependence on external reference data and complex large language models, thereby greatly improving the practicality and autonomy of confidence assessment.

[0085] The method provided by the embodiment of the present invention performs text quality evaluation on search results to obtain a text quality score vector, then performs timeliness evaluation on search results to obtain a timeliness score vector, performs authority evaluation on search results to obtain an authority score vector, then performs topic relevance evaluation on search results to obtain a topic relevance score vector, and finally, based on the text quality score vector, timeliness score vector, authority score vector and topic relevance score vector, determines the search result characteristics, and evaluates the search results from multiple key dimensions, which can more comprehensively reflect the search result characteristics, thereby avoiding the deviation of search result characteristics that may be caused by single-dimensional evaluation.

[0086] Based on the above embodiment, determining text difference features based on the text differences between the query instruction and the search results in step 120 includes: Step 121, performing word segmentation processing on the query instruction and the search result respectively to obtain a query instruction word segmentation result and a search word segmentation result; Step 122: performing keyword recognition on the query instruction segmentation result to obtain query keywords; Step 123: Determine the frequency and the frequency of the query keyword in the search word segmentation results, and determine the text difference feature based on the frequency and the frequency.

[0087] Specifically, Figure 7 This is a flow chart of the steps for generating text difference features provided by the present invention. Figure 7 As shown, a word segmenter can be used to segment the query instruction (query) and the search result (doc), respectively, to obtain the query instruction word segmentation result and the search word segmentation result. The word segmenter can accurately segment continuous text strings into independent word units based on specific language rules and algorithms, laying the foundation for subsequent keyword recognition and analysis. Here, the word segmenter can be THULAC, NLTK (Natural Language Toolkit), Jieba, etc., and this embodiment of the present invention does not specifically limit this.

[0088] The Term Weight algorithm is then used to identify keywords in the query segmentation results, yielding the query keywords. This algorithm comprehensively considers the importance of words in the text, analyzing factors such as their frequency of occurrence, their uniqueness in specific contexts, and their associations with other words to select key terms as keywords. The frequency and recurrence of these keywords in the search segmentation results are then retrieved. Based on these frequency and recurrence, text difference features are constructed to characterize the similarity between the query and the document text.

[0089] It can be understood that the text difference feature effectively quantifies the hits of keywords in the query instructions in the search results, so that the correlation between the query instructions and the search results at the text level can be presented in an intuitive and computable form, providing strong data support for accurately evaluating the text similarity between the query instructions and the search results.

[0090] Based on the above embodiments, Figure 8 This is the fourth flow chart of the confidence assessment method provided by the present invention, such as Figure 8 As shown, the method includes: Step 810: Obtain a query instruction and search results corresponding to the query instruction, and determine a query feature corresponding to the query instruction.

[0091] Specifically, a query is the text or command that a user submits to an information retrieval system to express their information needs. A query can include keywords, phrases, or more complex query expressions, instructing the system to find documents or data relevant to the user's needs. Search results are the relevant information that the information retrieval system retrieves from its indexed document collection or database based on the query submitted by the user.

[0092] In this embodiment of the present invention, query features are extracted using state-of-the-art natural language processing models. Specifically, dependency syntactic analysis techniques are used to determine the dependencies between terms in a query, thereby extracting dependency syntactic features. At the same time, typos and grammatical errors in the query are identified, thereby generating text quality features. Ultimately, query features are constructed by combining dependency syntactic features with text quality features.

[0093] Step 820: perform text quality evaluation on the search results to obtain a text quality score vector; perform timeliness evaluation on the search results to obtain a timeliness score vector; perform authority evaluation on the search results to obtain an authority score vector; perform topic relevance evaluation on the search results to obtain a topic relevance score vector; determine the search result features based on the text quality score vector, the timeliness score vector, the authority score vector, and the topic relevance score vector.

[0094] Specifically, the search results are evaluated to obtain score vectors for text quality, timeliness, authority, and topic relevance, as follows: Text Quality Score Vector: This uses the LanguageTool open-source model to detect grammatical correctness, vocabulary usage, and sentence structure in search results. It also analyzes the number and severity of spelling, grammar, and punctuation errors to quantify text quality.

[0095] Timeliness score vector: Use web crawlers to capture the release time information of relevant topic documents, compare the time difference between the search results and the latest documents, and analyze the update frequency trend of the topic documents to determine whether the search results are outdated.

[0096] Authority Score Vector: Using knowledge graph technology, we match and analyze the citation source, author information, and publishing organization in the search results with authoritative data sources to determine the authority level. Authors with high reputation and citations from core journals or authoritative institutions are given a higher authority score.

[0097] Topic relevance score vector: Use Sentence-BERT or CoSENT to convert the title and body content into semantic vectors and calculate vector similarity to measure topic relevance. High similarity indicates close fit, while low similarity indicates low fit.

[0098] Furthermore, in practical applications, the open-source model can be optimized and trained using real-world corpora accumulated from multiple apps. Ultimately, one or more feature vectors are generated based on the evaluation results of each dimension. These feature vectors comprehensively and accurately characterize the document's characteristics, providing strong support for the no-reference confidence evaluation of the RAG retrieval module.

[0099] Step 830: Encode the query instruction and the search result respectively to obtain a first feature and a second feature, and determine the semantic similarity between the first feature and the second feature; input the query instruction and the search result into an encoder for feature interaction to obtain basic features output by the encoder, and based on the basic features, determine the text pair similarity corresponding to the text pair constructed by the query instruction and the search result; determine the semantic difference feature based on the semantic similarity and the text pair similarity.

[0100] Specifically, the query instruction and search result can be encoded separately to obtain a first feature and a second feature, and the semantic similarity between the first feature and the second feature can be determined. The encoding model can use SimCSE, ConSERT, and CoSENT models. In the initial stage, the query instruction and search result are processed separately, and encoded through a deep neural network to obtain the first feature and the second feature. The semantic similarity between the first feature and the second feature is then calculated to obtain the semantic similarity between the query and the document.

[0101] Then, the query instruction and search results are input into the encoder for feature interaction to obtain the basic features output by the encoder, and based on the basic features, the text pair similarity corresponding to the text pair constructed by the query instruction and the search results is determined.

[0102] It is understandable that feature interaction is performed between the query instruction and the search result at the encoding stage, and this feature interaction can capture the direct relationship between the query instruction and the search result.

[0103] Finally, semantic difference features can be determined based on semantic similarity and text pair similarity.

[0104] Step 840, perform word segmentation processing on the query instruction and the search result respectively to obtain a query instruction word segmentation result and a search word segmentation result; perform keyword recognition on the query instruction word segmentation result to obtain a query keyword; determine the frequency and frequency of the query keyword in the search word segmentation result, and determine the text difference feature based on the frequency and the frequency.

[0105] Specifically, a word segmenter can be used to segment the query instruction (query) and the search results (doc), respectively, to obtain the query instruction word segmentation results and the search word segmentation results. The word segmenter can accurately segment continuous text strings into independent word units based on specific language rules and algorithms, laying the foundation for subsequent keyword identification and analysis. Here, the word segmenter can be THULAC, NLTK (Natural Language Toolkit), Jieba, etc., and this embodiment of the present invention does not specifically limit this.

[0106] The Term Weight algorithm is then used to identify keywords in the query segmentation results, yielding the query keywords. This algorithm comprehensively considers the importance of words in the text, analyzing factors such as their frequency of occurrence, their uniqueness in specific contexts, and their associations with other words to select key terms as keywords. The frequency and recurrence of these keywords in the search segmentation results are then retrieved. Based on these frequency and recurrence, text difference features are constructed to characterize the similarity between the query and the document text.

[0107] It can be understood that the text difference feature effectively quantifies the hits of keywords in the query instructions in the search results, so that the correlation between the query instructions and the search results at the text level can be presented in an intuitive and computable form, providing strong data support for accurately evaluating the text similarity between the query instructions and the search results.

[0108] In addition, in actual applications, the real corpus accumulated by the app can be used as training data to optimize the training of the above-mentioned word segmenter and term weight model.

[0109] Step 850 : Confidence prediction is performed based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0110] Specifically, after obtaining query features, search result features, semantic difference features, and textual difference features, these features can be used to perform confidence prediction, thereby obtaining a confidence score. Specifically, these features can be regarded as confidence feature vectors formed after feature engineering processing and input into the confidence model to predict the confidence score between the query instruction and the search result.

[0111] In an embodiment of the present invention, following the principle of Occam's razor, complex and easily overfitting models are abandoned, and instead basic machine learning models such as MLP, SVM, and XGBoost are used as confidence prediction models to complete the confidence prediction task. The confidence model can output a confidence score between 0 and 1, which helps to efficiently evaluate the performance of the RAG retrieval module. In this way, it is possible to get rid of the dependence on external reference data and complex large language models, thereby greatly improving the practicality and autonomy of confidence assessment.

[0112] It should be noted that the embodiments of this invention introduce a working model that combines large-scale language model annotation with manual spot checking and correction. Based on the aforementioned data annotation standards, the prompts used for large-scale model annotation were repeatedly refined. Through extensive experimentation and debugging, a stable and efficient query and document confidence evaluation workflow was ultimately established. Notably, this process not only plays a core role in the production of supervised training data, effectively ensuring its quality and scale, but can also be directly applied to the end-to-end evaluation of general retrieval solutions, providing a reliable and convenient approach for performance evaluation and optimization of the entire retrieval system.

[0113] Based on the above embodiment, in step 830, the query instruction and the search result are input into the encoder for feature interaction to obtain the basic features output by the encoder, including: Step 831: The query instruction and the search result are spliced and input into the encoder for lexical interaction, syntactic interaction and semantic interaction respectively, so as to obtain lexical features, syntactic features and semantic features output by the encoder.

[0114] Specifically, the query and search results can be concatenated and fed into an encoder for lexical, syntactic, and semantic interactions, resulting in the encoder's output of lexical, syntactic, and semantic features. Lexical interaction refers to the relationship between identical or similar words in the query and search results. Syntactic interaction refers to the similarity in sentence structure between the query and search results. Semantic interaction refers to the degree of semantic match between the query and search results.

[0115] The method provided in the embodiment of the present invention comprehensively evaluates the correlation between query instructions and search results from three levels: lexical, syntactic and semantic, and can more accurately reflect the degree of match between the two. Moreover, by comprehensively considering vocabulary, sentence structure and semantic information, it can improve the accuracy and relevance of retrieval results and better meet user needs.

[0116] Based on the above embodiments, Figure 9 It is a flowchart of the steps of determining the query features provided by the present invention, such as Figure 9 As shown, the method includes: Step 910: performing dependency syntax analysis on the query instruction to determine the dependency relationship type in the query instruction and the relationship type frequency corresponding to the dependency relationship type; Step 920: Determine dependency syntactic features based on the dependency relationship type and the relationship type frequency; Step 930: Identify typos and grammatical errors in the query instruction based on the recognition model, and determine text quality features based on the typos and grammatical errors. Step 940: Determine the query feature based on the dependency syntactic feature and the text quality feature.

[0117] Specifically, the query instruction is subjected to dependency syntax analysis to determine the dependency relationship type in the query instruction and the relationship type frequency corresponding to the dependency relationship type. Table 2 below is a dependency relationship type table: Table 2 Next, dependency syntactic features are determined based on the dependency relationship types and their frequencies. Dependency relationship types reflect the grammatical relationships between words in a sentence, such as subject-verb, verb-object, and preposition-object. These relationship types are the foundation of dependency syntactic analysis. The frequency of a relationship type refers to the number of times different dependency relationship types appear in a sentence. For example, a sentence may contain multiple "attributive modifier" relationships. This frequency information can be used as a feature for subsequent natural language processing tasks.

[0118] Furthermore, the result of dependency parsing can be represented as a dependency tree, where each node represents a word and each edge represents a dependency relationship. Structural features of the dependency tree, such as its depth and number of branches, can also be used as part of the dependency syntactic features.

[0119] Among them, dependency syntactic features are used to reflect the grammatical structural relationship between words in a sentence, including the types of dependency relationships between words, the frequency of occurrence of these relationships, the structural information of the dependency tree, and the dependency path between words.

[0120] It is understandable that this method can accurately capture the complex intent features and long and difficult sentence features contained in the query instructions, thereby providing an important basis for the subsequent understanding of the semantic connotation of the query instructions.

[0121] Furthermore, the recognition model can be used to identify spelling errors and grammatical errors in the query, and based on these errors, text quality features can be determined. The recognition model can be a BERT (Bidirectional encoder representations from transformers) model or an Ernie (Enhanced Representation through Knowledge Integration) model, etc., and is not specifically limited in this embodiment of the present invention.

[0122] Here, the recognition model may be obtained through pre-training. For example, the recognition model may be obtained through pre-training for typos recognition and incorrect sentence recognition.

[0123] Finally, query features can be determined based on dependency syntactic features and text quality features.

[0124] Based on the above embodiments, Figure 10 It is a flow chart of the training steps of the recognition model provided by the present invention, such as Figure 10 As shown, the method includes: Step 1010: Obtain a first sample text, a labeled typo text of the first sample text, and an initial model; Step 1020: input the first sample text into the initial model to obtain a predicted error text output by the initial model; Step 1030: determining a first loss based on the difference between the predicted error text and the labeled typo text, and performing parameter iteration on the initial model based on the first loss to obtain an initial recognition model; Step 1040: Obtain a second sample text and labeled incorrect sentences of the second sample text; Step 1050: input the second sample text into the initial recognition model to obtain a predicted error sentence output by the initial recognition model; Step 1060: Determine a second loss based on the difference between the predicted error sentence and the labeled error sentence, and perform parameter iteration on the initial recognition model based on the second loss to obtain the recognition model.

[0125] Specifically, a first sample text, a labeled typo text of the first sample text, and an initial model can be obtained. Here, the parameters of the initial model can be pre-set or randomly generated, and the embodiment of the present invention does not specifically limit this.

[0126] Here, the first sample text can be directly input by the user, or obtained by transcribing the collected audio, or obtained by collecting an image through an image collection device such as a scanner, mobile phone, camera, tablet, etc., and performing OCR (Optical Character Recognition) on the image. This embodiment of the present invention does not specifically limit this.

[0127] After obtaining the first sample text, the first sample text may be input into the initial model to obtain the predicted error text output by the initial model.

[0128] Furthermore, a first loss can be determined based on the difference between the predicted error text and the label typo text, and the parameters of the initial model can be iterated based on the first loss, and the initial model after the parameter iteration is completed is used as the initial recognition model.

[0129] It can be understood that the greater the difference between the predicted error text and the labeled typo text, the greater the first loss; the smaller the difference between the predicted error text and the labeled typo text, the smaller the first loss.

[0130] Here, the initial model may be a Bert model or an Ernie model, etc., which is not specifically limited in the embodiment of the present invention.

[0131] After the initial model is trained on typos to obtain an initial recognition model, the initial recognition model can be trained on incorrect sentence recognition to obtain a final recognition model, as follows: A second sample text and labeled incorrect sentences of the second sample text are obtained, wherein the second sample text may be the same as or different from the first sample text, which is not specifically limited in the embodiment of the present invention.

[0132] The second sample text may be directly input by the user, or may be obtained by transcribing the collected audio, or may be obtained by collecting an image through an image collection device such as a scanner, mobile phone, camera, tablet, etc. and performing OCR on the image. This embodiment of the present invention does not specifically limit this.

[0133] After obtaining the second sample text, the second sample text may be input into the initial recognition model to obtain a predicted error sentence output by the initial recognition model.

[0134] Furthermore, based on the difference between the predicted error sentences and the label error sentences, a second loss is determined, and the parameters of the initial recognition model are iterated based on the second loss, and the initial recognition model after the parameter iteration is completed is used as the recognition model.

[0135] It can be understood that the greater the difference between the predicted error sentence and the label error sentence, the greater the second loss; the smaller the difference between the predicted error sentence and the label error sentence, the smaller the second loss.

[0136] Here, the loss function of the first loss may be a cross entropy loss function (Cross Entropy Loss Function) or a mean square error loss function (MSE), etc., which is not specifically limited in the embodiment of the present invention.

[0137] Here, the loss function of the second loss may be a cross entropy loss function or a mean square error loss function, etc., which is not specifically limited in the embodiment of the present invention.

[0138] It should be noted that the initial model has undergone two stages of training, so that the recognition model obtained based on the initial model training has the ability to identify typos and incorrect sentences, which improves the accuracy and reliability of the recognition model's typo recognition and incorrect sentence recognition.

[0139] The confidence assessment device provided by the present invention is described below. The confidence assessment device described below and the confidence assessment method described above can be referred to in correspondence with each other.

[0140] Based on any of the above embodiments, the present invention provides a confidence assessment device, Figure 11 Schematic diagram of the structure of the confidence assessment device provided by the present invention, such as Figure 11 As shown, the device includes: The acquisition unit 1110 is configured to acquire a query instruction and a search result corresponding to the query instruction, and determine a query feature corresponding to the query instruction, a query feature corresponding to the query feature, a search result feature corresponding to the search result, and a search result feature corresponding to the search result feature; a difference determining unit 1120 configured to determine a semantic difference feature based on a semantic difference between the query instruction and the search result, and to determine a textual difference feature based on a textual difference between the query instruction and the search result; The confidence score determination unit 1130 is configured to perform confidence prediction based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0141] The device provided by the embodiment of the present invention first determines the query features corresponding to the query instruction to accurately capture the key information and semantic features of the query instruction; secondly, determines the search result features corresponding to the search results to comprehensively characterize the core points and text characteristics of the document; thirdly, based on the semantic differences between the query instruction and the search results, determines the semantic difference features, and based on the text differences between the query instruction and the search results, determines the text difference features, and deeply explores the intrinsic connection between the two at the semantic and text levels. Since the query features, search result features, semantic difference features and text difference features can comprehensively reflect the matching degree and quality information between the query instruction and the search results, confidence prediction is performed based on the query features, search result features, semantic difference features and text difference features to obtain a confidence score result, which can efficiently evaluate the performance of the RAG retrieval module, get rid of the dependence on external reference data and complex large language models, and greatly improve the practicality and flexibility of confidence evaluation.

[0142] Based on any of the above embodiments, the method further includes a search result feature determination unit, wherein the search result feature determination unit is specifically configured to: Performing text quality evaluation on the search results to obtain a text quality score vector; Performing a timeliness evaluation on the search results to obtain a timeliness score vector; Performing an authority evaluation on the search results to obtain an authority score vector; Performing a topic relevance evaluation on the search results to obtain a topic relevance score vector; The search result feature is determined based on the text quality score vector, the timeliness score vector, the authority score vector, and the topic relevance score vector.

[0143] Based on any of the above embodiments, the difference determining unit 1120 is specifically configured to: Performing word segmentation processing on the query instruction and the search result respectively to obtain a query instruction word segmentation result and a search word segmentation result; Perform keyword recognition on the query instruction word segmentation result to obtain query keywords; The frequency and frequency of the query keyword in the search word segmentation results are determined, and the text difference feature is determined based on the frequency and the frequency.

[0144] Based on any of the above embodiments, the difference determining unit 1120 specifically includes: a semantic similarity determination unit, configured to encode the query instruction and the search result respectively to obtain a first feature and a second feature, and determine the semantic similarity between the first feature and the second feature; a text pair similarity determination unit, configured to input the query instruction and the search result into an encoder for feature interaction, obtain basic features output by the encoder, and determine text pair similarity corresponding to a text pair constructed by the query instruction and the search result based on the basic features; The semantic difference feature determination unit is configured to determine the semantic difference feature based on the semantic similarity and the text pair similarity.

[0145] Based on any of the above embodiments, the unit for determining text pair similarity is specifically configured to: The query instruction and the search result are spliced and input into the encoder for lexical interaction, syntactic interaction and semantic interaction respectively, so as to obtain lexical features, syntactic features and semantic features output by the encoder.

[0146] Based on any of the above embodiments, the system further includes a query feature determination unit, wherein the query feature determination unit is specifically configured to: Performing dependency syntax analysis on the query instruction to determine a dependency relationship type in the query instruction and a relationship type frequency corresponding to the dependency relationship type; Determining dependency syntactic features based on the dependency relationship type and the relationship type frequency; Based on the recognition model, identifying typos and incorrect sentences in the query instruction, and determining text quality features based on the typos and incorrect sentences; The query feature is determined based on the dependency syntactic feature and the text quality feature.

[0147] Based on any of the above embodiments, the further comprising a training unit, wherein the training unit is specifically configured to: Obtaining a first sample text, a labeled typo text of the first sample text, and an initial model; Inputting the first sample text into the initial model to obtain a predicted error text output by the initial model; Determining a first loss based on a difference between the predicted error text and the labeled typo text, and iterating parameters of the initial model based on the first loss to obtain an initial recognition model; Obtaining a second sample text and labeled incorrect sentences of the second sample text; Inputting the second sample text into the initial recognition model to obtain a predicted error sentence output by the initial recognition model; Based on the difference between the predicted error sentence and the label error sentence, a second loss is determined, and based on the second loss, parameters of the initial recognition model are iterated to obtain the recognition model.

[0148] Figure 12 Schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 may call logic instructions in the memory 1230 to execute a confidence assessment method, which includes: obtaining a query instruction and a search result corresponding to the query instruction, and determining a query feature corresponding to the query instruction and a search result feature corresponding to the search result; determining a semantic difference feature based on a semantic difference between the query instruction and the search result, and determining a text difference feature based on a text difference between the query instruction and the search result; and performing a confidence prediction based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

[0149] Furthermore, the logic instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product, stored in a storage medium, includes instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a mobile hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0150] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the confidence assessment method provided by the above methods, which includes: obtaining a query instruction and a search result corresponding to the query instruction, and determining a query feature corresponding to the query instruction and a search result feature corresponding to the search result; determining a semantic difference feature based on the semantic difference between the query instruction and the search result, and determining a text difference feature based on the text difference between the query instruction and the search result; performing confidence prediction based on the query feature, the search result feature, the semantic difference feature and the text difference feature to obtain a confidence score result.

[0151] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the confidence assessment method provided by the above-mentioned methods, the method comprising: obtaining a query instruction and a search result corresponding to the query instruction, and determining a query feature corresponding to the query instruction and a search result feature corresponding to the search result; determining a semantic difference feature based on the semantic difference between the query instruction and the search result, and determining a text difference feature based on the text difference between the query instruction and the search result; performing confidence prediction based on the query feature, the search result feature, the semantic difference feature and the text difference feature to obtain a confidence score result.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0153] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0154] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A confidence assessment method, characterized in that: include: Obtaining a query instruction and a search result corresponding to the query instruction, and determining a query feature corresponding to the query instruction and a search result feature corresponding to the search result; Determining a semantic difference feature based on a semantic difference between the query instruction and the search result, and determining a text difference feature based on a text difference between the query instruction and the search result; A confidence prediction is performed based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

2. The confidence assessment method according to claim 1, wherein: The step of determining the search result feature corresponding to the search result includes: Performing text quality evaluation on the search results to obtain a text quality score vector; Performing a timeliness evaluation on the search results to obtain a timeliness score vector; Performing an authority evaluation on the search results to obtain an authority score vector; Performing a topic relevance evaluation on the search results to obtain a topic relevance score vector; The search result feature is determined based on the text quality score vector, the timeliness score vector, the authority score vector, and the topic relevance score vector.

3. The confidence assessment method according to claim 1, wherein: The determining of text difference features based on the text difference between the query instruction and the search result includes: Performing word segmentation processing on the query instruction and the search result respectively to obtain a query instruction word segmentation result and a search word segmentation result; Perform keyword recognition on the query instruction word segmentation result to obtain query keywords; The frequency and frequency of the query keyword in the search word segmentation results are determined, and the text difference feature is determined based on the frequency and the frequency.

4. The confidence assessment method according to any one of claims 1 to 3, characterized in that: The determining of a semantic difference feature based on the semantic difference between the query instruction and the search result includes: Encoding the query instruction and the search result respectively to obtain a first feature and a second feature, and determining a semantic similarity between the first feature and the second feature; Inputting the query instruction and the search result into an encoder for feature interaction, obtaining basic features output by the encoder, and determining text pair similarity corresponding to text pairs constructed by the query instruction and the search result based on the basic features; The semantic difference feature is determined based on the semantic similarity and the text pair similarity.

5. The confidence assessment method according to claim 4, wherein: Inputting the query instruction and the search result into an encoder for feature interaction to obtain basic features output by the encoder includes: The query instruction and the search result are spliced and input into the encoder for lexical interaction, syntactic interaction and semantic interaction respectively, so as to obtain lexical features, syntactic features and semantic features output by the encoder.

6. The confidence assessment method according to any one of claims 1 to 3, characterized in that: The step of determining the query feature corresponding to the query instruction includes: Performing dependency syntax analysis on the query instruction to determine a dependency relationship type in the query instruction and a relationship type frequency corresponding to the dependency relationship type; Determining dependency syntactic features based on the dependency relationship type and the relationship type frequency; Based on the recognition model, identifying typos and incorrect sentences in the query instruction, and determining text quality features based on the typos and incorrect sentences; The query feature is determined based on the dependency syntactic feature and the text quality feature.

7. The confidence assessment method according to claim 6, characterized in that: The training step of the recognition model includes: Obtaining a first sample text, a labeled typo text of the first sample text, and an initial model; Inputting the first sample text into the initial model to obtain a predicted error text output by the initial model; Determining a first loss based on a difference between the predicted error text and the labeled typo text, and iterating parameters of the initial model based on the first loss to obtain an initial recognition model; Obtaining a second sample text and labeled incorrect sentences of the second sample text; Inputting the second sample text into the initial recognition model to obtain a predicted error sentence output by the initial recognition model; Based on the difference between the predicted error sentence and the label error sentence, a second loss is determined, and based on the second loss, parameters of the initial recognition model are iterated to obtain the recognition model.

8. A confidence assessment device, characterized in that: include: an acquiring unit, configured to acquire a query instruction and a search result corresponding to the query instruction, and determine a query feature corresponding to the query instruction and a search result feature corresponding to the search result; a difference determining unit, configured to determine a semantic difference feature based on a semantic difference between the query instruction and the search result, and to determine a text difference feature based on a text difference between the query instruction and the search result; A confidence score determination unit is configured to perform confidence prediction based on the query feature, the search result feature, the semantic difference feature, and the text difference feature to obtain a confidence score result.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the confidence assessment method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the confidence assessment method according to any one of claims 1 to 7 is implemented.

Images