Credibility evaluation method based on large model knowledge generation

Abstract

The invention discloses a credibility evaluation method based on large model knowledge generation, and relates to the field of large model application and knowledge mapping, and the method comprises the steps: firstly, generating a plurality of natural language expression forms of a triple by using a large model; searching related literatures by means of a search engine, evaluating semantic similarity of the literatures and triads and credibility of source websites, and selecting high-quality related evidences; thirdly, credibility evaluation is carried out on dimensions such as the literature support degree, consistency and reliability of the triad and revenue evidence through a large model; and finally, fusing scores of multiple dimensions by using a multi-layer perceptron model, and generating a final confidence coefficient of each triple. The method can effectively combine the support degree of the literature, the reliability of the source and the potential reverse evidence, improves the credibility judgment accuracy of the knowledge, and is suitable for automatic credibility evaluation and verification of a large-scale knowledge base.

Description

Technical Field

[0001] The present invention relates to the field of large model applications and knowledge graphs, and in particular to a credibility assessment method based on large model knowledge generation. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and may not constitute prior art.

[0003] With the rapid development of artificial intelligence and large language model technology, knowledge generation systems have been widely used in many fields, including decision support, information verification, intelligent question answering, etc. However, the credibility assessment of current knowledge generation systems still faces many challenges. Traditional credibility assessment methods mostly rely on manual annotation or simple rule-based algorithms. These methods cannot effectively process complex multi-dimensional information, especially when faced with large-scale and diverse text data. Their reliability and efficiency often cannot meet actual needs.

[0004] The credibility assessment methods in the prior art usually rely on the following means: First, the method based on document retrieval mainly evaluates the credibility of triples through keyword matching or simple text similarity calculation, but this method often ignores the contextual information, semantic relationships and diverse expressions of the text, resulting in uneven quality of the recalled related documents. Secondly, although the evaluation methods based on statistical models or machine learning models can automate the evaluation process to a certain extent, they still have the following limitations: First, they rely on relatively simple features and fail to fully explore the semantic depth and complexity of the text; second, they lack effective processing of rebuttal evidence and fail to fully consider potential errors or deviations in knowledge generation; third, they cannot effectively integrate information from different dimensions, resulting in insufficient accuracy and credibility of the evaluation results.

[0005] Therefore, when dealing with complex and diversified information, existing credibility assessment methods have defects such as insufficient assessment dimensions, inaccurate assessment results, and inadequate handling of rebuttal evidence. There is an urgent need for a more comprehensive, precise, and automated technology to improve the reliability and accuracy of the assessment. Summary of the invention

[0006] The purpose of the present invention is to provide a credibility assessment method based on large model knowledge generation to address the defects of existing methods. By combining a multi-dimensional assessment strategy, the credibility of generated knowledge can be judged more comprehensively and accurately. Specifically, the present invention significantly improves the assessment accuracy through the following innovations:

[0007] Multi-expression enhancement: Generates multiple natural language expressions of triples through a large model, which enhances the comprehensiveness of knowledge representation and helps improve retrieval accuracy.

[0008] Comprehensive document scoring: This method combines semantic similarity analysis with website credibility assessment, taking into account multiple dimensions such as document support, website authority, and text quality, thus avoiding evaluation bias caused by a single factor.

[0009] Refutation evidence evaluation: A new evaluation dimension for refutation evidence can effectively identify information that contradicts the triple content and further improve the reliability of credibility assessment.

[0010] Multi-layer perceptron fusion scoring: By training a multi-layer perceptron model, the scores of multiple dimensions are fused to generate the final confidence of each triplet, ensuring the accuracy and stability of the evaluation results.

[0011] The technical solution of the present invention is as follows:

[0012] A credibility assessment method based on large model knowledge generation, comprising:

[0013] Step S1: Generate multi-expression event sentences for triples using the large model;

[0014] Step S2: Retrieve relevant proof documents for the event sentence using a search engine, and sort the retrieved relevant proof documents using a scoring module, retaining the N relevant proof documents with the highest scores;

[0015] Step S3: Input the retrieved proof documents and triples into the big model, and let the big model score the credibility of the triples from multiple dimensions in the form of thought chain;

[0016] Step S4: Train a multi-layer perceptron to fuse the scores of multiple dimensions and generate the final confidence of each triple.

[0017] Furthermore, the step S1 comprises:

[0018] Step S11: construct a large model to generate prompts of various expressions;

[0019] Step S12: Use prompt to convert the triples into event sentences expressed in different natural languages ​​using the pre-trained large model Qwen-7B-Chat; each triple (S, P, O) will generate a set ε = {e1, e2, ..., e n}, where each e i Each is a possible expression of (S,P,O).

[0020] Furthermore, the step S2 comprises:

[0021] Step S21: Use each event sentence e i∈ε as the query condition, use the search engine to retrieve relevant proof documents, and return the document set D = {d1, d2, ..., d m};

[0022] Step S22: Calculate the value of each document d in the document set D. j ∈D and event sentence e i The semantic similarity score S between semantic (d j ,e i );

[0023] Step S23: scoring the credibility of the website to which the recalled document belongs through multiple dimensions;

[0024] Step S24: Combine the semantic similarity score and the website credibility score to obtain the final score S for each document. total (d j ,e i );

[0025] Step S25: According to S total (d j ,e i ) values ​​to sort all retrieved documents and select the top N documents with the highest scores

[0026] Furthermore, the semantic similarity score S semantic (d j ,e i ) is calculated by the Sentence-BERT model.

[0027] Furthermore, the multiple dimensions in step S23 include:

[0028] Domain Authority Score D domain and text quality score T text ;

[0029] Furthermore, the step S23 includes:

[0030] Step S231: Construct a domain name and its score correspondence table to obtain the domain name authority score D domain ;

[0031] Step S232: Use the large model to score the quality of the target document in multiple dimensions, and use the average value as the text quality score T text ;

[0032] Step S233: Based on the domain name authority score D domain and text quality score T text , find the weighted average of the two, which is the website credibility score.

[0033] Furthermore, the step S232 includes:

[0034] The target documents are scored based on three dimensions: grammar and language standardization, content depth and information comprehensiveness, and logic and consistency.

[0035] Furthermore, the step S3 comprises:

[0036] The big model scores the credibility of the triples in the form of a chain of thought from three dimensions: literature support, consistency and reliability, and rebuttal evidence.

[0037] Furthermore, the step S4 comprises:

[0038] Step S41: using the scores of multiple dimensions as input features to construct a training data set;

[0039] Step S42: Train a multi-layer perceptron model based on the training data set, whose input is the features of each dimension and output is the final confidence C of the triplet final ;

[0040] Step S43: The multi-layer perceptron model is trained using mean square error loss, with the goal of minimizing the error between the confidence of the model prediction and the true label.

[0041] Step S44: Use the completed model to evaluate the credibility of the triplet.

[0042] Furthermore, N=5.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] 1. Compared with the prior art, the present invention not only improves the comprehensiveness and accuracy of the evaluation, but also can realize automatic and rapid credibility evaluation in a large-scale knowledge base, and has strong practicality and promotion value.

[0045] 2. This invention uses a multi-dimensional comprehensive evaluation of the credibility of knowledge generated by the large model, combined with factors such as literature support, website authority, text quality, and refutation evidence, to effectively improve the accuracy and comprehensiveness of credibility assessment. By generating multiple natural language expressions, automating the evaluation process, and integrating the scoring of the multi-layer perceptron model, this technology significantly improves the evaluation efficiency, reduces manual intervention, and has broad application prospects, suitable for knowledge base construction, intelligent question answering, and decision support.

[0046] 3. This invention uses a multi-dimensional comprehensive evaluation of the credibility of knowledge generated by the large model, combined with factors such as literature support, website authority, text quality and rebuttal evidence, to effectively improve the accuracy and comprehensiveness of credibility assessment. By generating multiple natural language expressions, automating the evaluation process and fusing scores with a multi-layer perceptron model, this technology significantly improves evaluation efficiency, reduces manual intervention, and has broad application prospects, suitable for knowledge base construction, intelligent question answering, decision support and other fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a flow chart of a credibility assessment method based on large model knowledge generation;

[0048] Figure 2 Schematic diagram of the overall framework of knowledge credibility assessment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0050] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0051] Embodiment 1

[0052] See also Figure 1 , a credibility assessment method based on large model knowledge generation, specifically comprising the following steps:

[0053] Step S1: Generate multi-expression event sentences for triples using the large model; this step is done by Figure 2 The large model event sentence generation module is implemented in;

[0054] Step S2: Use the search engine to retrieve relevant proof documents for the event sentence, and use the scoring module to sort the retrieved relevant proof documents, and retain the 5 relevant proof documents with the highest scores; this step is achieved by Figure 2 The semantic similarity evaluation module and website authority evaluation module in are implemented;

[0055] Step S3: Input the retrieved proof documents and triples into the big model, and let the big model score the credibility of the triples from multiple dimensions in the form of thought chain; this step is done by Figure 2 The large model multi-dimensional confidence scoring module is implemented in;

[0056] Step S4: Train a multi-layer perceptron to fuse the scores of multiple dimensions and generate the final confidence of each triplet; this step is done by Figure 2 The multi-dimensional score fusion network is implemented in ;

[0057] In this embodiment, specifically, step S1 includes:

[0058] Step S11: construct a large model to generate prompts of various expressions;

[0059] Step S12: Use prompt to convert the triples into event sentences expressed in different natural languages ​​using the pre-trained large model Qwen-7B-Chat; each triple (S, P, O) will generate a set ε = {e1, e2, ..., e n}, where each e i are all possible expressions of (S, P, O); in this way, the comprehensive representation of knowledge is enhanced and the accuracy of recalled documents is enhanced.

[0060] In this embodiment, specifically, step S2 includes:

[0061] Step S21: Use each event sentence e i ∈ε as the query condition, use the Bing search engine to retrieve relevant proof documents, and return the document set D = {d1, d2, ..., d m}; The content of these documents concerns possible supporting evidence for the event sentences;

[0062] Step S22: Calculate the value of each document d in the document set D. j ∈D and event sentence e i The semantic similarity score S between semantic (d j ,e i ); Optionally, use the Sentence-BERT model to calculate d for each document j ∈D and event sentence e i The semantic similarity score S between semantic (d j ,e i ):

[0063] S semantic (d j ,e i )=cosine similarity(embedding(d j ),embedding(e i ))

[0064] This score reflects the degree of semantic match between the document and the event sentence;

[0065] Step S23: scoring the credibility of the website to which the recalled document belongs through multiple dimensions;

[0066] Step S24: Combine the semantic similarity score and the website credibility score to obtain the final score S for each document. total (d j ,e i ); optionally, these two factors can be combined in a weighted average:

[0067] S total (d j ,e i )=α·S semantic (d j ,e i )+(1-α)·C website (d j )

[0068] Among them, α∈[0,1] is a weight parameter used to control the relative importance of semantic similarity and website credibility. This parameter is obtained through training;

[0069] Step S25: According to S total (d j ,e i ) values ​​to sort all retrieved documents and select the top N documents with the highest scores That is, according to S total (d j ,e i ) values ​​to sort all retrieved documents and select the top 5 documents with the highest scores:

[0070] D top5 ={d top1 ,d top2 ,d top3 ,d top4 ,d top5}

[0071] These five documents will serve as the most reliable evidence documents related to the event sentence.

[0072] In this embodiment, specifically, the multiple dimensions in step S23 include:

[0073] Domain Authority Score D domain and text quality score T text;

[0074] Among them, the domain name authority score D domain :edu, gov and other educational and government domain names are usually more credible than other domain names; build a table of domain names and their scores to obtain the authority score of the website;

[0075] Text quality score T text :Use the big model to score the quality of the target document in multiple dimensions; score the document from three dimensions: grammar and language standardization, content depth and information comprehensiveness, and logic and consistency, and use the average value as the text quality score.

[0076] That is, in this embodiment, specifically, step S23 includes:

[0077] Step S231: Construct a domain name and its score correspondence table to obtain the domain name authority score D domain ;

[0078] Step S232: Use the large model to score the quality of the target document in multiple dimensions, and use the average value as the text quality score T text ;

[0079] Step S233: Based on the domain name authority score D domain and text quality score T text , find the weighted average of the two, which is the website credibility score.

[0080] C website =w1×D domain +w2×T text

[0081] In this embodiment, specifically, step S3 includes:

[0082] The big model scores the credibility of the triples in the form of thought chains from three dimensions: literature support, consistency and reliability, and rebuttal evidence;

[0083] Specifically, document support: This dimension assesses whether the triple has significant support from the retrieved documents. Support can be direct (explicitly stating the fact of the triple) or indirect (inferring the credibility of the triple through the context);

[0084] Consistency and Reliability: This dimension assesses the internal consistency of the document content and the reliability of the source. It focuses on whether the document has internal contradictions or whether it comes from a credible and authoritative source;

[0085] Refuting evidence: This dimension evaluates whether there is evidence in the document that contradicts the content of the triple. Considering the differences of opinion between different sources, evaluating the refuting evidence will help judge the credibility of the triple.

[0086] In this embodiment, it should also be noted that the prompt for building a large model credibility assessment is as follows:

[0087] Evaluate the credibility of the following triples. Please evaluate the credibility of the following triples based on the provided literature {d top1 ,d top2 ,d top3 ,d top4 ,d top5} and the website source to which it belongs, and give the reasoning process of each evaluation dimension and the final score of the dimension. The score ranges from 0 (completely unreliable) to 1 (completely credible). Scoring is required based on three dimensions: literature support, consistency and reliability, and rebuttal evidence.

[0088] The triple is: Subject: {S} Relationship: {P} Object: {O}

[0089] Please follow the steps below to perform reasoning:

[0090] (1) Document support: Please check whether the retrieved documents clearly support the triple. Whether the evidence document provides detailed support or evidence for the triple, whether it is direct support (explicitly mentioning the triple content) or indirect support (through contextual reasoning). Evaluate the support level of the evidence document based on the above scoring criteria and give a score.

[0091] (2) Consistency and reliability: Please analyze whether the content of the evidence document is consistent. If multiple documents support the content of the triple, the credibility is high. Consider the source of the evidence document (such as official documents, academic research, etc.) and the consistency of the content between the evidence documents. Give a score for consistency and reliability.

[0092] (3) Rebuttal evidence: Please check whether there is any evidence in the evidence document that refutes the triple. If so, check whether it comes from an authoritative website source. Based on the above criteria, give a score for the rebuttal evidence.

[0093] The results are given in the following format:

[0094] {

[0095] "Literature support evaluation":

[0096] “Rating Basis”:

[0097] "Fraction":

[0098] },

[0099] "Consistency and reliability assessment":

[0100] “Rating Basis”:

[0101] "Fraction":

[0102] }

[0103] ,

[0104] "Rebuttal Evidence Assessment":{

[0105] “Rating Basis”:

[0106] "Fraction":

[0107] }

[0108] }

[0109] In this embodiment, specifically, step S4 includes:

[0110] Step S41: Use the scores of multiple dimensions as input features to construct a training data set; it should be noted that the features corresponding to each triple (S, P, O) include literature support, consistency and reliability, and rebuttal evidence scores;

[0111] Step S42: Train a multi-layer perceptron model based on the training data set, whose input is the features of each dimension and output is the final confidence C of the triplet final ; It should be noted that the training data set is: {(X1,Y1),(X2,Y2),...,(X n ,Y n )};

[0112] X i =[Document support score, consistency and reliability score, rebuttal evidence score], Y i is the corresponding true confidence label. Based on this, a multi-layer perceptron model is trained, whose input is the features of each dimension and output is the final confidence C of the triplet final ;

[0113] Step S43: The multi-layer perceptron model is trained using mean square error loss, with the goal of minimizing the error between the confidence predicted by the model and the true label;

[0114]

[0115] Step S44: Use the completed model to evaluate the credibility of the triplet.

[0116] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

[0117] This background section is provided to generally present the context of the invention, and the work of the presently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither explicitly nor implicitly admitted to be prior art to the present invention.

Claims

1. A credibility assessment method based on large model knowledge generation, characterized in that: include: Step S1: Generate multi-expression event sentences for triples using the large model; Step S2: Retrieve relevant proof documents for the event sentence using a search engine, and sort the retrieved relevant proof documents using a scoring module, retaining the N relevant proof documents with the highest scores; Step S3: Input the retrieved proof documents and triples into the big model, and let the big model score the credibility of the triples from multiple dimensions in the form of thought chain; Step S4: Train a multi-layer perceptron to fuse the scores of multiple dimensions and generate the final confidence of each triple.

2. The credibility assessment method based on large model knowledge generation according to claim 1 is characterized in that: The step S1 comprises: Step S11: construct a large model to generate prompts of various expressions; Step S12: Use prompt to convert the triples into event sentences expressed in different natural languages ​​using the pre-trained large model Qwen-7B-Chat; each triple (S, P, O) will generate a set ε = {e1, e2, ..., e n }, where each e i Each is a possible expression of (S,P,O).

3. The credibility assessment method based on large model knowledge generation according to claim 2 is characterized in that: The step S2 comprises: Step S21: Use each event sentence e i ∈ε as the query condition, use the search engine to retrieve relevant proof documents, and return the document set D = {d1, d2, ..., d m }; Step S22: Calculate the value of each document d in the document set D. j ∈D and event sentence e i The semantic similarity score S between semantic (d j ,e i ); Step S23: scoring the credibility of the website to which the recalled document belongs through multiple dimensions; Step S24: Combine the semantic similarity score and the website credibility score to obtain the final score S for each document. total (d j ,e i ); Step S25: According to S total (d j ,e i ) values ​​to sort all retrieved documents and select the top N documents with the highest scores 4. The credibility assessment method based on large model knowledge generation according to claim 3 is characterized in that: The semantic similarity score S semantic (d j ,e i ) is calculated by the Sentence-BERT model.

5. The credibility assessment method based on large model knowledge generation according to claim 4 is characterized in that: The multiple dimensions in step S23 include: Domain Authority Score D domain and text quality score T text .

6. The credibility assessment method based on large model knowledge generation according to claim 5 is characterized in that: The step S23 comprises: Step S231: Construct a domain name and its score correspondence table to obtain the domain name authority score D domain ; Step S232: Use the large model to score the quality of the target document in multiple dimensions, and use the average value as the text quality score T text ; Step S233: Based on the domain name authority score D domain and the text quality score T text , find the weighted average of the two, which is the website credibility score.

7. The credibility assessment method based on large model knowledge generation according to claim 6 is characterized in that: The step S232 includes: The target documents are scored based on three dimensions: grammar and language standardization, content depth and information comprehensiveness, and logic and consistency.

8. The credibility assessment method based on large model knowledge generation according to claim 6 is characterized in that: The step S3 comprises: The big model scores the credibility of the triples in the form of a chain of thought from three dimensions: literature support, consistency and reliability, and rebuttal evidence.

9. The credibility assessment method based on large model knowledge generation according to claim 8 is characterized in that: The step S4 comprises: Step S41: using the scores of multiple dimensions as input features to construct a training data set; Step S42: Train a multi-layer perceptron model based on the training data set, whose input is the features of each dimension and output is the final confidence C of the triplet final ; Step S43: The multi-layer perceptron model is trained using mean square error loss, with the goal of minimizing the error between the confidence of the model prediction and the true label. Step S44: Use the completed model to evaluate the credibility of the triplet.

10. The credibility assessment method based on large model knowledge generation according to claim 1 is characterized in that: N＝5。

Images