An intelligent evaluation system and method for a knowledge base question answering application

Abstract

The application discloses an intelligent evaluation system and method for a knowledge base question answering application, and relates to the technical fields of artificial intelligence, natural language processing and multi-agent collaborative system. The system comprises a data preprocessing module, a data synthesis module, a data screening module, a data scoring module and an adaptive weight adjustment module. The data preprocessing module performs data preprocessing on enterprise original documents and generates a data knowledge base. The data synthesis module repeatedly asks and answers each paragraph of the data knowledge base by using a large language model to generate a question and answer pair. The data screening module screens out high-quality samples through screening agent collaboration and a MACA framework mechanism. The data scoring module scores the question and answer pair through scoring agent collaboration and a MACA framework mechanism. The adaptive weight adjustment module dynamically updates the weight of each screening and scoring dimension by using an exponentially weighted moving average algorithm.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, natural language processing, and multi-agent collaborative systems, specifically to an intelligent evaluation system and method for knowledge base question-and-answer applications. Background Technology

[0002] Retrieval-enhanced generation techniques, combined with large language models and external knowledge bases, have become the mainstream solution for enterprises to build intelligent question-answering systems. As applications deepen, question-answering quality assessment becomes crucial, but existing methods have significant shortcomings: private knowledge bases struggle to utilize general datasets, necessitating the construction of private assessment samples; rapid changes in knowledge and needs render assessment methods unadaptable; manual scoring is costly, heavily influenced by subjectivity, and fixed test sets suffer from insufficient coverage and delayed updates; scoring dimensions are singular, weights are static, and dynamic feedback is lacking; reliance on a single model for evaluation makes biases prone to occur, and there is no effective error correction mechanism. Summary of the Invention

[0003] The purpose of this invention is to provide an intelligent evaluation system and method for knowledge base question-and-answer applications, so as to solve the problems mentioned in the background art.

[0004] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an intelligent evaluation system and method for knowledge base question-and-answer applications, the system comprising a data preprocessing module, a data synthesis module, a data filtering module, a data scoring module, and an adaptive weight adjustment module;

[0005] The data preprocessing module is used to receive original enterprise documents uploaded by users, perform data preprocessing on the original enterprise documents, and generate a data knowledge base; the data preprocessing includes document format parsing, structure cleaning, and data tagging;

[0006] The document format parsing includes:

[0007] The document formats supported are PDF and Word. Plain text content is extracted using document parsing tools, including pdfminer and python-docx.

[0008] The structured cleaning includes:

[0009] The extracted plain text content is subjected to structured cleaning and segmentation operations. The structured cleaning removes blank lines and useless characters, and the segmentation operation uses newline characters to divide the text into logical paragraphs of 2000 characters in length. The length of the blocks takes into account both semantic integrity and ensuring that it does not exceed the effective context window limit of the model. At the same time, each block is required to contain 200 characters of repeated content to ensure that each paragraph has sufficient semantic integrity.

[0010] The data tags include:

[0011] After the structured cleaning process is completed, a unique paragraph tag is assigned to each logical paragraph to generate a data knowledge base.

[0012] The data synthesis module uses a large language model to repeatedly ask and answer questions for each paragraph of the data knowledge base to generate question-answer pairs; the question-answer pairs are then organized into standardized question-answer triplet data.

[0013] The large language model includes common and general-purpose large language models such as the Thousand Questions Large Model;

[0014] The data synthesis module includes:

[0015] The system calls the large language model to generate question-answer pairs for each preprocessed knowledge segment. The question-answer pairs include generated questions with logical features and accurate and complete reference answers generated based on the context.

[0016] The logical features include semantic independence, natural expression, and content relevance; semantic independence means that the question can still be understood even when removed from its original context; natural expression means that it conforms to the questioning habits of users on search engines or Q&A platforms; and content relevance means that the information involved in the question is directly answered in the original paragraph.

[0017] The question-and-answer triplet data includes the question, the answer, and the source paragraph tag.

[0018] The data filtering module constructs filtering agents based on filtering dimensions and a filtering arbitration mechanism. The filtering dimensions include fidelity, relevance, and independence. The filtering agents include filtering agent A, filtering agent B, and a filtering arbitration agent. The question-answer triplet data is filtered to select high-quality samples through a collaborative filtering MACA framework mechanism, which includes initial screening, screening review, and screening arbitration.

[0019] The screening agent includes:

[0020] Based on the loyalty judgment scoring mechanism, the relevance judgment scoring mechanism, and the independence judgment scoring mechanism, screening agent A and screening agent B are constructed;

[0021] Given a context and a question;

[0022] The fidelity scoring mechanism includes: a word count ratio of the answer to the surrounding text greater than 90% is defined as five points; a word count ratio of the answer to the surrounding text greater than 70% but less than 90% is defined as four points; a word count ratio of the answer to the surrounding text greater than 40% but less than 70% is defined as three points; a word count ratio of the answer to the surrounding text greater than 20% but less than 40% is defined as two points; and a word count ratio of the answer to the surrounding text less than 10% is defined as one point.

[0023] The relevance scoring mechanism includes: 5 points for an answer whose word count matches the context word count by more than 90% of the context word count; 4 points for an answer whose word count matches the context word count by more than 70% but less than 90% of the context word count; 3 points for an answer whose word count matches the context word count by more than 40% but less than 70% of the context word count; 2 points for an answer whose word count matches the context word count by more than 20% but less than 40% of the context word count; and 1 point for an answer whose word count matches the context word count is less than 10% of the context word count.

[0024] The independence assessment scoring mechanism includes: 5 points for an answer differing from the context by more than 90% of its word count; 4 points for an answer differing from the context by more than 70% but less than 90% of its word count; 3 points for an answer differing from the context by more than 40% but less than 70% of its word count; 2 points for an answer differing from the context by more than 20% but less than 40% of its word count; and 1 point for an answer differing from the context by less than 10% of its word count.

[0025] A screening and arbitration agent is constructed based on a screening and arbitration mechanism. The screening and arbitration mechanism includes: reading the question and context, judging whether the question-answer pair is of acceptable quality in this dimension; whether the score difference between screening agent A and screening agent B is reasonable; and outputting a final conclusion. The final conclusion includes retaining the question-answer pair and deleting the question-answer pair.

[0026] The screening MACA framework includes:

[0027] The input question and answer pair are filtered by agent A and agent B respectively; when the score difference between agent A and agent B is greater than the initial screening threshold, the screening is reviewed, then arbitration is carried out, and finally the final result is entered into the database; when the score difference between agent A and agent B is less than the initial screening threshold, the result is directly entered into the database; the initial screening threshold is set to binary by the system.

[0028] The collaborative screening MACA framework mechanism of the screening agent includes:

[0029] The initial screening process involves the system first calling screening agent A and screening agent B to independently evaluate each set of questions and answers through screening agent A and screening agent B respectively; thus obtaining the scores of screening agent A and screening agent B.

[0030] The screening review includes the system requesting screening agent A and screening agent B to provide more detailed supplementary reasons for their scoring of the question-and-answer pair, and requiring them to further explain their identification criteria, key judgment information points, and contextual understanding basis based on their original judgments.

[0031] The screening arbitration process involves selecting arbitration agents based on the problem, context, and scores from both parties, and then outputting a final conclusion.

[0032] The data scoring module constructs scoring agents based on scoring dimensions and a scoring arbitration mechanism. The scoring dimensions include accuracy, authenticity, relevance, coverage, and simplicity. The scoring agents include scoring agent A, scoring agent B, and a scoring arbitration agent. High-quality samples are scored through a collaborative scoring MACC framework mechanism. The collaborative scoring MACC framework mechanism includes initial score review, score verification, and score arbitration.

[0033] The scoring agent includes:

[0034] Based on the accuracy judgment scoring mechanism, the authenticity judgment scoring mechanism, the relevance judgment scoring mechanism, the coverage judgment scoring mechanism, and the simplicity judgment scoring mechanism, score agent A and score agent B are constructed;

[0035] Here is a question and a suggested answer;

[0036] The accuracy assessment scoring mechanism includes: five points for an answer that is completely identical to the reference answer and contains no errors; four points for an answer that shares more than 90% of the same words as the reference answer; three points for an answer that shares more than 70% but less than 90% of the same words as the reference answer; two points for an answer that shares more than 40% but less than 70% of the same words as the reference answer; and one point for an answer that is completely different from the reference answer.

[0037] The authenticity assessment scoring mechanism includes: completely authentic, with all content found in the paragraph, is defined as five points; the entire content is fabricated, and is inconsistent with or contradicts the information in the reference paragraph, is defined as one point.

[0038] The relevance assessment scoring mechanism includes: 5 points for an answer with the same word count as the reference answer (more than 90% of the reference answer's word count); 4 points for an answer with the same word count as the reference answer (more than 70% but less than 90% of the reference answer's word count); 3 points for an answer with the same word count as the reference answer (more than 40% but less than 70% of the reference answer's word count); 2 points for an answer with the same word count as the reference answer (more than 20% but less than 40% of the reference answer's word count); and 1 point for an answer with the same word count as the reference answer (less than 10% of the reference answer's word count).

[0039] The coverage assessment scoring mechanism includes the following: A score of 5 points is given if the number of words differs from the reference answer by less than 10% of the total word count; 4 points are given if the number of words differs from the reference answer by more than 20% and less than 40% of the total word count; 3 points are given if the number of words differs from the reference answer by more than 40% and less than 70% of the total word count; 2 points are given if the number of words differs from the reference answer by more than 70% and less than 90% of the total word count; and 1 point is given if the number of words differs from the reference answer by more than 90% of the total word count.

[0040] The conciseness assessment scoring mechanism includes: five points for answers with the highest number of identical words less than 10%; four points for answers with the highest number of identical words greater than 20% and less than 40%; three points for answers with the highest number of identical words greater than 40% and less than 70%; two points for answers with the highest number of identical words greater than 70% and less than 90%; and one point for answers with the highest number of identical words greater than 90%.

[0041] A scoring arbitration agent is constructed based on a scoring arbitration mechanism; the scoring arbitration mechanism includes: reading the question and context, judging whether the question-answer pair is of acceptable quality in this dimension; whether the scoring difference between scoring agent A and scoring agent B is reasonable; and outputting a final conclusion; the final conclusion includes retaining the question-answer pair and deleting the question-answer pair.

[0042] The scoring MACA framework includes:

[0043] The input question and answer pair are scored by intelligent agent A and intelligent agent B respectively; when the difference between the scores of intelligent agent A and intelligent agent B is greater than the initial score review threshold, the score is reviewed, and then the score is arbitrated, and finally the final result is entered into the database; when the difference between the scores of intelligent agent A and intelligent agent B is less than the initial score review threshold, the result is directly entered into the database; the initial score review threshold is set to two by the system.

[0044] The collaborative scoring MACA framework mechanism of the scoring agent includes:

[0045] The initial scoring review includes the system first calling scoring agent A and scoring agent B to independently evaluate each set of questions and answers through scoring agent A and scoring agent B respectively; and obtaining the score of scoring agent A and the score of scoring agent B.

[0046] The scoring review includes the system requesting scoring agent A and scoring agent B to provide detailed supplementary reasons for their scoring of the question-and-answer pair, and requiring them to further explain their identification criteria, key judgment information points, and contextual understanding basis based on their original judgments.

[0047] The scoring arbitration includes a scoring arbitration agent that combines the problem, context, and scores from both parties to output a final conclusion.

[0048] The adaptive weight adjustment module uses an exponentially weighted moving average algorithm to dynamically update the weights of each screening and scoring dimension; the screening and scoring dimensions include fidelity, relevance, independence, accuracy, authenticity, correlation, coverage, and simplicity.

[0049] The exponentially weighted moving average algorithm includes:

[0050] The exponentially weighted moving average is calculated by using the exponentially weighted moving average formula to determine the average score for each screening scoring dimension; the exponentially weighted moving average formula is as follows:

[0051] ;

[0052] in Let i be the average score of the screening scoring dimension after the t-th evaluation; This is the result of a single evaluation for the i-th evaluation dimension; The average score of screening scoring dimension i after the (t-1)th evaluation; For smoothing coefficients;

[0053] The weights are updated based on a comparison between the average score of each screening scoring dimension and the threshold for that dimension; the threshold for each screening scoring dimension is set by the system; the threshold for each screening scoring dimension includes a high threshold for that dimension. and low threshold for screening scoring dimensions ;

[0054] The updated weights include:

[0055] When the average score of a screening scoring dimension is greater than the high threshold of that dimension, the weights corresponding to that dimension are reduced using a weight reduction formula; the weight reduction formula is as follows:

[0056] ;

[0057] When the average score of a screening scoring dimension is less than the low threshold of that dimension, the weight corresponding to that dimension is increased using a weight increase formula; the weight increase formula is as follows:

[0058] ;

[0059] in To filter the updated weights for rating dimension i; Let i be the weight of the filtering score dimension i after the t-th update; This is the weighting adjustment factor;

[0060] The weights of the screening dimensions are normalized after each adjustment using a screening normalization formula, which is:

[0061] ;

[0062] in To filter the updated weights of dimension i; Here are the updated weights for filtering dimension j; j is the number of filtering dimensions.

[0063] The weights of the rating dimensions are normalized after each adjustment using a rating normalization formula, which is:

[0064] ;

[0065] in The updated weights for the rating dimension r; Here is the updated weight of rating dimension e; e is the number of rating dimensions.

[0066] The final overall screening score is calculated based on the total screening score formula; the total screening score formula is:

[0067] ;

[0068] Where P is the final overall score for screening; The average score for the selection dimension n;

[0069] The final overall score is calculated according to the total score formula; the total score formula is:

[0070] ;

[0071] Where M is the final overall score; The average score for the rating dimension g.

[0072] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0073] Automatically selects high-quality evaluation samples, reducing reliance on manual intervention and improving efficiency and data quality; multi-agent collaborative mechanism reduces subjective bias and enhances evaluation credibility and reproducibility; multiple scoring dimensions with dynamic weight feedback; multi-model evaluation reduces the likelihood of bias and improves credibility. Attached Figure Description

[0074] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0075] Figure 1 This is a schematic diagram of the structure of an intelligent evaluation system for knowledge base question answering application according to the present invention;

[0076] Figure 2 This is a schematic diagram of the collaborative screening (MACA) framework of an intelligent evaluation system for knowledge base question answering applications according to the present invention.

[0077] Figure 3 This is a schematic diagram of the collaborative scoring (MACA) framework of an intelligent evaluation system for knowledge base question-and-answer applications according to the present invention. Detailed Implementation

[0078] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0079] Please see Figures 1-3 The present invention provides the following technical solution:

[0080] An intelligent evaluation system and method for knowledge base question-and-answer applications, the system comprising a data preprocessing module, a data synthesis module, a data filtering module, a data scoring module, and an adaptive weight adjustment module;

[0081] The data preprocessing module is used to receive original enterprise documents uploaded by users, perform data preprocessing on the original enterprise documents, and generate a data knowledge base; the data preprocessing includes document format parsing, structure cleaning, and data tagging;

[0082] In this embodiment, document format parsing:

[0083] Upload a Word document and extract the plain text content using python-docx. The core paragraphs are as follows:

[0084] "In the first quarter of 2024, the company plans to release three products: A) a smartwatch that supports heart rate monitoring and GPS, has a 7-day battery life, and is priced at 1299 yuan; B) wireless earphones with active noise cancellation and an 8-hour battery life, priced at 899 yuan; and C) a charger with 65W output, compatible with multiple devices, priced at 199 yuan. All products will be unveiled at the spring launch event on March 15."

[0085] Structured cleaning:

[0086] Remove blank lines and redundant symbols. The text length is approximately 380 characters, no splitting is required (less than 2000 characters).

[0087] Data tags:

[0088] Assign a unique ID: paragraph ID=P001, and store it in the database.

[0089] The data synthesis module uses a large language model to repeatedly ask and answer questions in each paragraph of the data knowledge base to generate question-answer pairs; then it organizes the question-answer pairs into standardized question-answer triplet data.

[0090] In this embodiment, a large language model is invoked to generate question-answer pairs:

[0091] Question: "Which three products will be released in the first quarter of 2024? What will their prices be?"

[0092] Answer: "In the first quarter of 2024, we plan to release three products: A smartwatch, B wireless earphones, and C charger. The prices of each product are as follows: A smartwatch is priced at 1299 yuan, B wireless earphones are priced at 899 yuan, and C charger is priced at 199 yuan."

[0093] Triple data: Question, Answer, Paragraph ID=P001.

[0094] The data filtering module constructs filtering agents based on filtering dimensions and a filtering arbitration mechanism. The filtering dimensions include fidelity, relevance, and independence. The filtering agents include filtering agent A, filtering agent B, and a filtering arbitration agent. Question-answer triplet data is filtered to select high-quality samples through a collaborative filtering MACA framework mechanism, which includes initial screening, screening review, and screening arbitration.

[0095] In this embodiment, the above question-and-answer pair (question + answer + P001) is input.

[0096] The selection agent A and selection agent B were rated separately, and the results are as follows:

[0097] Screening agent A:

[0098] Fidelity: 4 points are awarded if the word count of the answer is greater than 70% but less than 90% of the word count of the surrounding text;

[0099] Relevance: 5 points are awarded if the answer is identical to the context in terms of word count as a percentage of the context word count;

[0100] Independence: 1 point is awarded if the answer differs from the context in word count and the answer is less than 10% of the total word count.

[0101] Filtering agent B:

[0102] Fidelity: 4 points are awarded if the word count of the answer is greater than 70% but less than 90% of the word count of the surrounding text;

[0103] Relevance: 5 points are awarded if the answer is identical to the context in terms of word count as a percentage of the context word count;

[0104] Independence: 1 point is awarded if the answer differs from the context in word count and the answer is less than 10% of the total word count.

[0105] Poor rating:

[0106] Fidelity: 0 points; Relevance: 0 points; Independence: 0 points;

[0107] Judgment: If the score difference across all dimensions is ≤2 (the initial screening threshold), no further review / arbitration is required.

[0108] The data scoring module constructs scoring agents based on scoring dimensions and a scoring arbitration mechanism. The scoring dimensions include accuracy, authenticity, relevance, coverage, and simplicity. The scoring agents include scoring agent A, scoring agent B, and a scoring arbitration agent. High-quality samples are scored through a collaborative scoring MACC framework mechanism. The collaborative scoring MACC framework mechanism includes initial score review, score verification, and score arbitration.

[0109] In this embodiment, the question-and-answer system under test answered: "In the first quarter of 2024, we plan to release two products: A smartwatch and B wireless earphones. The A smartwatch will be priced at 1299 yuan and the B wireless earphones at 899 yuan. They support heart rate monitoring, GPS, active noise cancellation, and have a battery life of 7 days and 8 hours, respectively."

[0110] Enter the above question and answer pair (question + answer + P001).

[0111] The selection agent A and selection agent B were rated separately, and the results are as follows:

[0112] Scoring agent A:

[0113] Accuracy: 4 points are awarded if the number of words in the answer that are identical to the reference answer is greater than 90%;

[0114] Authenticity: Completely authentic; all content of the answer can be found in the paragraph. 5 points.

[0115] Relevance: The percentage of the number of words in the answer that are the same as the reference answer; if the reference answer has more than 90% of its word count, 5 points will be awarded.

[0116] Coverage: Answers differing in word count from the reference answer; answers with more than 70% but less than 90% of the word count receive 3 points.

[0117] Conciseness: 5 points are awarded if the highest number of identical words in the answer is less than 10%;

[0118] Scoring agent B:

[0119] Accuracy: 4 points are awarded if the number of words in the answer that are identical to the reference answer is greater than 90%;

[0120] Authenticity: Completely authentic; all content of the answer can be found in the paragraph. 5 points.

[0121] Relevance: The percentage of the number of words in the answer that are the same as the reference answer; if the reference answer has more than 90% of its word count, 5 points will be awarded.

[0122] Coverage: Answers with a word count different from the reference paragraphs will receive 3 points if the word count of the answer is greater than 70% but less than 90%.

[0123] Conciseness: 4 points are awarded if the highest number of identical words in the answer accounts for more than 20% but less than 40% of the total.

[0124] Poor rating:

[0125] Accuracy: 0 points; Authenticity: 0 points; Relevance: 0 points; Coverage: 0 points; Conciseness: 1 point;

[0126] Judgment: If the score difference across all dimensions is ≤2 (the initial screening threshold), no further review / arbitration is required.

[0127] The adaptive weight adjustment module uses an exponentially weighted moving average algorithm to dynamically update the weights of each screening and scoring dimension; the screening and scoring dimensions include fidelity, relevance, independence, accuracy, authenticity, correlation, coverage, and simplicity.

[0128] The exponentially weighted moving average is calculated by using the exponentially weighted moving average formula to determine the average score for each screening scoring dimension; the exponentially weighted moving average formula is as follows:

[0129] ;

[0130] in Let i be the average score of the screening scoring dimension after the t-th evaluation; This is the result of a single evaluation for the i-th evaluation dimension; The average score of screening scoring dimension i after the (t-1)th evaluation; For smoothing coefficients;

[0131] In this embodiment, the definition is as follows: It is 0.3; defined Both are 4.0;

[0132] Loyalty: ;

[0133] Correlation: ;

[0134] Independence: ;

[0135] accuracy: ;

[0136] Authenticity: ;

[0137] Relevance: ;

[0138] Coverage: ;

[0139] Simplicity: ;

[0140] The weights are updated based on a comparison between the average score of each screening scoring dimension and the threshold for that dimension; the threshold for each screening scoring dimension is set by the system; the threshold for each screening scoring dimension includes a high threshold for that dimension. and low threshold for screening scoring dimensions ;

[0141] The updated weights include:

[0142] When the average score of a screening scoring dimension is greater than the high threshold of that dimension, the weights corresponding to that dimension are reduced using a weight reduction formula; the weight reduction formula is as follows:

[0143] ;

[0144] When the average score of a screening scoring dimension is less than the low threshold of that dimension, the weight corresponding to that dimension is increased using a weight increase formula; the weight increase formula is as follows:

[0145] ;

[0146] in To filter the updated weights for rating dimension i; Let i be the weight of the filtering score dimension i after the t-th update; This is the weighting adjustment factor;

[0147] In this embodiment, the definition is as follows: =0.05; defined 4.5; Definition It is 3.5;

[0148] The weights of the selection and scoring dimensions remain unchanged, so no normalization is required. ; .

[0149] The final overall screening score is calculated based on the total screening score formula; the total screening score formula is:

[0150] ;

[0151] Where P is the final overall score for screening; The average score for the selection dimension n;

[0152] In this embodiment, the average score of the screening dimension is as follows:

[0153] ;

[0154] The final overall score is calculated according to the total score formula; the total score formula is:

[0155] ;

[0156] Where M is the final overall score; The average score for the rating dimension g.

[0157] In this embodiment, the average score for the rating dimension is as follows:

[0158] .

[0159] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0160] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An intelligent evaluation system for knowledge base question-and-answer applications, characterized in that, include: The system includes a data preprocessing module, a data synthesis module, a data filtering module, a data scoring module, and an adaptive weight adjustment module. The data preprocessing module is used to receive original enterprise documents uploaded by users, perform data preprocessing on the original enterprise documents, and generate a data knowledge base; the data preprocessing includes document format parsing, structure cleaning, and data tagging; The data synthesis module uses a large language model to repeatedly ask and answer questions to generate question-answer pairs for each paragraph of the data knowledge base. Organize the question-answer pairs into standardized question-answer triplet data; The data filtering module constructs filtering agents based on filtering dimensions and a filtering arbitration mechanism. The filtering dimensions include fidelity, relevance, and independence. The filtering agents include filtering agent A, filtering agent B, and a filtering arbitration agent. Based on the loyalty judgment scoring mechanism, the relevance judgment scoring mechanism, and the independence judgment scoring mechanism, screening agent A and screening agent B are constructed; Construct a screening arbitration agent based on the screening arbitration mechanism; The screening and arbitration mechanism includes: reading the question and context to determine whether the question-answer pair is of acceptable quality in that dimension; determining whether the score difference between screening agent A and screening agent B is reasonable; and outputting a final conclusion. The final conclusion includes retaining the question-answer pair and deleting the question-answer pair. The question-answer triplet data is screened for high-quality samples through the screening agent collaborative screening MACA framework mechanism, which includes initial screening, screening review, and screening arbitration. The data scoring module constructs scoring agents based on scoring dimensions and a scoring arbitration mechanism. The scoring dimensions include accuracy, authenticity, relevance, coverage, and simplicity. The scoring agents include scoring agent A, scoring agent B, and a scoring arbitration agent. Scoring agents A and B are constructed based on accuracy, authenticity, relevance, coverage, and simplicity judgment mechanisms. A scoring arbitration agent is constructed based on the scoring arbitration mechanism, which includes: reading the question and context to determine if the question-answer pair is of acceptable quality in that dimension; determining if the difference in scores between scoring agents A and B is reasonable; and outputting a final conclusion, which includes retaining or deleting the question-answer pair. High-quality samples are scored using a collaborative scoring MACC framework mechanism, which includes initial score review, score verification, and score arbitration. The adaptive weight adjustment module uses an exponentially weighted moving average algorithm to dynamically update the weights of each screening and scoring dimension; the screening and scoring dimensions include fidelity, relevance, independence, accuracy, authenticity, correlation, coverage, and simplicity.

2. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 1, characterized in that, The data preprocessing includes: The document format parsing includes: The document formats supported are PDF and Word. Plain text content is extracted using document parsing tools, including pdfminer and python-docx. The structured cleaning includes: The extracted plain text content is subjected to structured cleaning and segmentation operations. The structured cleaning is to remove blank lines and useless characters, and the segmentation operation is to use newline characters to divide the text into logical paragraphs of 2000 characters in length. The data tags include: After the structured cleaning process is completed, a unique paragraph tag is assigned to each logical paragraph to generate a data knowledge base.

3. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 1, characterized in that, The data synthesis module includes: The system calls the large language model to generate question-answer pairs for each preprocessed knowledge segment. The question-answer pairs include generated questions with logical features and accurate and complete reference answers generated based on the context. The logical features include semantic independence, natural expression, and content relevance; semantic independence means that the question can still be understood even when removed from its original context; natural expression means that it conforms to the questioning habits of users on search engines or Q&A platforms; and content relevance means that the information involved in the question is directly answered in the original paragraph. The question-and-answer triplet data includes the question, the answer, and the source paragraph tag.

4. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 1, characterized in that: Given a context and a question; The fidelity scoring mechanism includes: a word count ratio of the answer to the surrounding text greater than 90% is defined as five points; a word count ratio of the answer to the surrounding text greater than 70% but less than 90% is defined as four points; a word count ratio of the answer to the surrounding text greater than 40% but less than 70% is defined as three points; a word count ratio of the answer to the surrounding text greater than 20% but less than 40% is defined as two points; and a word count ratio of the answer to the surrounding text less than 10% is defined as one point. The relevance scoring mechanism includes: 5 points for an answer whose word count matches the context word count by more than 90% of the context word count; 4 points for an answer whose word count matches the context word count by more than 70% but less than 90% of the context word count; 3 points for an answer whose word count matches the context word count by more than 40% but less than 70% of the context word count; 2 points for an answer whose word count matches the context word count by more than 20% but less than 40% of the context word count; and 1 point for an answer whose word count matches the context word count is less than 10% of the context word count. The independence assessment scoring mechanism includes: 5 points for an answer whose word count differs from the context by more than 90% of the total word count; 4 points for an answer whose word count differs from the context by more than 70% but less than 90% of the total word count; 3 points for an answer whose word count differs from the context by more than 40% but less than 70% of the total word count; 2 points for an answer whose word count differs from the context by more than 20% but less than 40% of the total word count; and 1 point for an answer whose word count differs from the context by less than 10% of the total word count.

5. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 4, characterized in that, The screening MACA framework includes: The input question and answer pair are filtered by agent A and agent B respectively; when the score difference between agent A and agent B is greater than the initial screening threshold, the screening is reviewed, then arbitration is carried out, and finally the final result is entered into the database; when the score difference between agent A and agent B is less than the initial screening threshold, the result is directly entered into the database; the initial screening threshold is set to binary by the system.

6. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 5, characterized in that, The collaborative screening MACA framework mechanism of the screening agent includes: The initial screening process involves the system first calling screening agent A and screening agent B to independently evaluate each set of questions and answers through screening agent A and screening agent B respectively; thus obtaining the scores of screening agent A and screening agent B. The screening review includes the system requesting screening agent A and screening agent B to provide more detailed supplementary reasons for their scoring of the question-and-answer pair, and requiring them to further explain their identification criteria, key judgment information points, and contextual understanding basis based on their original judgments. The screening arbitration process involves selecting arbitration agents based on the problem, context, and scores from both parties, and then outputting a final conclusion.

7. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 1, characterized in that: Here is a question and a suggested answer; The accuracy assessment scoring mechanism includes: five points for an answer that is completely identical to the reference answer and contains no errors; four points for an answer that shares more than 90% of the same words as the reference answer; three points for an answer that shares more than 70% but less than 90% of the same words as the reference answer; two points for an answer that shares more than 40% but less than 70% of the same words as the reference answer; and one point for an answer that is completely different from the reference answer. The authenticity assessment scoring mechanism includes: completely authentic, with all content found in the paragraph, is defined as five points; the entire content is fabricated, and is inconsistent with or contradicts the information in the reference paragraph, is defined as one point. The relevance assessment scoring mechanism includes: 5 points for an answer with the same word count as the reference answer (more than 90% of the reference answer's word count); 4 points for an answer with the same word count as the reference answer (more than 70% but less than 90% of the reference answer's word count); 3 points for an answer with the same word count as the reference answer (more than 40% but less than 70% of the reference answer's word count); 2 points for an answer with the same word count as the reference answer (more than 20% but less than 40% of the reference answer's word count); and 1 point for an answer with the same word count as the reference answer (less than 10% of the reference answer's word count). The coverage assessment scoring mechanism includes the following: A score of 5 points is given if the number of words differs from the reference answer by less than 10% of the total word count; 4 points are given if the number of words differs from the reference answer by more than 20% and less than 40% of the total word count; 3 points are given if the number of words differs from the reference answer by more than 40% and less than 70% of the total word count; 2 points are given if the number of words differs from the reference answer by more than 70% and less than 90% of the total word count; and 1 point is given if the number of words differs from the reference answer by more than 90% of the total word count. The conciseness assessment scoring mechanism includes: five points for answers with the highest number of identical words less than 10%; four points for answers with the highest number of identical words greater than 20% and less than 40%; three points for answers with the highest number of identical words greater than 40% and less than 70%; two points for answers with the highest number of identical words greater than 70% and less than 90%; and one point for answers with the highest number of identical words greater than 90%.

8. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 7, characterized in that, The scoring MACA framework includes: The input question and answer pair are scored by intelligent agent A and intelligent agent B respectively; when the difference between the scores of intelligent agent A and intelligent agent B is greater than the initial score review threshold, the score is reviewed, and then the score is arbitrated, and finally the final result is entered into the database; when the difference between the scores of intelligent agent A and intelligent agent B is less than the initial score review threshold, the result is directly entered into the database; the initial score review threshold is set to two by the system.

9. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 8, characterized in that, The collaborative scoring MACA framework mechanism of the scoring agent includes: The initial scoring review includes the system first calling scoring agent A and scoring agent B to independently evaluate each set of questions and answers through scoring agent A and scoring agent B respectively; and obtaining the score of scoring agent A and the score of scoring agent B. The scoring review includes the system requesting scoring agent A and scoring agent B to provide detailed supplementary reasons for their scoring of the question-and-answer pair, and requiring them to further explain their identification criteria, key judgment information points, and contextual understanding basis based on their original judgments. The scoring arbitration includes a scoring arbitration agent that combines the problem, context, and scores from both parties to output a final conclusion.

10. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 1, characterized in that, The exponentially weighted moving average algorithm includes: The exponentially weighted moving average is calculated by using the exponentially weighted moving average formula to determine the average score for each screening scoring dimension; the exponentially weighted moving average formula is as follows: ； in Let i be the average score of the screening scoring dimension after the t-th evaluation; This is the result of a single evaluation for the i-th evaluation dimension; The average score of screening scoring dimension i after the (t-1)th evaluation; This is the smoothing coefficient.

11. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 10, characterized in that, The weights are updated based on a comparison between the average score of each screening scoring dimension and the threshold for that dimension; the threshold for each screening scoring dimension is set by the system; the threshold for each screening scoring dimension includes a high threshold for that dimension. and low threshold for screening scoring dimensions ; The updated weights include: When the average score of a screening scoring dimension is greater than the high threshold of that dimension, the weights corresponding to that dimension are reduced using a weight reduction formula; the weight reduction formula is as follows: ； When the average score of a screening scoring dimension is less than the low threshold of that dimension, the weight corresponding to that dimension is increased using a weight increase formula; the weight increase formula is as follows: ； in To filter the updated weights for rating dimension i; Let i be the weight of the filtering score dimension i after the t-th update; This is the weighting adjustment coefficient.

12. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 11, characterized in that, The weights of the screening dimensions are normalized after each adjustment using a screening normalization formula, which is: ； in To filter the updated weights of dimension i; To filter the updated weights of dimension j; j represents the number of filtering dimensions; The weights of the rating dimensions are normalized after each adjustment using a rating normalization formula, which is: ； in The updated weights for the rating dimension r; The updated weights for rating dimension e; e represents the number of scoring dimensions.

13. The intelligent evaluation system for knowledge base question-and-answer applications according to claim 12, characterized in that, The final overall screening score is calculated based on the total screening score formula; the total screening score formula is: ； Where P is the final overall score for screening; The average score for the selection dimension n; The final overall score is calculated according to the total score formula; the total score formula is: ； Where M is the final overall score; The average score for the rating dimension g.

Images