Knowledge graph-based large model answer effect evaluation method and system

Abstract

The invention discloses a large model answer effect evaluation method and system based on a knowledge graph, and relates to the field of large language models. The method comprises the following steps: preprocessing a plurality of questions; carrying out convolution calculation on the preprocessed sentences to obtain keyword nodes; keyword attributes are extracted through the keyword nodes; screening the keyword attributes according to the attention influence score of the keyword attribute of each keyword node; constructing a context graph by using the screened keyword attributes, and classifying keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes; generating a keyword knowledge graph by using the keyword nodes as entities and the classification results as entity relationships; questions input by the user are answered and answered through the large model; and evaluating the result of the large model. The answering effect of the question answering system can be automatically evaluated, and the evaluation efficiency and accuracy are improved.

Description

Technical Field

[0001] The present invention relates to the field of large language models, and in particular to a large model answer effect evaluation method and system based on knowledge graph. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent question-answering systems based on large models have been widely used in many fields such as medicine, education, and finance. However, the answer effect of these systems often depends on the accuracy of the model and the richness of the data. Most of the existing evaluation methods rely on manual evaluation, which is inefficient and subjective, and it is difficult to meet the needs of large-scale applications. As a structured knowledge representation method, knowledge graphs can effectively integrate and represent entities, attributes, and relationships in a field, and provide rich background knowledge and semantic information for question-answering systems. However, how to effectively use knowledge graphs to objectively and comprehensively evaluate the answer effect of large models is still a problem to be solved. Based on this, the present invention proposes a large model answer effect evaluation method and system based on knowledge graphs, which aims to automatically evaluate the answer effect of the question-answering system by constructing a domain knowledge graph and combining the output of a large model, thereby improving the efficiency and accuracy of the evaluation. Summary of the invention

[0003] The technical problem to be solved by the present invention is how to effectively use the knowledge graph to objectively and comprehensively evaluate the answer effect of the large model. The purpose is to provide a large model answer effect evaluation method and system based on the knowledge graph. By constructing a domain knowledge graph and combining the output of the large model, the answer effect of the question and answer system is automatically evaluated to improve the efficiency and accuracy of the evaluation.

[0004] The present invention is achieved through the following technical solutions:

[0005] A method for evaluating the answer effect of a large model based on a knowledge graph, comprising:

[0006] Build a keyword extraction model, including:

[0007] Pre-process multiple questions;

[0008] The preprocessed sentences are used to obtain keyword nodes through convolution calculation;

[0009] Extract keyword attributes through the above keyword nodes; filter each of the above keyword attributes according to the attention influence score of the keyword attributes of each keyword node; construct a context graph using the filtered keyword attributes, and classify the above keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes;

[0010] The multiple groups of keyword head nodes and keyword tail nodes extracted by the above keyword extraction model are entities, and the classification results of each group of keyword head nodes and keyword tail nodes are entity relationships to generate a keyword knowledge graph;

[0011] Evaluate the above keyword knowledge graph;

[0012] Use the big model to answer the questions input by the user;

[0013] The results of the above large model are evaluated.

[0014] The above preprocesses multiple questions, including text processing and vectorization processing;

[0015] The text processing includes standardizing one or more of the text format, address name and abbreviation vocabulary;

[0016] The above-mentioned vectorization processing includes performing matrix multiplication by the embedding weight matrix and the ID of each character mapping of the question sentence to obtain the sentence vector representation of the question sentence.

[0017] The above preprocessed sentences are used to obtain keyword nodes through convolution calculation, including:

[0018] The sentence is split into characters by sliding the window, and each character is convolved and activated by the tanh function;

[0019] Perform maximum pooling on each sliding window to obtain the features corresponding to the specific filter;

[0020] The extracted features are activated by the sigmoid activation function;

[0021] Perform matrix operations on the results before and after activation by the sigmoid activation function to obtain keyword nodes;

[0022] The above keyword nodes are calculated through two fully connected layers.

[0023] The keyword attributes extracted through the keyword node include:

[0024] The above keyword nodes are subjected to graph network learning through graph neural network;

[0025] The above keyword nodes are pooled through the pooling layer to extract keyword attributes;

[0026] Concatenate the maximum values ​​of the above keyword attributes on the last dimension of each keyword.

[0027] The above keyword nodes are subjected to graph network learning through graph neural network, including:

[0028] Construct the GCN network model, expressed as:

[0029] ;

[0030] ;

[0031] ;

[0032] in, Indicates that the keyword node after convolution is in The feature vector of the layer; is the degree matrix of the vertex; is a Laplacian matrix; Represents the feature vector output by the keyword node through the convolutional network layer; Indicates Parameters of layer convolution; represents the activation function; is the adjacency matrix of the graph; is the identity matrix, that is, a matrix with 1 on the diagonal and 0 on the rest; is included OK The degree matrix of the vertices of the column; is the vertex The adjacency matrix of the graph .

[0033] The above-mentioned keyword attributes are screened according to the attention influence score of the keyword attribute of each keyword node, including:

[0034] Calculate the self-attention score of each keyword node, expressed as:

[0035] ;

[0036] in, Represents the self-attention score of each keyword node; are the parameters of the convolution of the above pooling layer, represents the activation function;

[0037] The self-attention score of each of the above keyword attributes is normalized to obtain the attention influence score, which is expressed as:

[0038] ;

[0039] in, represents the attention impact score of each of the above keyword attributes after normalization; Indicates The self-attention scores of the above keyword attributes; The natural constant e Power;

[0040] Filter the above keyword attributes that meet the threshold, including:

[0041] ;

[0042] in, To meet the minimum score for attention impact; yes The standard deviation of is a tunable hyperparameter.

[0043] After the above keyword nodes are classified by the multi-layer perceptron, the following steps are also included:

[0044] The softmax function is used to calculate the probabilities of different classification results to obtain the final multiple groups of keyword head nodes and keyword tail nodes.

[0045] The above-mentioned evaluation of the keyword knowledge graph includes: any one or more of attribute evaluation, concept evaluation and relationship evaluation;

[0046] The above-mentioned attribute evaluation includes any one or both of repetitiveness detection and abnormal character detection; the above-mentioned repetitiveness detection is whether the name of the entity is repeated; the above-mentioned abnormal character detection is whether the repetitive rate of each character in the entity is abnormal;

[0047] The above concept evaluation is to detect whether the keyword tail node of the entity is the most fine-grained entity;

[0048] The above relationship evaluation is to evaluate whether the keyword tail node of the same relationship of the keyword head node is unique.

[0049] The above evaluation of the results of the above large model includes:

[0050] The structure of the reasoning consistency evaluation model is defined to include a linear layer, a multi-head matrix calculation layer, and a fully connected layer in sequence;

[0051] The input of the linear layer includes multiple groups of keyword head nodes, relations, and keyword tail nodes, which convert the input characters into multi-dimensional vector representations;

[0052] After the result of the linear layer calculation is input into the multi-head matrix calculation layer, it is subjected to a binary classification judgment on whether the relationship is established through the fully connected layer. After the result is normalized using the softmax function, the final judgment result on whether the relationship is established is obtained.

[0053] Screening of uncertain consistency evaluation results for manual review;

[0054] The determined consistency evaluation results and population audit results are returned to the user's front end through the http interface for display.

[0055] A large model answer effect evaluation system based on knowledge graph, including:

[0056] Keyword extraction model, including:

[0057] A data preprocessing module preprocesses multiple questions;

[0058] Convolutional network layer, the preprocessed sentences are calculated through convolution to obtain keyword nodes;

[0059] The neural network layer extracts keyword attributes through the keyword nodes; screens the keyword attributes according to the attention influence score of the keyword attributes of each keyword node; constructs a context graph using the filtered keyword attributes, and classifies the keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes;

[0060] A knowledge graph generation module uses the multiple groups of keyword head nodes and keyword tail nodes extracted by the above keyword extraction model as entities, and the classification results of each group of keyword head nodes and keyword tail nodes are entity relationships to generate a keyword knowledge graph;

[0061] A graph evaluation module, which evaluates the above keyword knowledge graph;

[0062] The big model question answering module inputs the user's question into the big model for question answering;

[0063] The large model evaluation module evaluates the results of the above large model.

[0064] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0065] The present application provides a method for evaluating the answer effect of a large model based on a knowledge graph, which comprises preprocessing a plurality of questions; obtaining keyword nodes through convolution calculation of the preprocessed sentences; extracting keyword attributes through the keyword nodes; screening each keyword attribute according to the attention influence score of the keyword attribute of each keyword node; constructing a context graph using the screened keyword attributes, and classifying the keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes; generating a keyword knowledge graph using the keyword nodes as entities and the keyword classification results as entity relationships; evaluating the results of the large model through the knowledge graph, and effectively using the knowledge graph to objectively and comprehensively evaluate the answer effect of the large model, thereby improving the efficiency and accuracy of the evaluation. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without creative work. In the drawings:

[0067] Figure 1 This is a flowchart of a method for evaluating the effectiveness of a large model answer based on a knowledge graph in an embodiment of the present application. DETAILED DESCRIPTION

[0068] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0069] Example

[0070] like Figure 1 As shown, the embodiment of the present application provides a large model answer effect evaluation method based on knowledge graph, including:

[0071] Pre-process multiple questions;

[0072] The preprocessed sentences are used to obtain keyword nodes through convolution calculation;

[0073] Extract keyword attributes through the above keyword nodes; filter each of the above keyword attributes according to the attention influence score of the keyword attributes of each keyword node; construct a context graph using the filtered keyword attributes, and classify the above keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes;

[0074] Using the above keyword nodes as entities and the keyword classification results as entity relationships, a keyword knowledge graph is generated;

[0075] Evaluate the above keyword knowledge graph;

[0076] Use the big model to answer the questions input by the user;

[0077] The results of the above large model are evaluated.

[0078] The above preprocesses multiple questions, including text processing and vectorization processing;

[0079] The text processing includes standardizing one or more of the text format, address name and abbreviation vocabulary;

[0080] The above-mentioned vectorization processing includes performing matrix multiplication by the embedding weight matrix and the ID of each character mapping of the question sentence to obtain the sentence vector representation of the question sentence.

[0081] The above preprocessed sentences are used to obtain keyword nodes through convolution calculation, including:

[0082] The sentence is split into characters by sliding the window, and each character is convolved and activated by the tanh function;

[0083] Perform maximum pooling on each sliding window to obtain the features corresponding to the specific filter;

[0084] The extracted features are activated by the sigmoid activation function;

[0085] Perform matrix operations on the results before and after activation by the sigmoid activation function to obtain keyword nodes;

[0086] The above keyword nodes are calculated through two fully connected layers.

[0087] The keyword attributes extracted through the keyword node include:

[0088] The above keyword nodes are subjected to graph network learning through graph neural network;

[0089] The above keyword nodes are pooled through the pooling layer to extract keyword attributes;

[0090] Concatenate the maximum values ​​of the above keyword attributes on the last dimension of each keyword.

[0091] The above keyword nodes are subjected to graph network learning through graph neural network, including:

[0092] Construct the GCN network model, expressed as:

[0093] ;

[0094] ;

[0095] ;

[0096] in, Indicates that the keyword node after convolution is in The feature vector of the layer; is the degree matrix of the vertex; is a Laplacian matrix; Represents the feature vector output by the keyword node through the convolutional network layer; Indicates Parameters of layer convolution; represents the activation function; is the adjacency matrix of the graph; is the identity matrix, that is, a matrix with 1 on the diagonal and 0 on the rest; is included OK The degree matrix of the vertices of the column; is the vertex The adjacency matrix of the graph .

[0097] The above-mentioned keyword attributes are screened according to the attention influence score of the keyword attribute of each keyword node, including:

[0098] Calculate the self-attention score of each keyword node, expressed as:

[0099] ;

[0100] in, Represents the self-attention score of each keyword node; are the parameters of the convolution of the above pooling layer, represents the activation function;

[0101] The self-attention score of each of the above keyword attributes is normalized to obtain the attention influence score, which is expressed as:

[0102] ;

[0103] in, represents the attention impact score of each of the above keyword attributes after normalization; Indicates The self-attention scores of the above keyword attributes; The natural constant e Power;

[0104] Filter the above keyword attributes that meet the threshold, including:

[0105] ;

[0106] in, To meet the minimum score for attention impact; yes The standard deviation of is a tunable hyperparameter.

[0107] After the above keyword nodes are classified by the multi-layer perceptron, the following steps are also included:

[0108] The softmax function is used to calculate the probabilities of different classification results to obtain the final multiple groups of keyword head nodes and keyword tail nodes.

[0109] The above-mentioned evaluation of the keyword knowledge graph includes: any one or more of attribute evaluation, concept evaluation and relationship evaluation;

[0110] The above-mentioned attribute evaluation includes any one or both of repetitiveness detection and abnormal character detection; the above-mentioned repetitiveness detection is whether the name of the entity is repeated; the above-mentioned abnormal character detection is whether the repetitive rate of each character in the entity is abnormal;

[0111] The above concept evaluation is to detect whether the keyword tail node of the entity is the most fine-grained entity;

[0112] The above relationship evaluation is to evaluate whether the keyword tail node of the same relationship of the keyword head node is unique.

[0113] The above evaluation of the results of the above large model includes:

[0114] The structure of the reasoning consistency evaluation model is defined to include a linear layer, a multi-head matrix calculation layer, and a fully connected layer in sequence;

[0115] The input of the linear layer includes multiple groups of keyword head nodes, relations, and keyword tail nodes, which convert the input characters into multi-dimensional vector representations;

[0116] After the result of the linear layer calculation is input into the multi-head matrix calculation layer, it is subjected to a binary classification judgment on whether the relationship is established through the fully connected layer. After the result is normalized using the softmax function, the final judgment result on whether the relationship is established is obtained.

[0117] Screening of uncertain consistency evaluation results for manual review;

[0118] The determined consistency evaluation results and population audit results are returned to the user's front end through the http interface for display.

[0119] After the user chooses to initiate an evaluation command, the detection command is returned to the backend through the HTTP interface. After receiving the incoming API request, the backend begins to call the evaluation system to evaluate the current model's question and answer effect.

[0120] For the raw data used to evaluate the large model, it is first preprocessed. Data preprocessing includes three parts: 1) Text standardization: convert traditional Chinese to simplified Chinese, unify uppercase numbers to Arabic numerals, unify Roman numerals, convert English characters to uppercase and lowercase, and unify punctuation marks; 2) Address standardization: Address standardization is mainly standardized mapping. The specific approach is: use the standard address database for comparison to ensure that each address component meets the standard specifications. For example, if "Sanlitun Road" has a more standardized name "Sanlitun Street" in the database, it will be replaced accordingly. 3) Abbreviation standardization: Build an abbreviation word list in advance to save the mapping of common abbreviation word lists and full-written words in the current field. For the input text, we directly match it with the current abbreviation word list to determine whether there is text in the abbreviation word list. If so, replace it and replace the abbreviation words in the text with full-written words.

[0121] Define prompt words: pre_prompt, end_prompt, where:

[0122] ①pre_prompt: records the text data information currently input;

[0123] ②end_prompt: records the role of a specific field assigned to the model, such as the medical field. The constructed end_prompt is:

[0124] "You are an expert in the medical field. Please combine all the above information and give a professional and comprehensive response with a scientific and rigorous attitude."

[0125] Join the contents of pre_prompt and end_prompt together, record it as merge_prompt, and input merge_prompt into the LLM model to answer.

[0126] The structure of the keyword extraction model is:

[0127] Embedding layer:

[0128] By performing matrix matching between the embedding weight matrix and the ID mapped to the input question, the embedding word vector is obtained as the embedding matrix representation of the input data. The vector dimension is 256. The original ID mapping data is output as the sentence vector representation x after embedding. Among them, the embedding weight matrix is ​​a preset two-dimensional numerical matrix filled with random numbers. The input question is mapped to the digital ID one by one by a preset vocabulary that stores a sufficient number of words, and then the vector representation of the corresponding matrix row is found in the embedding matrix according to the ID.

[0129] The calculation process of the convolutional network layer is:

[0130] First, the calculation of a single-layer CNN network: a filter with a window length of h can slide the sentence Divided into single character representation , for the window After convolution and activation by tanh function, the formula is:

[0131] ;

[0132] Where n is the number of characters in the sentence vector representation x.

[0133] Sliding the convolution window and reactivating it gives:

[0134] ;

[0135] The sliding window can be used to obtain the results of n-h+1 sliding window extractions, and the results are placed in a collection. express.

[0136] Next, the Max pooling layer receives the output after the convolution kernel activation. , perform maximum pooling to obtain the features corresponding to a specific filter, the purpose of which is to capture the most important features for each feature map:

[0137] ;

[0138] in, Feature mapping for a filter;

[0139] The output of the pooling layer is:

[0140] ;

[0141] Next, calculate the output of the pooling layer and pass z through the sigmoid activation function:

[0142] ;

[0143] Then, z_gated and z activated by the sigmoid activation function are subjected to matrix operations to obtain the output of the model layer:

[0144]

[0145] The output z will be calculated after two fully connected layers, and the calculation formula is expressed as:

[0146]

[0147] After that, the graph network is learned through the three-layer graph neural network GCN network layer. The GCN network model is constructed as follows:

[0148] ;

[0149] ;

[0150] ;

[0151] Among them, z1 represents the output of the node convolutional network layer, represents the feature vector of the node in the kth layer after convolution, represents the parameters of the k-1th layer of convolution, and ReLU represents the activation function. D is the degree matrix (diagonal matrix) of the vertex, A is the adjacency matrix of the graph, and the part composed of matrices A and D is a Laplacian matrix. The I in is the identity matrix, that is, a matrix with 1 on the diagonal and 0 on the rest.

[0152] After that, it passes through the pooling layer, which performs a pooling operation on the vector output by the GCN network layer for further feature extraction; the maximum value of the last dimension of each input vector is taken, and then the maximum values ​​of all input vectors are concatenated to output the concatenated representation vector;

[0153] ;

[0154] Where N is the number of nodes in the graph, F is the embedding representation vector of the node feature, is the output of the current network layer.

[0155] Next, to select relevant entity attribute nodes, we use a self-attention score Z:

[0156] ;

[0157] in represents the feature vector calculation result of the node in the kth layer, Z represents the calculated self-attention score, is the convolution parameter of the pooling layer, and tanh represents the activation function. D is the degree matrix (diagonal matrix) of the vertex, A is the adjacency matrix of the graph, and the part composed of matrices A and D is a Laplacian matrix. I in A+I is the unit matrix, that is, a matrix with 1 on the diagonal and 0 on the rest.

[0158] The obtained attention score is passed through softmax to obtain the normalized attention influence score , To calculate the exponential function with the natural constant e as the base:

[0159] = /

[0160] After calculation by the softmax formula, the value is mapped to between 0 and 1.

[0161] Filter attribute nodes that meet the threshold using the calculated attention score:

[0162] idx=max( )- );

[0163] in: ) is the standard deviation of Zscore, α is a manually set adjustable hyperparameter, and idx is the calculated attention score that meets the minimum score.

[0164] Next, build the context graph (CG):

[0165] ;

[0166] ;

[0167] in, The node representation vector set representing the attention score above idx, is the corresponding attention mask.

[0168] Next, the MLP multi-layer perceptron is input, and the entity type is classified by the multi-layer perceptron MLP. The softmax function is used to calculate the relationship probability to obtain the final extraction question type judgment result. The final result is:

[0169] result = softmax(MLP( )).

[0170] The graph evaluation is divided into two parts: the evaluation of knowledge within the graph and the evaluation of knowledge outside the graph. First, the abnormal content within the graph generated by the keyword extraction model is evaluated, and then the reasoning consistency evaluation model of the corresponding field is established based on the knowledge graph + manual review method to evaluate the model results.

[0171] Currently, the knowledge graph detection system includes three parts: attribute evaluation, concept evaluation and relationship evaluation.

[0172] ① Attribute evaluation: includes repetitive detection and abnormal character detection.

[0173] 1) Repeatability test

[0174] In the knowledge graph, there cannot be two entities with the same name in the entity set of the same type. We can directly match the entity name to determine whether there are duplicate entities. This ensures the uniqueness of entities under each type and avoids data duplication.

[0175] 2) Abnormal character detection: Use the probability of each word in the current knowledge graph as an indicator for detection.

[0176] Assumptions:

[0177] S – current entity;

[0178] N – sentence length;

[0179] p(wi) – probability of the ith word;

[0180] p(wi|w1w2w3…wi-1) – This means the probability of the i-th word is calculated based on the first i-1 words;

[0181] The formula for calculating the ppl of the current entity is:

[0182] ;

[0183] The final calculated PP(S) is the opposite of the probability that the entity is a word in the current knowledge graph. The smaller the PP(S), the greater the probability that the entity is a word in the current knowledge graph. The threshold of this probability is 0.5. For entities below the threshold, it is considered that they may contain abnormal characters and need further manual review. For entities with a probability greater than 0.5, we consider them to be correct.

[0184] ② Concept evaluation: For a definition-based knowledge graph, one or more top-level entity types and the finest-grained entity type need to be defined. All fine-grained entity types belong to the definition of the top-level concept, and it is necessary to ensure that the entity type of each tail node in the knowledge graph is the finest-grained. This directly determines whether there are subsets under the relationship corresponding to the entity type of the tail node of the knowledge graph. If there are subsets, it does not belong to the finest-grained entity type. Otherwise, it belongs to the finest-grained entity type.

[0185] ③ Relationship evaluation: mainly includes relationship uniqueness detection. For each entity, there should be only one relationship of the same type. There cannot be two relationships with different names but expressing the same meaning at the same time. The relationship that may exist is found out through rule verification. For an entity E1, the set of all its subset entities is S1. For entities of the same type in S1, the relationship type between them and the current entity E1 is judged. After all entities under each entity type in S1 are judged, if there are multiple relationships with different names under the corresponding entity type, there may be a situation where the relationship is not unique. These relationships need to be screened out for review and correction.

[0186] Evaluation of off-graph knowledge: The structure of the reasoning consistency evaluation model is defined as follows:

[0187] The structure of the model is linear layer + multi-head matrix calculation layer + fully connected layer.

[0188] The input of the model is the concatenation of the head entity + relation + tail entity extracted by the keyword extraction model. Then, the linear layer is used to convert the input characters into multi-dimensional vector representation. The specific calculation is as follows:

[0189] =x* +b;

[0190] Among them, x represents the digital id of the input character Token, W represents the weight matrix of the linear layer, T represents the matrix transposition, and b represents the bias weight matrix of the linear layer. It is the final output result of the linear layer.

[0191] The result of the linear layer calculation is input into the multi-head matrix calculation layer, and the calculation of the multi-head matrix calculation layer is:

[0192] =( * +b1)*( * +b2)*( * +b2);

[0193] in, is the output result after the linear layer, , 2, 3 are different weight matrices, b1, b2, b3 are different bias matrices, Compute the final output for the multi-head matrix computation layer.

[0194] After that, it passes through the MLP multi-layer perceptron to perform a binary classification judgment on whether the triple is valid. The softmax function is used to convert the result into a value between 0 and 1 to obtain the final judgment result on whether the triple is valid, that is, the reasoning consistency evaluation result. The final result is:

[0195] result = softmax(MLP( ));

[0196] Next, we screened the evaluation results, and for data with consistency evaluation results below 0.95, we conducted manual scoring and review.

[0197] The results of the above evaluations are then returned to the front end through the http interface to display the evaluation results for users to view. Through this knowledge graph review system, the accuracy and shortcomings of the knowledge in the big model question and answer results can be understood, so as to take the next step of targeted correction measures such as constructing relevant data, which will help to better judge the big model and improve the accuracy and reliability of the big model's reply knowledge.

[0198] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large model answer effect evaluation method based on knowledge graph, characterized in that: include: Build a keyword extraction model, including: Pre-process multiple questions; The preprocessed sentences are used to obtain keyword nodes through convolution calculation; Extracting keyword attributes through the keyword nodes; screening each keyword attribute according to the attention influence score of the keyword attribute of each keyword node; constructing a context graph using the screened keyword attributes, and classifying the keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes; The multiple groups of keyword head nodes and keyword tail nodes extracted by the keyword extraction model are entities, and the classification results of each group of keyword head nodes and keyword tail nodes are entity relationships to generate a keyword knowledge graph; Evaluating the keyword knowledge graph; Use the big model to answer the questions input by the user; The results of the large model are evaluated.

2. According to the method for evaluating the effect of a large model answer based on a knowledge graph according to claim 1, it is characterized in that: The preprocessing of the multiple questions includes: text processing and vectorization processing; The text processing includes standardizing one or more of the text format, address name and abbreviation vocabulary; The vectorization process includes performing matrix multiplication of the embedding weight matrix and the ID of each character mapping of the question sentence to obtain a sentence vector representation of the question sentence.

3. According to the method for evaluating the effect of a large model answer based on a knowledge graph in claim 1, it is characterized in that: The preprocessed sentence is subjected to convolution calculation to obtain keyword nodes, including: The sentence is split into characters by sliding the window, and each character is convolved and activated by the tanh function; Perform maximum pooling on each sliding window to obtain the features corresponding to the specific filter; The extracted features are activated by the sigmoid activation function; Perform matrix operations on the results before and after activation by the sigmoid activation function to obtain keyword nodes; The keyword nodes are calculated through two fully connected layers.

4. According to the method for evaluating the effect of a large model answer based on a knowledge graph in claim 1, it is characterized in that: The extracting keyword attributes through the keyword node includes: The keyword nodes are subjected to graph network learning through a graph neural network; Performing a pooling operation on the keyword nodes through a pooling layer to extract keyword attributes; The maximum values ​​of the keyword attributes on the last dimension of each keyword are concatenated.

5. According to the method for evaluating the effect of a large model answer based on a knowledge graph according to claim 4, it is characterized in that: The keyword nodes are subjected to graph network learning through a graph neural network, including: Construct the GCN network model, expressed as: ； ； ； in, Indicates that the keyword node after convolution is in The feature vector of the layer; is the degree matrix of the vertex; is a Laplacian matrix; Represents the feature vector output by the keyword node through the convolutional network layer; Indicates Parameters of layer convolution; represents the activation function; is the adjacency matrix of the graph; is the identity matrix, that is, a matrix with 1 on the diagonal and 0 on the rest; is included OK The degree matrix of the columns' vertices; is the vertex The adjacency matrix of the graph .

6. According to the method for evaluating the effect of a large model answer based on a knowledge graph according to claim 5, it is characterized in that: The step of screening each keyword attribute according to the attention influence score of the keyword attribute of each keyword node includes: Calculate the self-attention score of each keyword node, expressed as: ； in, Represents the self-attention score of each keyword node; are the parameters of the convolution of the pooling layer, represents the activation function; The self-attention score of each keyword attribute is normalized to obtain the attention influence score, which is expressed as: ； in, represents the attention impact score of each keyword attribute after normalization; Indicates The self-attention scores of the keyword attributes; The natural constant e Power; Screening the keyword attributes that meet the threshold specifically includes: ； in, To meet the minimum score for attention impact; yes The standard deviation of is a tunable hyperparameter.

7. According to the method for evaluating the effect of a large model answer based on a knowledge graph in claim 1, it is characterized in that: After the keyword nodes are classified by the multi-layer perceptron, the method further includes: The softmax function is used to calculate the probabilities of different classification results to obtain the final multiple groups of keyword head nodes and keyword tail nodes.

8. According to the method for evaluating the effect of a large model answer based on a knowledge graph in claim 1, it is characterized in that: The evaluating of the keyword knowledge graph includes: any one or more of attribute evaluation, concept evaluation and relationship evaluation; The attribute evaluation includes any one or both of repetitiveness detection and abnormal character detection; the repetitiveness detection is whether the name of the entity is repeated; the abnormal character detection is whether the repetitive rate of each character in the entity is abnormal; The concept evaluation is to detect whether the keyword tail node of the entity is the most fine-grained entity; The relationship evaluation is to evaluate whether the keyword tail node of the same relationship as the keyword head node is unique.

9. According to the method for evaluating the effect of a large model answer based on a knowledge graph in claim 1, it is characterized in that: The step of evaluating the result of the large model comprises: The structure of the reasoning consistency evaluation model is defined to include a linear layer, a multi-head matrix calculation layer, and a fully connected layer in sequence; The input of the linear layer includes multiple groups of keyword head nodes, relations, and keyword tail nodes, which convert the input characters into multi-dimensional vector representations; After the result calculated by the linear layer is input into the multi-head matrix calculation layer, it is subjected to a binary classification judgment of whether the relationship is established through the fully connected layer, and the result is normalized by using the softmax function to obtain the final judgment result of whether the relationship is established; Screening of uncertain consistency evaluation results for manual review; The determined consistency evaluation results and population audit results are returned to the user's front end through the http interface for display.

10. A large model answer effect evaluation system based on knowledge graph, characterized in that: include: Keyword extraction model, including: A data preprocessing module preprocesses multiple questions; Convolutional network layer, the preprocessed sentences are calculated through convolution to obtain keyword nodes; A neural network layer extracts keyword attributes through the keyword nodes; screens the keyword attributes according to the attention influence score of the keyword attributes of each keyword node; constructs a context graph using the screened keyword attributes, and classifies the keyword nodes through a multi-layer perceptron to obtain multiple groups of keyword head nodes and keyword tail nodes; A knowledge graph generation module, which uses the multiple groups of keyword head nodes and keyword tail nodes extracted by the keyword extraction model as entities, and the classification results of each group of keyword head nodes and keyword tail nodes as entity relationships to generate a keyword knowledge graph; A graph evaluation module, which evaluates the keyword knowledge graph; The big model question answering module inputs the user's question into the big model for question answering; The large model evaluation module evaluates the results of the large model.

Images