Large model evaluation method, system and device, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a large model evaluation method, system and device, a storage medium and a program product, and the method comprises the steps: obtaining interaction data of each interaction round in a complete session of a to-be-evaluated interaction large model, and extracting evaluation indexes of a plurality of evaluation dimensions from the interaction data; the interaction data comprises an interaction text input by a user, a feedback text generated by the interaction large model according to the interaction text and user evaluation; evaluating the interactive large model on each evaluation dimension based on the evaluation index to obtain an evaluation score of the interactive large model on each evaluation dimension; and performing weighted summation on the evaluation scores of the evaluation dimensions to obtain an evaluation score of the interaction maturity of the interaction large model so as to determine whether the dialogue system based on the evaluated interaction large model has the online capability or not. And rapid and complete multi-dimensional evaluation is carried out on the interaction large model based on multiple rounds of interaction, so that the manpower and time cost required by the evaluation of the interaction large model is saved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a large model evaluation method, system, device, storage medium and program product. Background Art

[0002] At present, the evaluation of interactive big models in big model-based dialogue systems mainly relies on manual annotation evaluation and user-based feedback evaluation. Users evaluate the interactive capabilities of big models based on their experience of using big model dialogues, and the dialogue system of interactive big models is evaluated based on user evaluations. A dialogue of a big model may involve multiple rounds of interaction, and the big model can interact based on the context. However, the existing evaluation method is based on a single round of interaction, which is greatly affected by many factors such as user evaluation feedback coverage, user feedback positivity, and user feedback completeness. It also requires a lot of manpower and time costs, making it difficult to fully and quickly evaluate whether the dialogue system has the ability to go online. Summary of the invention

[0003] The present application provides a large model evaluation method, system, device, storage medium and program product to solve the defects in the prior art that the evaluation of large models based on single-round interaction is carried out by relying on manual annotation and user feedback, which requires a lot of manpower and time costs and is difficult to fully and quickly evaluate whether the dialogue system has the ability to go online.

[0004] The present application provides a large model evaluation method, comprising: Obtaining interaction data of each interaction round in a complete conversation of the interaction model to be evaluated, and extracting evaluation indicators of multiple evaluation dimensions from the interaction data; Based on the evaluation index, the interactive large model is evaluated in each of the evaluation dimensions to obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The evaluation scores of the evaluation dimensions are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction macro model.

[0005] In one embodiment, the multiple evaluation dimensions include a business dimension, and the evaluation index includes a business maturity index; based on the evaluation index, evaluating the interactive big model on each of the evaluation dimensions to obtain an evaluation score of the interactive big model on each of the evaluation dimensions includes: Obtaining a first indicator weight corresponding to a business maturity indicator in the evaluation indicator; the business maturity indicator includes at least one of a problem solving rate, interaction accuracy, interaction completeness, interaction friendliness, and interaction timeliness; The business maturity indicators are weighted and summed based on the first indicator weights to evaluate the interaction big model in the business dimension, thereby obtaining an evaluation score of the business maturity of the interaction big model in the business dimension.

[0006] In one embodiment, the multiple evaluation dimensions include a technical dimension, and the evaluation index includes a technical maturity index; based on the evaluation index, the interactive big model is evaluated on each of the evaluation dimensions to obtain an evaluation score of the interactive big model on each of the evaluation dimensions, including: Obtaining a second indicator weight corresponding to a technology maturity indicator in the evaluation indicator; the technology maturity indicator includes at least one of intent recognition accuracy, dialogue state judgment accuracy, information extraction accuracy, emotion recognition accuracy, and context rewriting accuracy; The technical maturity index is weighted and summed based on the second index weight to evaluate the interactive big model in the technical dimension, so as to obtain an evaluation score of the technical maturity of the interactive big model in the technical dimension.

[0007] In one embodiment, the multiple evaluation dimensions include an engineering dimension, and the evaluation index includes an engineering maturity index; based on the evaluation index, the interactive big model is evaluated on each of the evaluation dimensions to obtain an evaluation score of the interactive big model on each of the evaluation dimensions, including: Obtaining a third indicator weight corresponding to an engineering maturity indicator in the evaluation indicator; the engineering maturity indicator includes at least one of an average response time and a rejection interaction ratio; The engineering maturity index is weighted and summed based on the third index weight to evaluate the interactive big model in the engineering dimension, so as to obtain an evaluation score of the engineering maturity of the interactive big model in the engineering dimension.

[0008] In one embodiment, the interaction data includes interaction text input by the user, feedback text generated by the interaction macro model according to the interaction text, and user evaluation; the step of extracting evaluation indicators of multiple evaluation dimensions from the interaction data includes: Extracting user intention, key information of user needs and user emotions from the interaction text in the interaction data; Extracting structured user intention features and user emotion features according to the user intention, the key information of the user demand and the user emotion; Based on the user intention feature and the user emotion feature, evaluation indicators of multiple evaluation dimensions are extracted from the feedback text and user evaluation in the interaction data.

[0009] In one embodiment, the weighted sum of the evaluation scores of the evaluation dimensions to obtain the evaluation score of the interaction maturity of the interaction macro model includes: Determining the evaluation weight of each evaluation dimension according to the interaction type in the interaction data; the interaction type includes task-based, question-and-answer-based, and chat-based; The evaluation scores of the evaluation dimensions are weighted and summed based on the evaluation weights to obtain the evaluation score of the interaction maturity of the interaction macro model.

[0010] The present application also provides a large model evaluation system, including: A data collection module is used to obtain the interaction data of each interaction round in the complete conversation of the interaction macro model to be evaluated, and extract evaluation indicators of multiple evaluation dimensions from the interaction data; the interaction data includes the interaction text input by the user, the feedback text generated by the interaction macro model according to the interaction text, and the user evaluation; A multi-dimensional evaluation module, used to evaluate the interactive large model in each of the evaluation dimensions based on the evaluation indicators, and obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The overall evaluation module is used to perform weighted summation on the evaluation scores of each evaluation dimension to obtain an evaluation score of the interaction maturity of the interaction macro model.

[0011] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the large model evaluation method as described above is implemented.

[0012] The present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the large model evaluation methods described above.

[0013] The present application also provides a computer program product, including a computer program, which implements any of the large model evaluation methods described above when executed by a processor.

[0014] The big model evaluation method, system, device, storage medium and program product provided by the present application extract evaluation indicators of multiple evaluation dimensions from the interaction data of the interactive big model, and evaluate the interactive big model in multiple evaluation dimensions based on the extracted evaluation indicators, and perform weighted summation on the evaluation scores on each evaluation dimension to obtain an evaluation score for the interaction maturity of the interactive big model. The interactive big model can be completely and quickly evaluated based on multiple rounds of interaction to determine whether the dialogue system based on the evaluated interactive big model has the ability to go online, thereby saving the manpower and time costs required for the evaluation of the interactive big model. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1 It is a flow chart of the large model evaluation method provided in the embodiment of the present application.

[0017] Figure 2 It is a structural schematic diagram of the large model evaluation system provided by the present invention.

[0018] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] The embodiment of the present application provides a large model evaluation method, which evaluates the interaction maturity of the interactive large model in multiple dimensions, and can quickly and completely evaluate whether the dialogue system based on the interactive large model has the ability to go online. Figure 1 , Figure 1 is a flow chart of a large model evaluation method provided in an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps: Step 101, obtaining interaction data of each interaction round in a complete conversation of an interaction macro model to be evaluated, and extracting evaluation indicators of multiple evaluation dimensions from the interaction data; Step 102, based on the evaluation index, the interactive large model is evaluated on each of the evaluation dimensions to obtain the evaluation score of the interactive large model on each of the evaluation dimensions; Step 103, performing weighted summation on the evaluation scores of the evaluation dimensions to obtain an evaluation score of the interaction maturity of the interaction macro model.

[0021] The interaction data of each interaction round in the complete conversation of the interaction model to be evaluated is obtained, and evaluation indicators of multiple evaluation dimensions are extracted from the obtained interaction data. The obtained interaction data includes the interaction text input by the user, the feedback text generated by the interaction model according to the interaction text input by the user, and the user evaluation.

[0022] Optionally, the user inputs an interactive text to ask a question, and the interactive model generates a feedback text based on the interactive text input by the user to answer the question, which completes a round of interaction. In the interactive process of a complete conversation, one or more rounds of interaction may be generated. That is, a complete conversation may include one or more rounds of interaction.

[0023] The user evaluation may include the user's evaluation of each complete session, or may include the user's evaluation of each interaction round in the complete session.

[0024] Evaluation indicators of multiple evaluation dimensions are extracted from the acquired interaction data. Optionally, each evaluation dimension includes one or more evaluation indicators. Feature extraction is performed on the interaction data based on different evaluation dimensions to obtain evaluation indicators of each evaluation dimension. Optionally, the extracted evaluation indicators may include single interaction indicators extracted based on each interaction round, and multiple interaction statistical indicators extracted based on multiple interaction rounds.

[0025] Based on the extracted evaluation indicators, the interactive large model is evaluated on each evaluation dimension to obtain the evaluation score of the interactive large model on each evaluation dimension. Optionally, for each evaluation dimension, the interactive large model is evaluated based on the evaluation indicator corresponding to the evaluation dimension to obtain the evaluation score of the interactive large model on the evaluation dimension. Optionally, based on the extracted evaluation indicators, a multi-dimensional evaluation is performed on the interactive large model from different evaluation dimensions to obtain the evaluation score of the interactive large model on each evaluation dimension.

[0026] The evaluation scores of each evaluation dimension are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction macro model. The evaluation weights of each evaluation dimension used for weighted summing of the evaluation scores can be preset or determined based on the interaction data.

[0027] Optionally, the evaluation weight of each evaluation dimension is determined according to the proportion of the interaction data of the complete session in all the interaction data.

[0028] In this embodiment, by extracting evaluation indicators of multiple evaluation dimensions from the interaction data of the interactive big model, and evaluating the interactive big model in multiple evaluation dimensions based on the extracted evaluation indicators, the evaluation scores on each evaluation dimension are weighted and summed to obtain the evaluation score of the interaction maturity of the interactive big model. The interactive big model can be completely and quickly evaluated based on multiple rounds of interaction to determine whether the dialogue system based on the evaluated interactive big model has the ability to go online, thereby saving the manpower and time costs required for the evaluation of the interactive big model.

[0029] Optionally, the interaction data obtained includes the interaction text input by the user in each round of interaction, the feedback text generated by the interaction model based on the interaction text input by the user, and the user's evaluation of the interaction process. In step 101, the evaluation indicators of multiple evaluation dimensions are extracted from the interaction data, which may also include: Step 201, extracting user intention, key information of user needs and user emotions from the interaction text in the interaction data; Step 202, extracting structured user intention features and user emotion features according to the user intention, the key information of the user demand and the user emotion; Step 203: extracting evaluation indicators of multiple evaluation dimensions from the feedback text and user evaluation in the interaction data based on the user intention feature and the user emotion feature.

[0030] User intention, key information of user needs and user emotions are extracted from the interaction text in the interaction data, and structured user intention features and user emotion features are extracted based on the user intention, key information of user needs and user emotions.

[0031] In one embodiment, user intention, key information of user needs and user emotions are extracted from the interactive text input by the user, and the user intention, key information of user needs and user emotions are mapped into structured features through a word vector mapping model to obtain user intention features and user emotion features. The word vector mapping model may be a word2vec model or the like.

[0032] Based on the extracted user intention features and user emotion features, evaluation indicators of multiple evaluation dimensions are extracted from the feedback text and user evaluation in the interaction data.

[0033] Optionally, after preprocessing such as deduplication cleaning and clustering, the extracted user intention features and user emotion features can be used to construct and save a user intention database and a user emotion database to facilitate subsequent identification of user intentions and user emotions.

[0034] Furthermore, the multiple evaluation dimensions include at least two of a business dimension, a technical dimension, and an engineering dimension, and the evaluation indicators of the multiple evaluation dimensions can be represented in the form of indicator scores.

[0035] If the multiple evaluation dimensions include a business dimension, the extracted evaluation indicators of the multiple evaluation dimensions include a business maturity indicator. Based on this, step 102 may also include: Step 301, obtaining a first indicator weight corresponding to a business maturity indicator in the evaluation indicator; the business maturity indicator includes at least one of problem solving rate, interaction accuracy, interaction completeness, interaction friendliness and interaction timeliness; Step 302: Perform weighted summation on the business maturity indicators based on the first indicator weights to evaluate the interaction big model in the business dimension, and obtain an evaluation score of the business maturity of the interaction big model in the business dimension.

[0036] Obtain a first indicator weight corresponding to a business maturity indicator in the evaluation indicator, wherein the business maturity indicator includes at least one of a problem solving rate, interaction accuracy, interaction integrity, interaction friendliness, and interaction timeliness. The first indicator weight includes at least one of a problem solving rate weight, an interaction accuracy weight, an interaction integrity weight, an interaction friendliness weight, and an interaction timeliness weight.

[0037] Furthermore, the business maturity indicators are weighted and summed based on the first indicator weight, so as to evaluate the interactive big model in the business dimension and obtain an evaluation score of the business maturity of the interactive big model in the business dimension.

[0038] Optionally, the problem solving rate is the ratio of the number of successfully solved problems to the total number of problems. The total number of problems is the total number of problems raised by users to the interactive model. Whether the problem is successfully solved can be determined based on user evaluation. That is: .

[0039] The interaction accuracy can be evaluated from the aspects of the relevance between the feedback content of the interaction model and the user's question content, whether the interaction is accurate, and whether the interaction content is ambiguous. The ability of the interaction model to interact accurately can be measured in the form of a piecewise function. For example, if the score range of the interaction accuracy indicator is [0, 100], then: ; That is, if the interactive content is relevant to the user's question, accurate, unambiguous, and non-toxic, the index score is 100; if the interactive content is relevant to the user's question but cannot directly solve the user's problem, accurate, and unambiguous, the index score is 75; if the interactive content is to convert the user's question to a manual reply, accurate, and unambiguous, the index score is 50; if the interactive content is to refuse to recognize the user's question, accurate, and unambiguous, the index score is 25; if the interactive content is irrelevant to the user's question or contains hallucinations, the index score is 0; "&" represents logical "and", and "||" represents logical "or".

[0040] The interaction accuracy score is obtained by evaluating each interaction round in a complete conversation. If a complete conversation includes multiple rounds of interaction, represents the total number of interaction turns in a complete session, represents the interaction accuracy of each round of interaction. The interaction accuracy score of this session is the average of the interaction accuracy scores of each round of interaction: .

[0041] Interaction integrity is obtained by evaluating whether all user interaction requests generate interactions during a complete session, whether the interaction model provides complete information required by the user, whether the interaction semantics are coherent, and whether the interaction as a whole is consistent. The interaction integrity of the interaction model can also be measured in the form of a piecewise function. For example, if the score range of the interaction integrity indicator is [0, 100], then: ; That is, if the answer is complete, the information is complete, the semantics is coherent, and the interaction is consistent, the index score is 100; if the answer is incomplete, the information is complete, the semantics is coherent, and the interaction is consistent, the index score is 75; if the answer is incomplete, the information is incomplete, the semantics is coherent, and the interaction is consistent, the index score is 50; if the answer is incomplete, the information is incomplete, the semantics is incoherent, and the interaction is consistent, the index score is 25; if the answer is incomplete, the information is incomplete, the semantics is incoherent, and the interaction is inconsistent, the index score is 0.

[0042] Interaction friendliness is obtained by evaluating whether the interaction content is friendly, approachable, easy to understand, polished, and emotionally soothing during a round of interaction. Similarly, a discrete piecewise function can be used to evaluate whether the interaction model can provide friendly interaction during the interaction process. For example, the score range of the interaction friendliness index is [0, 100], then: ; That is, if the interactive content is friendly, approachable, easy to understand, polished, and emotionally soothing, the index score is 100; if the interactive content is friendly, approachable, easy to understand, polished, and not emotionally soothing, the index score is 75; if the interactive content is friendly, approachable, easy to understand, not polished, and not emotionally soothing, the index score is 50; if the interactive content format is standardized, monotonous, professional, not polished, and not emotionally soothing, the index score is 25; if the interactive content format is chaotic, monotonous, professional, not polished, and not emotionally soothing, the index score is 0. The index score of interactive friendliness is obtained based on the evaluation of a single round of interaction. In a complete conversation, if there are multiple rounds of interaction, the index score of interactive friendliness of this conversation is the average of the index scores of interactive friendliness of each round of interaction, that is: .

[0043] Interaction timeliness characterizes the response speed of the interactive model to the user's questions. The interactive model can generate feedback text and determine the response time of the user's questions. The response time is the difference between the time when the interactive model gives feedback to the user and the time when the user sends the interactive text for asking questions. For example, the indicator score range of interaction timeliness is [0, 100], then: ; That is, if the response time does not exceed 1 second, the indicator score is 100; if the response time is greater than 1 second and less than 3 seconds, the indicator score is 80; if the response time is greater than 3 seconds and less than 5 seconds, the indicator score is 60; if the response time is greater than 5 seconds and less than 7 seconds, the indicator score is 40; if the response time is greater than 7 seconds and less than 10 seconds, the indicator score is 20; if the response time is greater than 10 seconds, the indicator score is 0. The timeliness of interaction is evaluated based on a single round of interaction. When there are multiple rounds of interaction in a complete conversation, the timeliness score of this conversation is the average of the timeliness scores of each round of interaction, that is: ; If the multiple evaluation dimensions include a technical dimension, the extracted evaluation indicators of the multiple evaluation dimensions include a technical maturity indicator. Based on this, step 102 may also include: Step 401, obtaining a second indicator weight corresponding to a technology maturity indicator in the evaluation indicator; the technology maturity indicator includes at least one of the following: intention recognition accuracy, dialogue state judgment accuracy, information extraction accuracy, emotion recognition accuracy, and context rewriting accuracy; Step 402: Perform a weighted summation on the technical maturity index based on the second index weight to evaluate the interactive big model in the technical dimension, and obtain an evaluation score of the technical maturity of the interactive big model in the technical dimension.

[0044] The second indicator weight corresponding to the technology maturity indicator in the evaluation indicator is obtained, and the technology maturity indicator includes at least one of the following: intention recognition accuracy, dialogue state judgment accuracy, information extraction accuracy, emotion recognition accuracy, and context rewriting accuracy. The second indicator weight includes at least one of the following: intention recognition accuracy weight, dialogue state judgment accuracy weight, information extraction accuracy weight, emotion recognition accuracy weight, and context rewriting accuracy weight.

[0045] The technical maturity index is weighted and summed based on the weight of the second index to evaluate the interactive large model in the technical dimension, thereby obtaining an evaluation score of the technical maturity of the interactive large model in the technical dimension.

[0046] Optionally, the intent recognition accuracy rate refers to the ratio of the number of times the large model accurately recognizes the user's question intent to the total number of times the user asks questions: ; The accuracy of intent recognition can be evaluated based on the complete conversation. In a complete conversation, the ratio of the number of intents contained in the user's questions correctly identified by the interactive model to the total number of user questions in this conversation is the accuracy of intent recognition for this conversation.

[0047] The accuracy of dialogue state judgment is used to characterize the ratio of the number of times the interactive model accurately identifies the dialogue state of the user's question to the total number of user questions, that is: ; The conversation status includes clear user intent, multiple user intents, unfinished user question, unclear user expression, idle chat, sensitive input, and rejection.

[0048] The accuracy of information extraction refers to the proportion of interaction rounds in which the interactive model accurately extracts key information from user questions to the total number of interaction rounds. The indicator score of the accuracy of information extraction is calculated based on this proportion: ; The accuracy of information extraction can be obtained based on the evaluation of the complete conversation. The accuracy of information extraction for a conversation is the ratio of the number of interactions in which key information is correctly extracted in this conversation to the total number of interactions in this conversation.

[0049] The emotion recognition accuracy rate is the ratio of the number of interaction rounds in which the interactive model accurately recognizes the user's emotions in each round of interaction to the total number of interaction rounds. The emotion recognition accuracy index score is calculated based on this ratio.

[0050] ; In a complete conversation, the emotion recognition accuracy is the ratio of the number of interaction rounds in which the interaction model correctly identifies the user's emotions to the total number of interaction rounds in this conversation.

[0051] Contextual rewriting refers to the contextual rewriting of the interactive text fuzzily input by the user by the interactive model to make the semantics of the interactive text input by the user clear. The index score of contextual rewriting accuracy is an indicator for evaluating the similarity between the contextual rewritten text and the annotated rewritten text. Optionally, the BLEU (Bilingual Evaluation Understudy) score of the contextual text is used as the index score of contextual rewriting accuracy. Specifically, the interactive model rewritten text and the annotated rewritten text are first split into n-gram sequences, n=1,2,3,..., , is the maximum length of n-gram; then, the precision of n-gram phrases in the rewritten text of n-gram contexts with different lengths is calculated in turn, and the calculation method is shown in Formula 1: ; (1)

[0052] in, is the precision score of the interactive large model on n-gram phrases of length n, is the number of matches between the n-gram phrases in the context rewritten text and the annotated rewritten text, The total number of n-gram phrases in the context that are rewritten.

[0053] Furthermore, to avoid incorrect n-gram matching scores due to long texts, the precision score is adjusted using the brevity penalty. As a penalty term, the context rewriting accuracy index score range is [0,100], and the BLEU score of the context rewriting is calculated as shown in the following formula 2: . (2)

[0054] If the multiple evaluation dimensions include the engineering dimension, the extracted evaluation indicators of the multiple evaluation dimensions include the engineering maturity indicator. Based on this, step 102 may also include: Step 501, obtaining a third indicator weight corresponding to the engineering maturity indicator in the evaluation indicator; the engineering maturity indicator includes at least one of an average response time and a rejection interaction ratio; Step 502: Perform weighted summation on the engineering maturity index based on the third index weight to evaluate the interactive big model in the engineering dimension, and obtain an evaluation score of the engineering maturity of the interactive big model in the engineering dimension.

[0055] Obtain a third indicator weight corresponding to the engineering maturity indicator in the evaluation indicator, where the engineering maturity indicator includes at least one of an average response time and a rejection interaction ratio. The third indicator weight includes at least one of an average response time weight and a rejection interaction ratio weight.

[0056] Based on the weight of the third indicator, the engineering maturity index is weighted and summed, so as to evaluate the interactive large model in the engineering dimension and obtain the evaluation score of the engineering maturity of the interactive large model in the engineering dimension.

[0057] Optionally, the average response time is an indicator used to evaluate the inference performance of the interactive large model. In a complete session, the interaction response time from the user inputting the interactive text to the interactive large model returning the feedback text in each round of interaction in this session is first calculated, and then the average response time of each round of interaction in this session is calculated based on the total number of interaction rounds in this session.

[0058] The percentage of rejected interactions is an indicator used to evaluate the ability of the interactive large model to handle concurrency. K interactive requests are randomly selected from the test data set, and K=5 is optionally input into the interactive large model at the same time. If the interactive large model does not respond to an interactive request for more than the preset time due to reasons such as the user's question involving sensitive questions, it is considered as rejected. The percentage of rejected interactions is the ratio of the number of rejected interactions in the user's interactive requests to the total number of interactive requests. The indicator score of the percentage of rejected interactions is calculated based on this ratio.

[0059] Based on the evaluation scores of multiple evaluation dimensions such as business dimension, technical dimension and engineering dimension, the evaluation scores of business maturity, technical maturity and engineering maturity are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction big model. According to the evaluation score of the interaction maturity, it can be determined whether the dialogue system based on the interaction big model has the ability to go online.

[0060] In one embodiment, the evaluation weight of the weighted sum of the evaluation scores of each evaluation dimension is determined according to the interaction type, and step 103 may further include: Step 601, determining the evaluation weight of each evaluation dimension according to the interaction type in the interaction data; the interaction type includes task type, question-and-answer type and chat type; Step 601: Perform weighted summation on the evaluation scores of the evaluation dimensions based on the evaluation weights to obtain an evaluation score of the interaction maturity of the interaction macro model.

[0061] When performing weighted summation on the evaluation scores of each evaluation dimension, first determine the evaluation weight of each evaluation dimension based on the interaction type in the acquired interaction data. For different interaction types, the evaluation weight of the same evaluation dimension is different, and the evaluation weights of different evaluation dimensions under the same interaction type are different.

[0062] Optionally, the interaction types corresponding to different sessions may be the same or different, and in multiple interaction rounds in the same session, the interaction types corresponding to different interaction rounds may be the same or different.

[0063] Among them, the interaction types include task-based, question-and-answer, and chat-based. Task-based interaction refers to users completing designated business through the guidance of the dialogue big model; question-and-answer interaction refers to users getting answers from the dialogue big model by asking questions to the dialogue big model; chat-based interaction refers to non-task-based and non-question-and-answer-based dialogue interaction between users and the dialogue big model.

[0064] Based on the evaluation weight corresponding to each evaluation dimension, the evaluation score of each evaluation dimension is weighted and summed to obtain the evaluation score of the interaction maturity of the interactive large model.

[0065] Optionally, if the evaluation dimensions include business dimension, technical dimension and engineering dimension, the evaluation weight includes a first evaluation weight corresponding to the business dimension, a second evaluation weight corresponding to the technical dimension and a third evaluation weight corresponding to the engineering dimension. The evaluation score of the interaction maturity of the interaction big model is shown in the following formula 3: ; (3)

[0066] in, Indicates the evaluation score of the interaction maturity of the interaction model, , and They are the evaluation weights of business dimension, technical dimension and engineering dimension respectively. is the evaluation score of the business maturity of the business dimension, is the evaluation score of the technical maturity of the technical dimension, It is the evaluation score of engineering maturity in the engineering dimension.

[0067] For example, referring to Table 1, in the business dimension, the indicator weight of the problem resolution rate is 0.25; the indicator weight of the interaction accuracy is 0.25; the indicator weight of the interaction completeness is 0.2; the indicator weight of the interaction friendliness is 0.2; the indicator weight of the interaction timeliness is 0.1; the weighted sum of the indicator scores of each business maturity indicator in the business dimension is the business maturity score of the interaction model: ; (4)

[0068] In formula 4, is the indicator weight of the i-th indicator in the business maturity indicator, That is, a set of indicators of business maturity such as problem solving rate, interaction accuracy, interaction completeness, interaction friendliness and interaction timeliness. is the indicator score of the i-th indicator in the business maturity indicator.

[0069] Furthermore, as shown in Table 1, for the technical dimension, the indicator weight of intent recognition accuracy is 0.25; the indicator weight of dialogue state judgment accuracy is 0.2; the indicator weight of information extraction accuracy is 0.25; the indicator weight of emotion recognition accuracy is 0.1; the indicator weight of context rewriting accuracy is 0.2; the weighted sum of the indicator scores of each technical maturity indicator in the technical dimension is the evaluation score of the technical maturity of the interactive large model: ; (5)

[0070] In formula 5, is the indicator weight of the jth indicator in the technology maturity index, That is, a set of technical maturity indicators such as intent recognition accuracy, dialogue state judgment accuracy, information extraction accuracy, emotion recognition accuracy, and context rewriting accuracy. It is the indicator score of the jth indicator in the technology maturity index.

[0071] For the engineering dimension, the indicator weight of average response time is 0.5; the indicator weight of rejected interaction ratio is 0.5; the weighted sum of the indicator scores of each engineering maturity indicator in the engineering dimension is the evaluation score of the engineering maturity of the interactive large model: ; (6)

[0072] In Formula 6, is the indicator weight of the kth indicator in the technology maturity index, That is, a set of indicators of engineering maturity such as average response time and percentage of rejected interactions. is the indicator score of the kth indicator in the engineering maturity indicator.

[0073] Table 1:

[0074] Exemplarily, in Table 1, the evaluation weight of the business dimension is 0.5; the evaluation weight of the technical dimension is 0.3; and the evaluation weight of the engineering dimension is 0.2. As shown in Formula 3, the evaluation scores of the business maturity, the technical maturity, and the engineering maturity calculated in Formulas 4 to 6 are weighted and summed to obtain the evaluation score of the interaction maturity of the interactive large model.

[0075] In this embodiment, based on the indicator score and indicator weight of each evaluation indicator, the interactive big model is evaluated in multiple dimensions to obtain the evaluation score of each evaluation dimension, and then the evaluation weight of each evaluation dimension is determined according to the interaction type, and the evaluation score of each evaluation dimension is weighted and summed to obtain the evaluation score of the maturity of the interactive big model, thereby realizing a rapid and complete evaluation of the interactive big model.

[0076] Furthermore, in the relevant evaluation of the big model, the hallucination problem of the model is not considered, and the big model is a generative model, which may generate wrong answers to user questions, or generate corresponding answers to all user questions. When sensitive issues are involved, the responses generated by the big model may cause data leakage and pose security risks. In this embodiment, by introducing evaluation indicators related to hallucination phenomena and sensitive issues such as interaction accuracy and rejection interaction ratio, the interaction maturity of the interactive big model is evaluated, so that the interactive big model can be more adapted to the actual business, and the accuracy and completeness of the evaluation of the interactive big model are further improved.

[0077] The large model evaluation system provided in an embodiment of the present application is described below. The large model evaluation system described below and the large model evaluation method described above can be referenced to each other.

[0078] Reference Figure 2 The large model evaluation system provided in the embodiment of the present application includes: The data collection module 10 is used to obtain the interaction data of each interaction round in the complete conversation of the interaction macro model to be evaluated, and extract evaluation indicators of multiple evaluation dimensions from the interaction data; the interaction data includes the interaction text input by the user, the feedback text generated by the interaction macro model according to the interaction text, and the user evaluation; A multi-dimensional evaluation module 20, for evaluating the interactive large model in each of the evaluation dimensions based on the evaluation index, and obtaining an evaluation score of the interactive large model in each of the evaluation dimensions; The overall evaluation module 30 is used to perform weighted summation on the evaluation scores of each evaluation dimension to obtain an evaluation score of the interaction maturity of the interaction macro model.

[0079] In one embodiment, the multiple evaluation dimensions include a business dimension, the evaluation index includes a business maturity index; the multi-dimensional evaluation module 20 includes a business maturity evaluation module for: Obtaining a first indicator weight corresponding to a business maturity indicator in the evaluation indicator; the business maturity indicator includes at least one of a problem solving rate, interaction accuracy, interaction completeness, interaction friendliness, and interaction timeliness; The business maturity indicators are weighted and summed based on the first indicator weights to evaluate the interaction big model in the business dimension, thereby obtaining an evaluation score of the business maturity of the interaction big model in the business dimension.

[0080] In one embodiment, the multiple evaluation dimensions include a technical dimension, the evaluation index includes a technical maturity index; the multi-dimensional evaluation module 20 includes a technical maturity evaluation module for: Obtaining a second indicator weight corresponding to a technology maturity indicator in the evaluation indicator; the technology maturity indicator includes at least one of intent recognition accuracy, dialogue state judgment accuracy, information extraction accuracy, emotion recognition accuracy, and context rewriting accuracy; The technical maturity index is weighted and summed based on the second index weight to evaluate the interactive big model in the technical dimension, so as to obtain an evaluation score of the technical maturity of the interactive big model in the technical dimension.

[0081] In one embodiment, the multiple evaluation dimensions include an engineering dimension, the evaluation index includes an engineering maturity index; the multi-dimensional evaluation module 20 includes an engineering maturity evaluation module for: Obtaining a third indicator weight corresponding to an engineering maturity indicator in the evaluation indicator; the engineering maturity indicator includes at least one of an average response time and a rejection interaction ratio; The engineering maturity index is weighted and summed based on the third index weight to evaluate the interactive big model in the engineering dimension, so as to obtain an evaluation score of the engineering maturity of the interactive big model in the engineering dimension.

[0082] In one embodiment, the data acquisition module 10 is further used for: Extracting user intention, key information of user needs and user emotions from the interaction text in the interaction data; Extracting structured user intention features and user emotion features according to the user intention, the key information of the user demand and the user emotion; Based on the user intention feature and the user emotion feature, evaluation indicators of multiple evaluation dimensions are extracted from the feedback text and user evaluation in the interaction data.

[0083] In one embodiment, the overall evaluation module 30 is further used to: Determining the evaluation weight of each evaluation dimension according to the interaction type in the interaction data; the interaction type includes task-based, question-and-answer-based, and chat-based; The evaluation scores of the evaluation dimensions are weighted and summed based on the evaluation weights to obtain the evaluation score of the interaction maturity of the interaction macro model.

[0084] Figure 3 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 3 As shown, the electronic device may include: a processor 310, a communications interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communications interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call the logic instructions in the memory 330 to execute the large model evaluation method, which includes: Obtaining interaction data of each interaction round in a complete conversation of the interaction macro model to be evaluated, and extracting evaluation indicators of multiple evaluation dimensions from the interaction data; the interaction data includes interaction text input by the user, feedback text generated by the interaction macro model according to the interaction text, and user evaluation; Based on the evaluation index, the interactive large model is evaluated in each of the evaluation dimensions to obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The evaluation scores of the evaluation dimensions are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction macro model.

[0085] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0086] On the other hand, an embodiment of the present application further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the large model evaluation method provided by the above methods, the method includes: Obtaining interaction data of each interaction round in a complete conversation of the interaction macro model to be evaluated, and extracting evaluation indicators of multiple evaluation dimensions from the interaction data; the interaction data includes interaction text input by the user, feedback text generated by the interaction macro model according to the interaction text, and user evaluation; Based on the evaluation index, the interactive large model is evaluated in each of the evaluation dimensions to obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The evaluation scores of the evaluation dimensions are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction macro model.

[0087] In another aspect, the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the large model evaluation method provided by the above methods is implemented, and the method includes: Obtaining interaction data of each interaction round in a complete conversation of the interaction macro model to be evaluated, and extracting evaluation indicators of multiple evaluation dimensions from the interaction data; the interaction data includes interaction text input by the user, feedback text generated by the interaction macro model according to the interaction text, and user evaluation; Based on the evaluation index, the interactive large model is evaluated in each of the evaluation dimensions to obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The evaluation scores of the evaluation dimensions are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction macro model.

[0088] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0089] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A large model evaluation method, characterized in that: include: Obtaining interaction data of each interaction round in a complete conversation of the interaction model to be evaluated, and extracting evaluation indicators of multiple evaluation dimensions from the interaction data; Based on the evaluation index, the interactive large model is evaluated in each of the evaluation dimensions to obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The evaluation scores of the evaluation dimensions are weighted and summed to obtain the evaluation score of the interaction maturity of the interaction macro model.

2. The large model evaluation method according to claim 1, characterized in that: The multiple evaluation dimensions include a business dimension, and the evaluation index includes a business maturity index; based on the evaluation index, the interactive big model is evaluated on each of the evaluation dimensions to obtain an evaluation score of the interactive big model on each of the evaluation dimensions, including: Obtaining a first indicator weight corresponding to a business maturity indicator in the evaluation indicator; the business maturity indicator includes at least one of a problem solving rate, interaction accuracy, interaction completeness, interaction friendliness, and interaction timeliness; The business maturity indicators are weighted and summed based on the first indicator weights to evaluate the interaction big model in the business dimension, thereby obtaining an evaluation score of the business maturity of the interaction big model in the business dimension.

3. The large model evaluation method according to claim 1, characterized in that: The multiple evaluation dimensions include a technical dimension, and the evaluation index includes a technical maturity index; based on the evaluation index, the interactive big model is evaluated on each of the evaluation dimensions to obtain an evaluation score of the interactive big model on each of the evaluation dimensions, including: Obtaining a second indicator weight corresponding to a technology maturity indicator in the evaluation indicator; the technology maturity indicator includes at least one of intent recognition accuracy, dialogue state judgment accuracy, information extraction accuracy, emotion recognition accuracy, and context rewriting accuracy; The technical maturity index is weighted and summed based on the second index weight to evaluate the interactive big model in the technical dimension, so as to obtain an evaluation score of the technical maturity of the interactive big model in the technical dimension.

4. The large model evaluation method according to claim 1, characterized in that: The multiple evaluation dimensions include an engineering dimension, and the evaluation indicators include an engineering maturity indicator; based on the evaluation indicators, the interactive big model is evaluated on each of the evaluation dimensions to obtain an evaluation score of the interactive big model on each of the evaluation dimensions, including: Obtaining a third indicator weight corresponding to an engineering maturity indicator in the evaluation indicator; the engineering maturity indicator includes at least one of an average response time and a rejection interaction ratio; The engineering maturity index is weighted and summed based on the third index weight to evaluate the interactive big model in the engineering dimension, so as to obtain an evaluation score of the engineering maturity of the interactive big model in the engineering dimension.

5. The large model evaluation method according to claim 1, characterized in that: The interaction data includes the interaction text input by the user, the feedback text and the user evaluation generated by the interaction model according to the interaction text; The step of extracting evaluation indicators of multiple evaluation dimensions from the interaction data includes: Extracting user intention, key information of user needs and user emotions from the interaction text in the interaction data; Extracting structured user intention features and user emotion features according to the user intention, the key information of the user demand and the user emotion; Based on the user intention feature and the user emotion feature, evaluation indicators of multiple evaluation dimensions are extracted from the feedback text and user evaluation in the interaction data.

6. The large model evaluation method according to claim 1, characterized in that: The weighted sum of the evaluation scores of the evaluation dimensions to obtain the evaluation score of the interaction maturity of the interaction macro model includes: Determining the evaluation weight of each evaluation dimension according to the interaction type in the interaction data; the interaction type includes task-based, question-and-answer-based, and chat-based; The evaluation scores of the evaluation dimensions are weighted and summed based on the evaluation weights to obtain the evaluation score of the interaction maturity of the interaction macro model.

7. A large model evaluation system, characterized in that: include: A data collection module is used to obtain the interaction data of each interaction round in the complete conversation of the interaction macro model to be evaluated, and extract evaluation indicators of multiple evaluation dimensions from the interaction data; the interaction data includes the interaction text input by the user, the feedback text generated by the interaction macro model according to the interaction text, and the user evaluation; A multi-dimensional evaluation module, used to evaluate the interactive large model in each of the evaluation dimensions based on the evaluation indicators, and obtain the evaluation score of the interactive large model in each of the evaluation dimensions; The overall evaluation module is used to perform weighted summation on the evaluation scores of each evaluation dimension to obtain an evaluation score of the interaction maturity of the interaction macro model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the large model evaluation method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the large model evaluation method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the large model evaluation method according to any one of claims 1 to 6 is implemented.

Images