Assessment information output method and device of large language model, computer equipment and readable storage medium

Abstract

The invention relates to an evaluation information output method and device of a large language model, computer equipment, a computer readable storage medium and a computer program product. The method comprises the steps of obtaining an evaluation data set corresponding to each evaluation link of a preset dialogue state evaluation process; the evaluation data set corresponding to each evaluation link comprises to-be-analyzed statements under different evaluation dimensions and a corresponding dialogue state analysis result set; the dialogue state analysis result set comprises a correct analysis result and at least one analysis result different from the correct analysis result; inputting an evaluation data set corresponding to the evaluation link into a pre-trained large language model, and outputting a target prediction result corresponding to the to-be-analyzed statement; and comparing a correct analysis result corresponding to the to-be-analyzed statement in the evaluation data set with a corresponding target prediction result to obtain evaluation information of the pre-trained large language model in each evaluation link. By adopting the method, the dialogue state perception capability of the large language model can be evaluated more accurately.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, computer-readable storage medium, and computer program product for outputting evaluation information of a large language model. Background Art

[0002] With the rapid development of large language models (LLMs), people are full of expectations for their potential in dialogue state awareness. Although these models have demonstrated significant capabilities in tasks such as question answering, text generation, and automatic reply, research on their dialogue state awareness capabilities remains limited.

[0003] In related technologies, when evaluating the dialogue state perception ability of large language models, they rely on existing datasets. These datasets are often designed as pattern recognition tasks, encouraging models to rely on explicit information and frequent patterns. The evaluation method is relatively simple, resulting in inaccurate evaluation results of the dialogue state perception ability of large language models. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, computer-readable storage medium and computer program product for outputting evaluation information of a large language model that can more accurately evaluate the dialogue state perception ability of the large language model in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for outputting evaluation information of a large language model, comprising:

[0006] Obtaining an evaluation dataset corresponding to each evaluation step of a preset dialogue state evaluation process; the dialogue state evaluation process is a process for evaluating the dialogue state perception capability of a pre-trained large language model; the evaluation dataset corresponding to each evaluation step includes sentences to be analyzed under different evaluation dimensions and corresponding dialogue state analysis result sets; the sentences to be analyzed contain dialogue state information; and the dialogue state analysis result sets corresponding to each sentence to be analyzed include a correct analysis result and at least one analysis result different from the correct analysis result;

[0007] In each evaluation step, the evaluation data set corresponding to the evaluation step is input into the pre-trained large language model, and a target prediction result corresponding to the sentence to be analyzed in the evaluation data set is output; the target prediction result is the analysis result selected by the pre-trained large language model from the dialogue state analysis result set corresponding to the input sentence to be analyzed;

[0008] The correct analysis results corresponding to the sentences to be analyzed in the evaluation data set are compared with the corresponding target prediction results to obtain evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

[0009] In one embodiment, inputting the evaluation data set corresponding to the evaluation phase into the pre-trained large language model and outputting the target prediction result corresponding to the sentence to be analyzed in the evaluation data set includes:

[0010] For any sentence to be analyzed in the evaluation dataset, repeatedly inputting the sentence to be analyzed and the corresponding dialogue state analysis result set into the pre-trained large language model a first preset number of times, and outputting a prediction result corresponding to the sentence to be analyzed under each input;

[0011] The prediction result with the largest number of occurrences is used as the target prediction result corresponding to any of the sentences to be analyzed.

[0012] In one embodiment, the number of times the evaluation data set corresponding to each evaluation step is input into the pre-trained large language model satisfies a second preset number, and the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set is different in each input; there are multiple target prediction results corresponding to the sentence to be analyzed in the evaluation data set, which respectively correspond to the analysis results selected by the pre-trained large language model in the dialogue state analysis result set input each time;

[0013] The correct analysis results corresponding to the sentences to be analyzed in the evaluation data set are compared with the corresponding target prediction results to obtain evaluation information of the pre-trained large language model in each evaluation link, including:

[0014] Comparing the target prediction result outputted each time for the sentence to be analyzed in the evaluation data set with the corresponding correct analysis result, to obtain the comparison result corresponding to each input of the evaluation data set;

[0015] According to the comparison results corresponding to each input of the evaluation data set, the evaluation information of the pre-trained large language model in the evaluation link is determined.

[0016] In one embodiment, the comparison result includes ratio information, and the target prediction result outputted each time for the sentence to be analyzed in the evaluation dataset is compared with the corresponding correct analysis result to obtain the comparison result corresponding to each input of the evaluation dataset, including:

[0017] In the process of obtaining the ratio information corresponding to any input of the evaluation data set, comparing the correct analysis result corresponding to the sentence to be analyzed in the evaluation data set with the target prediction result corresponding to the any input to determine a first number; the first number is the number of sentences to be analyzed for which the target prediction result corresponding to the any input is the same as the corresponding correct analysis result;

[0018] The ratio information of the first quantity to the second quantity is obtained as the ratio information corresponding to the evaluation data set under any input; the second quantity is the total number of statements to be analyzed in the evaluation data set.

[0019] In one embodiment, determining the evaluation information of the pre-trained large language model in the evaluation phase according to the comparison result corresponding to each input of the evaluation dataset includes:

[0020] Determining average ratio information based on the ratio information corresponding to each input of the evaluation data set;

[0021] The average ratio information is used as evaluation information of the pre-trained large language model in the evaluation step.

[0022] In one embodiment, inputting the evaluation data set corresponding to the evaluation phase into the pre-trained large language model and outputting the target prediction result corresponding to the sentence to be analyzed in the evaluation data set includes:

[0023] Input the sentences to be analyzed in the evaluation dataset and the corresponding dialogue state analysis result set into the pre-trained large language model, and output the target prediction result corresponding to the sentences to be analyzed in the evaluation dataset under this input;

[0024] Rearrange the order of the results of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set, use the reordered dialogue state analysis result set as a new dialogue state analysis result set, and return to the step of inputting the sentence to be analyzed in the evaluation data set and the corresponding dialogue state analysis result set into the pre-trained large language model until the second preset number of times is met.

[0025] In a second aspect, the present application further provides a device for outputting evaluation information of a large language model, comprising:

[0026] A data acquisition module is configured to acquire an evaluation dataset corresponding to each evaluation step of a preset dialogue state evaluation process; the dialogue state evaluation process is a process for evaluating the dialogue state perception capability of a pre-trained large language model; the evaluation dataset corresponding to each evaluation step includes sentences to be analyzed under different evaluation dimensions and corresponding dialogue state analysis result sets; the sentences to be analyzed contain dialogue state information; and the dialogue state analysis result sets corresponding to each sentence to be analyzed include a correct analysis result and at least one analysis result different from the correct analysis result.

[0027] A prediction module is configured to input, in each evaluation step, an evaluation dataset corresponding to the evaluation step into the pre-trained large language model, and output a target prediction result corresponding to the sentence to be analyzed in the evaluation dataset; the target prediction result is an analysis result selected by the pre-trained large language model from the dialogue state analysis result set corresponding to the input sentence to be analyzed;

[0028] A comparison module is used to compare the correct analysis results corresponding to the sentences to be analyzed in the evaluation data set with the corresponding target prediction results to obtain evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

[0029] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the steps of the above method are implemented.

[0030] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0031] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.

[0032] The above-mentioned large language model evaluation information output method, device, computer equipment, computer-readable storage medium and computer program product obtain an evaluation data set corresponding to each evaluation link of a preset dialogue state evaluation process; the dialogue state evaluation process is a process for evaluating the dialogue state perception ability of a pre-trained large language model; the evaluation data set corresponding to each evaluation link includes the sentence to be analyzed and the corresponding dialogue state analysis result set under different evaluation dimensions; the sentence to be analyzed has dialogue state information; the dialogue state analysis result set corresponding to each sentence to be analyzed includes a correct analysis result and at least one analysis result different from the correct analysis result; in each evaluation link, the evaluation data set corresponding to the evaluation link is input into the pre-trained large language model, and the target prediction result corresponding to the sentence to be analyzed in the evaluation data set is output; the target prediction result is the analysis result selected by the pre-trained large language model in the dialogue state analysis result set corresponding to the input sentence to be analyzed; the correct analysis result and the corresponding target prediction result corresponding to the sentence to be analyzed in the evaluation data set are compared to obtain the evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

[0033] In this way, the present application obtains an evaluation data set corresponding to each evaluation link of the preset dialogue state evaluation process. Compared with the related art that only evaluates the dialogue state perception ability of the large language model in a single link, the present application can obtain a more comprehensive evaluation data set to evaluate the large language model; at the same time, the evaluation data set corresponding to each evaluation link includes the sentences to be analyzed under different evaluation dimensions and the corresponding dialogue state analysis result set, which can more objectively evaluate the dialogue state perception ability of the large language model in various dialogue scenarios, and can more comprehensively evaluate the performance of the large language model, thereby improving the accuracy and reliability of the large language model evaluation; in addition, the dialogue state analysis result set corresponding to each sentence to be analyzed includes the correct analysis result and at least one analysis result different from the correct analysis result. In the evaluation phase, the evaluation data set corresponding to the evaluation phase is input into the large language model, and the target prediction result selected by the large language model in the dialogue state analysis result set corresponding to the sentence to be analyzed is output. The correct analysis result corresponding to the sentence to be analyzed is compared with the corresponding target prediction result to determine whether the large language model has selected the correct analysis result. Compared with only inputting the sentence to be analyzed into the large language model and scoring the output analysis result, this application can directly evaluate whether the large language model has the ability to understand and select the correct dialogue state analysis result, with higher evaluation accuracy, and determine whether the large language model has selected the preset correct analysis result. It can also efficiently evaluate the dialogue state perception ability of the large language model under different evaluation dimensions in different evaluation phases, thereby improving evaluation efficiency and reducing evaluation costs. In summary, this application has established a multi-link, multi-dimensional, unified and standard evaluation system to conduct multi-faceted evaluations of large language models, ensuring the accuracy and reliability of the evaluation of the dialogue state perception ability of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments of the present application or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without any creative work.

[0035] Figure 1 1 is a flow chart of a method for outputting evaluation information of a large language model in one embodiment;

[0036] Figure 2 A flowchart of a method for constructing an evaluation dataset for a large language model in one embodiment is shown;

[0037] Figure 3 1 is a flowchart of a method for constructing an evaluation dataset for a large language model according to another embodiment;

[0038] Figure 4 A schematic flow chart of a method for constructing an evaluation dataset for a large language model in another embodiment;

[0039] Figure 5 A flowchart of a dialogue state response in one embodiment;

[0040] Figure 6 1 is a flow chart of a method for outputting evaluation information of a large language model in another embodiment;

[0041] Figure 7 A structural block diagram of a device for constructing an evaluation dataset for a large language model in one embodiment;

[0042] Figure 8 is a structural block diagram of an evaluation information output device for a large language model in one embodiment;

[0043] Figure 9 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] It should be noted that the terms "first", "second", etc. used in this application may be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "including" and "having" used in this application and any variations thereof are intended to cover non-exclusive inclusions. The term "plurality" used in this application refers to two or more. The term "and / or" used in this application refers to one of the solutions or any combination of multiple solutions.

[0046] In one embodiment, Figure 1 As shown, a method for outputting evaluation information of a large language model is provided. This embodiment uses the method applied to a computer device as an example. It is understandable that the computer device can be a terminal, a server, or a system including a terminal and a server. In this embodiment, the method includes the following steps:

[0047] Step S110: obtaining an evaluation data set corresponding to each evaluation step of a preset dialogue state evaluation process.

[0048] Among them, the dialogue state evaluation process is a process for evaluating the dialogue state perception ability of the pre-trained large language model.

[0049] Among them, the ability to perceive the dialogue state refers to the ability of the large language model to understand the user's dialogue state during the dialogue process and generate responses that are consistent with the dialogue state.

[0050] The conversation state refers to the state of the user during the conversation, such as the emotional state (emotional state). In practical applications, the conversation state perception capability can be named emotion perception capability.

[0051] Among them, each evaluation link has different evaluation dimensions.

[0052] Among them, the evaluation data set corresponding to each evaluation link includes the sentences to be analyzed under different evaluation dimensions and the corresponding dialogue state analysis result set.

[0053] The sentences to be analyzed may refer to dialogue sentences selected from an existing multi-round dialogue dataset.

[0054] The sentence to be analyzed contains dialogue state information, and the dialogue state information refers to information related to the dialogue state.

[0055] The dialog state analysis result set corresponding to each sentence to be analyzed includes a correct analysis result and at least one analysis result different from the correct analysis result.

[0056] The analysis result refers to the result obtained by analyzing the statement to be analyzed. The analysis result may include but is not limited to the dialogue state recognition and understanding result, dialogue decision result, and dialogue response result.

[0057] Among them, the dialogue state recognition and understanding results refer to the recognition results of the dialogue state category corresponding to the sentence to be analyzed (in practical applications, it can refer to the emotion category, such as happiness, anger, expectation, etc.) and the analysis results of the reasons for the generation of this dialogue state category.

[0058] Among them, the dialogue decision result refers to the decision result of which response mode (such as comfort mode, companionship mode, chat mode, etc.) to adopt to promote the next step of dialogue.

[0059] The dialogue response result refers to the text information used to respond to the statement to be analyzed.

[0060] The correct analysis result may refer to a correct analysis result manually generated for the sentence to be analyzed. In practical applications, a plurality of psychology or sociology experts may collaborate to design a corresponding correct analysis result for the sentence to be analyzed.

[0061] Analysis results that differ from the correct ones can be generated by a large language model, or they can be generated by a large language model and then corrected by data annotators to ensure logical soundness and a certain degree of obfuscation. This semi-automated process (large model generation and manual correction) effectively reduces data construction costs.

[0062] In a specific implementation, when evaluating the dialogue state perception capability of a pre-trained large language model, an evaluation data set corresponding to each evaluation link of a preset dialogue state evaluation process can be obtained first. The evaluation data set corresponding to each evaluation link includes sentences to be analyzed and corresponding dialogue state analysis result sets under different evaluation dimensions, and the dialogue state analysis result set corresponding to each sentence to be analyzed includes a correct analysis result and at least one analysis result different from the correct analysis result.

[0063] In actual applications, the dialogue state analysis result set corresponding to each sentence to be analyzed may include one correct analysis result and three analysis results different from the correct analysis result. The number of analysis results different from the correct analysis result can be set according to actual needs and is not specifically limited here.

[0064] Step S120: In each evaluation link, the evaluation data set corresponding to the evaluation link is input into the pre-trained large language model, and the target prediction result corresponding to the sentence to be analyzed in the evaluation data set is output.

[0065] Among them, the target prediction result is the analysis result selected by the pre-trained large language model in the dialogue state analysis result set corresponding to the input sentence to be analyzed.

[0066] In the specific implementation, in each evaluation link, the evaluation data set corresponding to the evaluation link can be input into the pre-trained large language model, instructing the pre-trained large language model to select the analysis result that matches the sentence to be analyzed from the dialogue state analysis result set corresponding to the sentence to be analyzed, and output it as the target prediction result.

[0067] In step S130 , the correct analysis results corresponding to the sentences to be analyzed in the evaluation data set are compared with the corresponding target prediction results to obtain the evaluation information of the pre-trained large language model in each evaluation link.

[0068] The evaluation information is used to characterize the dialogue state perception capability of the pre-trained large language model.

[0069] In a specific implementation, the computer device can compare the correct analysis results corresponding to the sentences to be analyzed in the evaluation data set with the corresponding target prediction results through an automated script, and determine whether the target prediction results output by the pre-trained large language model for the sentences to be analyzed are the same as the corresponding correct analysis results, thereby counting the number of correct analysis results selected by the pre-trained large language model in the evaluation data set corresponding to the evaluation link, and then quantifying the dialogue state perception ability of the pre-trained large language model in the evaluation link, so as to output the evaluation information of the pre-trained large language model in the evaluation link. In this way, by counting the number of correct analysis results selected by the pre-trained large language model in the evaluation data set corresponding to each evaluation link, the evaluation information of the pre-trained large language model in each evaluation link is obtained.

[0070] In the above-mentioned large language model evaluation information output method, an evaluation data set corresponding to each evaluation link of a preset dialogue state evaluation process is obtained; the dialogue state evaluation process is a process for evaluating the dialogue state perception ability of a pre-trained large language model; the evaluation data set corresponding to each evaluation link includes the sentences to be analyzed and the corresponding dialogue state analysis result set under different evaluation dimensions; the sentences to be analyzed have dialogue state information; the dialogue state analysis result set corresponding to each sentence to be analyzed includes the correct analysis result and at least one analysis result different from the correct analysis result; in each evaluation link, the evaluation data set corresponding to the evaluation link is input into the pre-trained large language model, and the target prediction result corresponding to the sentence to be analyzed in the evaluation data set is output; the target prediction result is the analysis result selected by the pre-trained large language model in the dialogue state analysis result set corresponding to the input sentence to be analyzed; the correct analysis result and the corresponding target prediction result corresponding to the sentence to be analyzed in the evaluation data set are compared to obtain the evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

[0071] In this way, the present application obtains an evaluation data set corresponding to each evaluation link of the preset dialogue state evaluation process. Compared with the related art that only evaluates the dialogue state perception ability of the large language model in a single link, the present application can obtain a more comprehensive evaluation data set to evaluate the large language model; at the same time, the evaluation data set corresponding to each evaluation link includes the sentences to be analyzed under different evaluation dimensions and the corresponding dialogue state analysis result set, which can more objectively evaluate the dialogue state perception ability of the large language model in various dialogue scenarios, and can more comprehensively evaluate the performance of the large language model, thereby improving the accuracy and reliability of the large language model evaluation; in addition, the dialogue state analysis result set corresponding to each sentence to be analyzed includes the correct analysis result and at least one analysis result different from the correct analysis result. In the evaluation phase, the evaluation data set corresponding to the evaluation phase is input into the large language model, and the target prediction result selected by the large language model in the dialogue state analysis result set corresponding to the sentence to be analyzed is output. The correct analysis result corresponding to the sentence to be analyzed is compared with the corresponding target prediction result to determine whether the large language model has selected the correct analysis result. Compared with only inputting the sentence to be analyzed into the large language model and scoring the output analysis result, this application can directly evaluate whether the large language model has the ability to understand and select the correct dialogue state analysis result, with higher evaluation accuracy, and determine whether the large language model has selected the preset correct analysis result. It can also efficiently evaluate the dialogue state perception ability of the large language model under different evaluation dimensions in different evaluation phases, thereby improving evaluation efficiency and reducing evaluation costs. In summary, this application has established a multi-link, multi-dimensional, unified and standard evaluation system to conduct multi-faceted evaluations of large language models, ensuring the accuracy and reliability of the evaluation of the dialogue state perception ability of large language models.

[0072] In practical applications, the evaluation phase of the dialogue state evaluation process can include three phases: dialogue state recognition and understanding, dialogue state management, and general dialogue state response. In some other embodiments, the evaluation phase of the dialogue state evaluation process can further include a dialogue state response phase for specific scenarios. Dialogue state perception capabilities include dialogue state recognition and understanding capabilities, dialogue state management capabilities, and dialogue state response capabilities. The evaluation dataset corresponding to the dialogue state recognition and understanding phase is used to evaluate the dialogue state recognition and understanding capabilities of the large language model. The analysis results in the evaluation dataset corresponding to the dialogue state recognition and understanding phase can be dialogue state recognition and understanding results. The evaluation dataset corresponding to the dialogue state management phase is used to evaluate the dialogue state management capabilities of the large language model. The analysis results in the evaluation dataset corresponding to the dialogue state management phase can be dialogue decision results. The evaluation dataset corresponding to the dialogue state response phase is used to evaluate the dialogue state response capabilities of the large language model. The analysis results in the evaluation dataset corresponding to the dialogue state response phase can be dialogue response results.

[0073] In practical applications, conversation state recognition and understanding can be termed emotion recognition and understanding; conversation state management can be termed emotion management; and conversation state response can be termed emotion response. Below, we will specifically explain the different evaluation dimensions included in each evaluation phase and how to construct the conversation state analysis result sets corresponding to each evaluation phase. Specifically:

[0074] 1. Emotion recognition and understanding

[0075] The Emotion Recognition and Understanding Benchmark evaluates the conversation state perception capabilities of large language models based on the following four evaluation dimensions:

[0076] (1) Sources of Emotion (also known as sources of conversational state): including internal emotions and external emotions. This type of emotion source can refer to emotions generated within an individual or emotions triggered by external events.

[0077] (2) Emotional Expression Modes (also known as Conversational Expression Modes): These include direct expression and indirect expression. Direct expression involves describing one’s inner feelings, while indirect expression involves expressing one’s inner feelings through non-emotional language dialogue.

[0078] (3) Context of Emotion (also known as conversational context): covers everyday contexts, social interaction contexts, work and study contexts, and special event contexts. These contexts correspond to different social or personal activity scenarios, and the emotional expressions involved may vary depending on the context. Special event contexts include emotional reactions in unconventional situations.

[0079] (4) Dynamic Changes in Emotion (also known as Dynamic Changes in Dialogue): This includes emotion fluctuations and emotion transitions. Emotion fluctuations refer to the changes in the intensity of emotions in different situations, such as from happiness to sadness or from anxiety to relaxation; while emotion transitions emphasize the process of gradually transitioning from one emotion to another.

[0080] Each evaluation dimension is broken down into several subcategories, covering most possible emotional scenarios in life. This classification system not only meticulously categorizes emotions from multiple dimensions, but also incorporates direct and indirect emotional expression, internal and external emotional sources, and complex, multi-layered emotional contexts. This refined emotional classification structure will help comprehensively evaluate the recognition and understanding capabilities of large language models in diverse emotional scenarios, providing an effective benchmark and guidance for further improving the emotional intelligence of dialogue systems.

[0081] After determining the categories and subcategories of the benchmark, you can design the statements to be analyzed and the analysis results in each subcategory to ensure the quality and scientific nature of the test.

[0082] In actual applications, the evaluation data set corresponding to the emotion recognition and understanding link contains sentences to be analyzed corresponding to different emotion sources, different emotion expressions, different emotion contexts, and different emotion dynamic changes, in order to evaluate whether the large model can accurately understand the emotion categories and causes of emotions in the sentences to be analyzed in different evaluation dimensions.

[0083] Specifically, if Figure 2 As shown, a method for constructing an evaluation dataset for a large language model is provided, which is used to explain how to construct an evaluation dataset corresponding to the emotion recognition and understanding stages. The method includes the following steps:

[0084] Step S210: Obtain conversation status topic sentences corresponding to different evaluation dimensions.

[0085] The dialogue state topic sentence contains dialogue state information and topic scene information.

[0086] The subject scenario information refers to information related to the subject scenario of the conversation, such as the subject scenario of a conversation being the purchase of air tickets, medical consultation scenario, financial consultation scenario, etc.

[0087] In this embodiment, the evaluation dimension is used to evaluate the dialogue state recognition and understanding ability of the pre-trained large language model. In the emotion recognition and understanding link, the evaluation dimension may include the source of emotion, the mode of emotional expression, and the context of emotion. Emotion), Dynamic Changes in Emotion.

[0088] In a specific implementation, a computer device can obtain dialogue state topic sentences corresponding to different evaluation dimensions. For example, the dialogue state topic sentence can be "the sense of accomplishment after successfully completing a challenging task", the corresponding theme scenario is completing the task, and the dialogue state includes pride; for example, the dialogue state topic sentence can be "the sadness of the permanent loss of an important monument", the corresponding theme scenario is the loss of an item, and the dialogue state includes sadness.

[0089] In some embodiments, a computer device can obtain a multi-round dialogue data set and, according to the definition of the evaluation dimension, for example, according to the definition of each dialogue state information sub-category under the evaluation dimension, perform topic scenario expansion on the dialogue sentences containing dialogue state information to obtain dialogue state topic sentences.

[0090] Among them, the definitions of each dialogue state information sub-category under the evaluation dimension of the emotion recognition and understanding link (such as internal emotion, external emotion, direct expression, indirect expression, etc.) have been explained above and will not be repeated here.

[0091] In practical applications, the dialogue state topic sentence can be named as the emotion topic sentence.

[0092] Step S220: Expand the dialog state topic sentence to obtain an expanded dialog state topic sentence.

[0093] The expanded conversation state topic sentence includes preset key elements for analyzing the conversation state.

[0094] Among them, the preset key elements may include roles, dialogue status (emotions) and situations.

[0095] In a specific implementation, after obtaining the dialogue state topic sentence, the dialogue state topic sentence can be expanded to obtain an expanded dialogue state topic sentence. The expanded dialogue state topic sentence includes preset key elements for analyzing the dialogue state. The preset key elements may include roles, dialogue states (emotions) and situations.

[0096] In practice, users (e.g., emotion experts) can use large modeling tools to assist in constructing expanded conversation state topic sentences. At this stage, each expanded conversation state topic sentence must include three key elements: character, emotion, and context. The character component includes at least one protagonist, with one, two, or no supporting characters. The context element describes a specific situation or event, requiring clear logic and realistic accuracy. Finally, the emotion element requires that the outcome of the event indirectly reflect the protagonist's emotions.

[0097] For example, the topic sentence of a conversation state is "The sadness of losing an important souvenir forever." The expanded topic sentence of the conversation state can be "After the old photographer discovered that the negatives he had treasured for a lifetime were all moldy, he quietly put the empty albums back on the bookshelf and never picked up the camera again." Among them, the role element is: the old photographer; the situation element: all the negatives were moldy; the emotional elements: "silently put them back" and "never picked up the camera again". Restrained actions are used instead of wailing, and giving up professional habits is used as a mourning ritual to express a sense of powerlessness.

[0098] Step S230: Optimize the expanded conversation state topic sentence to obtain a sentence to be analyzed.

[0099] The richness of the dialogue state information of the sentence to be analyzed is higher than the richness of the dialogue state information of the expanded dialogue state subject sentence.

[0100] For example, the sentence to be analyzed may have more emotional ups and downs and emotional transitions compared to the expanded conversation state topic sentence.

[0101] In a specific implementation, the computer device can optimize the expanded dialogue state topic sentence to obtain the sentence to be analyzed for the emotion recognition and understanding link, so that the richness of the dialogue state information of the sentence to be analyzed is higher than the richness of the dialogue state information of the expanded dialogue state topic sentence.

[0102] In practice, the design requirement for the sentence to be analyzed might be: "Construct a misleading or turning point situation, so that the preceding and following information may potentially conflict in emotional expression." For example, the sentence to be analyzed might be, "After discovering that the negatives he had treasured his entire life had all gone moldy, the old photographer suddenly and cheerfully ordered the latest digital camera—until his family, while sorting through his belongings, discovered that all the new cameras' memory cards were blank." The surface behavior (misleading): "smiling and ordering a new camera" conveys a signal of optimism, suggesting a break from the shadows; the truth (turning point): "the blank memory cards" reveals that the photographer never actually clicked the shutter, masking despair with a false sense of positivity; the emotional conflict: The outwardly carefree exterior clashes with the inner desolation, reinforcing the end of his creative life due to the disappearance of the monument.

[0103] Step S240: Obtain a dialog state analysis result set corresponding to the sentence to be analyzed, and use the sentence to be analyzed and the corresponding dialog state analysis result set as an evaluation data set.

[0104] The dialog state analysis result set includes a correct analysis result and at least one analysis result that is different from the correct analysis result.

[0105] Among them, the evaluation dataset is used to evaluate the dialogue state perception ability of the pre-trained large language model.

[0106] In a specific implementation, a computer device can obtain a dialogue state analysis result set corresponding to the sentence to be analyzed, and use the sentence to be analyzed and the corresponding dialogue state analysis result set as an evaluation data set corresponding to the emotion recognition and understanding link, and evaluate the dialogue state perception ability of the pre-trained large language model through the evaluation data set.

[0107] In this embodiment, the dialogue state recognition and understanding capabilities of the pre-trained large language model are specifically evaluated using an evaluation dataset. The dialogue state recognition and understanding capabilities specifically include two aspects: dialogue state recognition capability and dialogue state understanding capability. The dialogue state recognition capability refers to the ability of the pre-trained large language model to accurately identify the dialogue state category corresponding to the sentence to be analyzed, and the dialogue state understanding capability refers to the ability of the pre-trained large language model to accurately understand the reasons for the generation of the dialogue state category. Accordingly, the dialogue state understanding capability can also be referred to as the ability to understand the reasons for the generation of the dialogue state category. Furthermore, the correct analysis result is used to represent the correct dialogue state category and the reasons for the generation of the correct dialogue state category, and the analysis result that differs from the correct analysis result is used to represent the dialogue state category that differs from the correct dialogue state category and the reasons for the generation of the dialogue state category that differs from the correct dialogue state category.

[0108] In the method for constructing the evaluation dataset of the above-mentioned large language model, the dialogue state topic sentences corresponding to different evaluation dimensions are obtained; the dialogue state topic sentences contain dialogue state information and topic scene information; the evaluation dimension is a dimension used to evaluate the dialogue state perception ability of the pre-trained large language model; the dialogue state topic sentence is expanded to obtain an expanded dialogue state topic sentence; the expanded dialogue state topic sentence includes preset key elements for analyzing the dialogue state; the expanded dialogue state topic sentence is optimized to obtain a sentence to be analyzed; the richness of the dialogue state information of the sentence to be analyzed is higher than the richness of the dialogue state information of the expanded dialogue state topic sentence; the dialogue state analysis result set corresponding to the sentence to be analyzed is obtained, and the sentence to be analyzed and the corresponding dialogue state analysis result set are used as the evaluation dataset; the dialogue state analysis result set includes a correct analysis result, and at least one analysis result different from the correct analysis result; the evaluation dataset is used to evaluate the dialogue state perception ability of the pre-trained large language model.

[0109] In this way, by obtaining dialogue state topic sentences with dialogue state information and topic scenario information corresponding to different evaluation dimensions, the dialogue state perception ability of the large language model in diverse and changeable dialogue scenarios can be more objectively evaluated. By expanding the dialogue state topic sentences, the expanded dialogue state topic sentences include preset key elements for analyzing the dialogue state, and then optimizing the expanded dialogue state topic sentences, the optimized sentences to be analyzed contain richer dialogue state information, thereby effectively ensuring the diversity and complexity of the sentences to be analyzed. Based on the sentences to be analyzed and the corresponding dialogue state analysis result set, an evaluation data set is constructed to evaluate the dialogue state perception ability of the pre-trained large language model, wherein the dialogue state analysis result set includes correct analysis results and at least one analysis result different from the correct analysis result, so as to increase the difficulty of testing the pre-trained large language model, comprehensively evaluate the model's dialogue state perception ability in real dialogues, and thus more accurately evaluate the dialogue state perception ability of the large language model.

[0110] In one embodiment, obtaining a dialogue state analysis result set corresponding to a statement to be analyzed includes: obtaining a first correct sub-result and a corresponding second correct sub-result corresponding to the statement to be analyzed; the first correct sub-result represents the correct dialogue state category corresponding to the statement to be analyzed; the second correct sub-result represents the reason for the correct dialogue state category; obtaining at least one first sub-result different from the first correct sub-result; the first sub-result represents the dialogue state category different from the correct dialogue state category; obtaining at least one second sub-result different from the second correct sub-result; the second sub-result represents the reason for the dialogue state category different from the correct dialogue state category.

[0111] The correct analysis result includes a first correct sub-result and a second correct sub-result, and each analysis result different from the correct analysis result includes a first sub-result and a second sub-result.

[0112] The conversation state category may include at least one of happiness, trust, fear, surprise, sadness, disgust, anger, and expectation, and may also include combinations and extended emotions thereof.

[0113] In a specific implementation, the correct analysis result includes a first correct sub-result and a second correct sub-result, and each analysis result different from the correct analysis result includes a first sub-result and a second sub-result. In the process of obtaining a dialogue state analysis result set corresponding to a statement to be analyzed, the computer device can obtain the first correct sub-result and the corresponding second correct sub-result corresponding to the statement to be analyzed; wherein the first correct sub-result represents the correct dialogue state category corresponding to the statement to be analyzed; the second correct sub-result represents the reason for the correct dialogue state category; and obtain at least one first sub-result different from the first correct sub-result; the first sub-result represents a dialogue state category different from the correct dialogue state category; and obtain at least one second sub-result different from the second correct sub-result; the second sub-result represents the reason for the dialogue state category different from the correct dialogue state category.

[0114] For example, the sentence to be analyzed is: "I'm going to meet a friend after get off work." The first correct sub-result is "looking forward to it," and the second correct sub-result is "meeting a friend." Each analysis result that differs from the correct analysis result includes a first sub-result and a second sub-result. For example, there are three analysis results that differ from the correct analysis result: Analysis Result 1, Analysis Result 2, and Analysis Result 3. Analysis Result 1 has the first sub-result "looking forward to it," and the second sub-result "getting off work." Analysis Result 2 has the first sub-result "happy," and the second sub-result "meeting a friend after get off work." Analysis Result 3 has the first sub-result "surprised," and the second sub-result "meeting a friend."

[0115] The technical solution of this embodiment is that the correct analysis result includes a first correct sub-result and a second correct sub-result, and each analysis result different from the correct analysis result includes a first sub-result and a second sub-result, by obtaining the first correct sub-result and the corresponding second correct sub-result corresponding to the sentence to be analyzed; the first correct sub-result represents the correct dialogue state category corresponding to the sentence to be analyzed; the second correct sub-result represents the reason for the correct dialogue state category; at least one first sub-result different from the first correct sub-result is obtained; the first sub-result represents the dialogue state category different from the correct dialogue state category; at least one second sub-result different from the second correct sub-result is obtained; the second sub-result represents the reason for the dialogue state category different from the correct dialogue state category.

[0116] In this way, the dialogue state analysis result set corresponding to the sentence to be analyzed includes the dialogue state category corresponding to the sentence to be analyzed and the reason for the dialogue state category. The pre-trained large language model is evaluated by the sentence to be analyzed and the corresponding dialogue state analysis result set. Not only can the dialogue state category recognition ability of the pre-trained large language model be evaluated, but also the understanding ability of the pre-trained large language model for the reason for the dialogue state category can be evaluated, thereby reflecting the deep dialogue state perception intelligence of the model and providing a benchmark for improving the dialogue state perception intelligence of the dialogue system.

[0117] In one embodiment, the method also includes: obtaining a first analysis result set corresponding to the statement to be analyzed; the first analysis result set includes a first category label and at least one second category label different from the first category label; the first category label represents the correct dialogue state category corresponding to the statement to be analyzed; the second category label represents the dialogue state category different from the correct dialogue state category; the first category label is used as the first correct sub-result, and the second category label is used as the first sub-result.

[0118] In a specific implementation, a computer device can obtain a first analysis result set corresponding to the statement to be analyzed; the first analysis result set includes a first category label and at least one second category label different from the first category label; wherein the first category label represents the correct dialogue state category corresponding to the statement to be analyzed; the second category label represents the dialogue state category different from the correct dialogue state category; the first category label is used as the first correct sub-result, and the second category label is used as the first sub-result.

[0119] In practical applications, first-category labels can be manually designed, such as by sentiment experts and data annotators, to design the most appropriate first-category label for each sentence to be analyzed. Specifically, four category labels can be designed for each sentence to be analyzed, one of which is a first-category label representing the correct conversation state category, and the other three are second-category labels representing conversation state categories that differ from the correct conversation state category. These second-category labels can be generated by a large language model and subsequently revised by data annotators to ensure logical rationality and a certain degree of obfuscation. The number of second-category labels can be set according to actual needs and is not specifically limited here.

[0120] The technical solution of this embodiment obtains a first analysis result set corresponding to the sentence to be analyzed; the first analysis result set includes a first category label and at least one second category label different from the first category label; the first category label represents the correct dialogue state category corresponding to the sentence to be analyzed; the second category label represents a dialogue state category different from the correct dialogue state category; the first category label is used as the first correct sub-result, and the second category label is used as the first sub-result. In this way, the correct dialogue state category is determined for the sentence to be analyzed, and a dialogue state category different from the correct dialogue state category is designed. The dialogue state recognition ability of the pre-trained large language model is evaluated to determine whether the pre-trained large language model can select the correct state category from multiple dialogue state category options, thereby increasing the difficulty of the dialogue state recognition ability test.

[0121] In one embodiment, the method also includes: obtaining a second analysis result set corresponding to the statement to be analyzed; the second analysis result set includes first reason information and at least one second reason information different from the first reason information; the first reason information represents the reason for the generation of a correct dialogue state category; the second reason information represents the reason for the generation of a dialogue state category different from the correct dialogue state category; the first reason information is used as the second correct sub-result, and the second reason information is used as the second sub-result.

[0122] In a specific implementation, the computer device can also obtain a second analysis result set corresponding to the statement to be analyzed; the second analysis result set includes first reason information and at least one second reason information different from the first reason information; the first reason information represents the reason for the generation of a correct dialogue state category; the second reason information represents the reason for the generation of a dialogue state category different from the correct dialogue state category; the first reason information is used as the second correct sub-result, and the second reason information is used as the second sub-result.

[0123] In practical applications, when determining the correct conversation state category corresponding to the sentence to be analyzed, the reason for the correct conversation state category can also be clarified as a second correct sub-result. Data annotators leverage the large language model to design three second sub-results that represent the reasons for conversation state categories that differ from the correct one, used to obfuscate the second correct sub-result. These second sub-results must not only be logically consistent and clearly stated, but must also be somewhat related to the second correct sub-result, but not actually be the reason for the correct conversation state category. The number of second sub-results is determined based on actual needs and is not specifically limited here.

[0124] The technical solution of this embodiment obtains a second analysis result set corresponding to the sentence to be analyzed; the second analysis result set includes first reason information and at least one second reason information different from the first reason information; the first reason information represents the reason for the generation of the correct dialogue state category; the second reason information represents the reason for the generation of a dialogue state category different from the correct dialogue state category; the first reason information is used as the second correct sub-result, and the second reason information is used as the second sub-result. In this way, the reason for the generation of the correct dialogue state category is determined for the sentence to be analyzed, and the reason for the generation of a dialogue state category different from the correct dialogue state category is designed. This can be used to evaluate the ability of a pre-trained large language model to understand the reason for the generation of dialogue state categories, determine whether the pre-trained large language model can select the correct dialogue state category from multiple dialogue state category reason options, and increase the difficulty of testing the ability to understand the reason for the generation of dialogue state categories.

[0125] In some embodiments, obtaining conversation state topic sentences corresponding to different evaluation dimensions includes: obtaining a conversation statement set from a preset multi-round conversation data set; conversation statements in the conversation statement set contain conversation state information; screening the conversation statement set to obtain a target conversation statement set; target conversation sentences in the target conversation statement set meet preset conversation quality conditions; obtaining an expanded target conversation sentence, and using the expanded target conversation sentence as the conversation state topic sentence.

[0126] The expanded target dialogue sentence is obtained by expanding the target dialogue sentence into a topic scenario according to the definition of the evaluation dimension.

[0127] In a specific implementation, when obtaining conversation state topic sentences corresponding to different evaluation dimensions, a computer device may extract conversation data containing conversation state information from a preset multi-round conversation dataset as conversation sentences to form a conversation sentence set. The conversation sentence set is then screened according to preset conversation quality conditions, and conversation sentences that meet the preset conversation quality conditions are selected as target conversation sentences. The target conversation sentences form the target conversation sentence set. The preset conversation quality conditions are used to screen out conversation sentences from the conversation sentence set that do not contain sensitive information and / or are logically sound.

[0128] In this way, we can derive conversational topic sentences corresponding to different evaluation dimensions based on the target conversational sentence set. Specifically, we can expand the topic scenario of the target conversational sentences according to the definitions of each conversational status information sub-category under each evaluation dimension corresponding to the emotion recognition and understanding phase, thereby deriving conversational topic sentences corresponding to different evaluation dimensions.

[0129] In practical applications, after extracting a large amount of conversation data containing dialogue state information from a pre-set multi-turn dialogue dataset as the basic material, a computer device can use semi-automated tools to screen for dialogue quality conditions, eliminating logically illogical or sensitive dialogue statements to obtain target dialogue statements. Alternatively, a rule-based regular expression can be used to filter out logically illogical or sensitive dialogue statements to obtain target dialogue statements. Subsequently, data annotators can expand the target dialogue statements into topic scenarios based on the definitions of each dialogue state information subcategory in the emotion recognition and understanding phase, designing dialogue state topic statements related to each dialogue state information subcategory. These dialogue state topic statements retain the dialogue state information in the original dialogue statements, providing a foundation for subsequent expansion.

[0130] The technical solution of this embodiment involves obtaining a set of dialogue sentences from a preset multi-round dialogue dataset; the dialogue sentences in the dialogue sentence set contain dialogue state information and topic scenario information; filtering the dialogue sentence set to obtain a target dialogue sentence set; the target dialogue sentences in the target dialogue sentence set meet preset dialogue quality conditions; obtaining expanded target dialogue sentences, and using the expanded target dialogue sentences as dialogue state topic sentences, wherein the expanded target dialogue sentences are obtained by expanding the target dialogue sentences according to the definition of the evaluation dimension. In this way, after filtering out the dialogue sentence set containing dialogue state information and topic scenario information from the multi-round dialogue dataset, the target dialogue sentence set that meets the preset dialogue quality conditions is then filtered out, thereby expanding the target dialogue sentences according to the definition of the evaluation dimension, effectively improving the quality of the obtained dialogue state topic sentences, making the dialogue state topic sentences more suitable for dialogue state awareness assessment, and thus effectively improving the credibility of the assessment when evaluating the dialogue state awareness capability of a pre-trained large language model based on the dialogue state topic sentences.

[0131] In one embodiment, an expanded dialog state topic sentence is optimized to obtain a sentence to be analyzed, including: inputting the expanded dialog state topic sentence into a pre-trained sentence optimization model and outputting the sentence to be analyzed; the pre-trained sentence optimization model adopts the following training method: obtaining a training sample set; the training sample set includes an original sentence and a corresponding label-optimized sentence; the richness of dialog state information of the label-optimized sentence is higher than the richness of dialog state information of the original sentence; inputting the original sentence into the sentence optimization model to be trained and outputting a predicted optimized sentence corresponding to the original sentence; adjusting the model parameters of the sentence optimization model to be trained based on the difference between the predicted optimized sentence corresponding to the original sentence and the corresponding label-optimized sentence and continuing iterative training until the trained sentence optimization model meets the training end condition and stops training to obtain the pre-trained sentence optimization model.

[0132] Among them, the pre-trained sentence optimization model can be the pre-trained large language model that needs to be evaluated for dialogue state perception ability as mentioned above, or it can be other pre-trained large language models, which are not specifically limited here.

[0133] In a specific implementation, when a computer device optimizes an expanded dialog state topic sentence to obtain a sentence to be analyzed, the expanded dialog state topic sentence can be input into a pre-trained sentence optimization model, and the expanded dialog state topic sentence can be optimized by the pre-trained sentence optimization model to output the sentence to be analyzed.

[0134] Among them, the pre-trained sentence optimization model adopts the following training method: the computer device can first obtain a training sample set; the training sample set includes the original sentence and the corresponding label-optimized sentence; the richness of the dialogue state information of the label-optimized sentence is higher than the richness of the dialogue state information of the original sentence.

[0135] The optimized sentence can be obtained by sentiment experts optimizing the original sentence. The design requirement for the optimized sentence can be: "Construct a misleading or turning point situation so that the preceding and following information may potentially conflict in emotional expression."

[0136] In this way, the original sentence is input into the sentence optimization model to be trained, and the predicted optimized sentence corresponding to the original sentence is output; then, according to the difference between the predicted optimized sentence corresponding to the original sentence and the corresponding label optimized sentence, the model parameters of the sentence optimization model to be trained can be adjusted and iterative training can be continued until the trained sentence optimization model meets the training end conditions, and the training is stopped to obtain the pre-trained sentence optimization model.

[0137] The technical solution of this embodiment is to input the expanded dialogue state topic sentence into a pre-trained sentence optimization model to output a sentence to be analyzed; the pre-trained sentence optimization model adopts the following training method: obtain a training sample set; the training sample set includes the original sentence and the corresponding label-optimized sentence; the richness of the dialogue state information of the label-optimized sentence is higher than the richness of the dialogue state information of the original sentence; input the original sentence into the sentence optimization model to be trained, and output the predicted optimized sentence corresponding to the original sentence; according to the difference between the predicted optimized sentence corresponding to the original sentence and the corresponding label-optimized sentence, adjust the model parameters of the sentence optimization model to be trained and continue iterative training until the trained sentence optimization model meets the training end condition and stops training to obtain the pre-trained sentence optimization model.

[0138] In this way, the expanded dialogue state topic sentence is input into the pre-trained sentence optimization model to output the sentence to be analyzed; the pre-trained sentence optimization model adopts the following training method: obtain a training sample set; the training sample set includes the original sentence and the corresponding label-optimized sentence; the richness of the dialogue state information of the label-optimized sentence is higher than the richness of the dialogue state information of the original sentence; the original sentence is input into the sentence optimization model to be trained, and the predicted optimized sentence corresponding to the original sentence is output; according to the difference between the predicted optimized sentence corresponding to the original sentence and the corresponding label-optimized sentence, the model parameters of the sentence optimization model to be trained are adjusted and iterative training is continued until the trained sentence optimization model meets the training end condition and stops training to obtain the pre-trained sentence optimization model.

[0139] In this way, the sentence optimization model to be trained is iteratively trained using the original sentences and the corresponding label-optimized sentences; wherein, the richness of the dialogue state information of the label-optimized sentences is higher than the richness of the dialogue state information of the original sentences, so that the pre-trained sentence optimization model obtained after training can optimize the dialogue state information of the input dialogue sentences, effectively improving the richness of the dialogue state information of the input dialogue sentences. By testing the dialogue state perception ability of the pre-trained large language model through dialogue sentences with richer dialogue state information, the test difficulty can be effectively increased.

[0140] In some other embodiments, Figure 3 As shown in FIG, a flow chart of a method for constructing an evaluation dataset of a large language model is provided. Figure 3 As shown, the process is divided into two parts: constructing the sentence to be analyzed (question construction) and designing the result set of the dialogue state analysis (answer design). The specific process is as follows Figure 3 As shown:

[0141] For the original data (conversation data) in the multi-round dialogue dataset, the dialogue data with dialogue state information and theme scene information are screened out to form a dialogue sentence set. The dialogue sentence set is screened according to the preset dialogue quality conditions, and the dialogue sentences that meet the preset dialogue quality conditions are screened out as the initial data (target dialogue sentences). Data labelers define the subcategories of each dialogue state information in the emotion recognition and understanding link, and design specific emotion sentences (dialogue state theme sentences) related to the subcategories based on the initial data. The emotion sentences are then expanded and optimized to obtain emotion problem sentences (sentences to be analyzed), and then the corresponding dialogue state analysis result set is designed for the emotion problem sentences.

[0142] In this way, a complete dataset for each emotion recognition and understanding test question is ultimately constructed. This process ensures the diversity and complexity of the questions, allowing them to not only test the model's emotion recognition capabilities but also assess its ability to understand the underlying causes of emotions. Each question and answer has been optimized and reviewed multiple times to provide a scientific, rigorous, and challenging benchmark test set that can truly reflect the model's performance in various emotional scenarios. This multi-dimensional, multi-level approach to constructing an emotion intelligence assessment dataset, through a refined emotion classification system and a rigorous data construction process, comprehensively evaluates the recognition capabilities and understanding of the underlying causes of emotions in complex emotion scenarios of large language models, providing a scientific benchmark for improving the emotional intelligence of dialogue systems.

[0143] In another embodiment, Figure 4 As shown, a flowchart of a method for constructing an evaluation dataset for a large language model is provided, including the following steps:

[0144] Step S402: Obtain a training sample set, input the original sentence into the sentence optimization model to be trained, and output the predicted optimized sentence corresponding to the original sentence.

[0145] Step S404: According to the difference between the predicted optimized statement corresponding to the original statement and the corresponding label optimized statement, the model parameters of the statement optimization model to be trained are adjusted and iterative training is continued until the trained statement optimization model meets the training end conditions, and the training is stopped to obtain the pre-trained statement optimization model.

[0146] Step S406: Obtain a conversation sentence set from a preset multi-round conversation dataset.

[0147] Step S408: Filter the dialogue sentence set to obtain a target dialogue sentence set.

[0148] Step S410: Obtain the expanded target dialogue sentence, and use the expanded target dialogue sentence as the dialogue state topic sentence.

[0149] Step S412: Expand the dialog state topic sentence to obtain an expanded dialog state topic sentence.

[0150] Step S414: input the expanded conversation state topic sentence into the pre-trained sentence optimization model, and output the sentence to be analyzed.

[0151] Step S416: Obtain the first correct sub-result and the corresponding second correct sub-result corresponding to the statement to be analyzed.

[0152] Step S418: Acquire at least one first sub-result different from the first correct sub-result, and acquire at least one second sub-result different from the second correct sub-result.

[0153] Step S420: Use the sentence to be analyzed and the corresponding dialog state analysis result set as an evaluation data set.

[0154] It should be noted that the specific limitations of the above steps can be found in the specific limitations of the method for constructing an evaluation dataset for a large language model mentioned above.

[0155] 2. Emotional Management

[0156] like Figure 5 The figure below provides a flow chart for responding to conversational states. Conversational emotion management refers to the process in which, in a human-computer conversation scenario, when the model identifies the user's emotional state (including the emotion category and the reason for the emotion), the robot makes a corresponding judgment based on the current topic and specific emotional state, thereby determining the model's response mode to advance the next step of the conversation. Emotional management closely combines the two aspects of "emotion recognition" and "emotional response." At its core, the model determines and predicts appropriate response strategies based on the user's current emotional state to ensure emotional support for the user and enhance the interactive experience.

[0157] Emotion management tasks provide emotional conversation robots with a reference for automatic emotional expression, helping them make reasonable and informed responses in emotional decision-making, thereby improving the overall user experience and user stickiness. Common emotion management modes in conversations can be divided into the following five categories:

[0158] (1) Soothing: Soothing refers to the process in which a robot takes a series of measures to stabilize a user's emotions when they are experiencing emotional fluctuations. As an emotion management strategy, soothing has the characteristics of reducing emotional intensity and restoring normal emotional levels. At the same time, through good communication, the robot can understand the reasons for the user's emotional fluctuations and provide targeted emotional guidance.

[0159] (2) Companionship: Companionship means that the robot spends some time with the user as a "companion" and provides emotional support. In emotional management, the role of companionship includes providing users with emotional support, enhancing their sense of security and confidence in facing difficulties, and releasing emotions through emotional communication.

[0160] (3) Small talk: Small talk refers to informal conversations conducted in a relaxed and enjoyable manner. In emotional management, small talk helps users release stress and regulate emotions. Through small talk, users can find common topics, increase interest in the conversation, and enhance interaction enthusiasm.

[0161] (4) Discussion: Discussion refers to a conversation in which the robot and the user exchange views on a certain issue, helping the user expand their thinking and find a suitable solution. Discussion not only allows users to fully express their own views and relieve emotional stress, but also helps them find solutions to problems and improve their ability to deal with problems.

[0162] (5) Psychological status monitoring: Psychological status monitoring refers to the robot's continuous emotional and psychological status assessment when facing an extremely unstable user, promptly identifying potential psychological problems and providing a basis for subsequent intervention. Its functions include assessing psychological status, formulating intervention plans, and conducting targeted psychological intervention based on the monitoring results.

[0163] In the era of large language models, emotion management has become a crucial foundation for improving models' ability to engage in emotional conversations. Equipping large language models with the ability to track and manage emotion helps them better consider coping strategies and emotional response patterns. Therefore, emotion management has become an essential capability for large language models in emotional conversations. To evaluate the coping and judgment capabilities of different large language models in emotional conversations, we have designed an emotion management task benchmark. This benchmark encompasses the following evaluation dimensions:

[0164] (1) Personal Scenarios: including Emotional Support, which involves personal emotional expression and psychological needs; Self-Development, which focuses on personal growth and self-improvement; and Health and Wellness, which involves events related to physical and mental health.

[0165] (2) Social Scenarios: including interpersonal communication, covering social activities and interpersonal relationship issues; academic and professional, events related to study and work; sociocultural dynamics, involving social events, cultural trends or public issues.

[0166] This benchmark simulates the thinking and judgment of a large language model in response to interlocutor messages from both personal and social perspectives, thereby assessing its emotional management capabilities. The personal scenario test primarily assesses the model's ability to propose emotional support strategies when the interlocutor has concerns or questions; the social scenario test focuses on testing the model's optimal response to socially relevant chat content from the user.

[0167] In actual applications, in the emotional management phase, the evaluation dataset can include the sentences to be analyzed and the corresponding dialogue state analysis result sets corresponding to multiple dialogue topics such as emotional support and self-development in the personal scenario dimension, as well as the sentences to be analyzed and the corresponding dialogue state analysis result sets corresponding to multiple dialogue topics such as interpersonal communication and academic career in the social scenario dimension. For example, in the emotional management phase, the evaluation dataset corresponding to the personal scenario dimension and the evaluation dataset corresponding to the social scenario dimension can each account for 50%. In actual applications, the proportion of evaluation datasets under different evaluation dimensions in the same evaluation phase can be set according to actual needs and is not specifically limited here.

[0168] To accurately evaluate emotion management performance, this proposal has designed an emotion management assessment scheme, similar to the benchmarks for emotion recognition and understanding. This assessment constructs emotional questions (sentences to be analyzed) and designs answer options (a set of dialogue state analysis results), ultimately forming a standard benchmark for emotion management. The specific process is similar to the construction method in the previous section, but the answer design involves emotion experts combining or deriving answers based on five emotion management models (comfort, companionship, small talk, discussion, and psychological status monitoring) to form the final correct analysis result. Additionally, other analysis results are semi-automatically generated by data annotators, also based on these five emotion management models, to supplement the design of combined or derived behaviors. In practice, each sentence to be analyzed has four analysis results: one correct analysis result and three different analysis results, which are selected by the large language model for evaluation.

[0169] For example, in the emotion management phase, the process of evaluating the pre-trained large language model using a sentence to be analyzed in the social scenario dimension and the corresponding dialogue state analysis results can be shown in Table 1 below:

[0170] Table 1

[0171]

[0172] As shown in Table 1, a sentence to be analyzed may include dialogue scenario information and dialogue question information. The dialogue scenario information is, "A music composer begins to realize that audiences prefer to hear music that is emotionally charged rather than so-called perfect." The dialogue question information is, "In this situation, what is the most effective approach for the music composer?" The dialogue state analysis result set corresponding to the sentence to be analyzed includes: "A: [Encourage him to be true to himself], B: [Discuss musical expression], C: [Analyze audience preferences], D: [Provide professional music production advice]," where "B: [Discuss musical expression]" is the correct analysis result. The sentence to be analyzed and the corresponding dialogue state analysis result set are input into a pre-trained large language model, which then outputs a target prediction result. For example, if the pre-trained large language model outputs the target prediction result of "C: [Analyze audience preferences]," it is determined that the pre-trained large language model did not select the correct analysis result. The computer device can further analyze the reason why the pre-trained large language model selected the incorrect result.

[0173] For example, consider a sentence to be analyzed and the corresponding conversation state analysis results for the topic of personal growth, self-improvement, and health and wellness in the personal scenario dimension. The sentence to be analyzed might be, "Robert recently got a satisfying job, but he also feels a lot of work pressure. He often works overtime and has an irregular diet." The corresponding conversation state analysis results might include "sharing work tips," "providing health advice," "listening to his stress," and "planning rest time together." The correct analysis result is "providing health advice."

[0174] 3. Universal emotional response

[0175] In robots and AI systems, the ability to express emotions is a key indicator of advanced emotional capabilities. The ability to respond emotionally richly and appropriately not only makes dialogue systems more humane but also reflects the long-term goal of AI technology in advancing machine emotional intelligence. The core of the emotional response task lies in generating conversational or non-conversational responses that incorporate emotional elements, enabling robots to demonstrate human-like emotional expression when interacting with humans. In this task, the robot first perceives and analyzes the user's emotional state and then, based on the current conversational context, provides the user with optimal emotional feedback.

[0176] Emotional responses are a typical generation task. Specifically, it combines emotional recognition and understanding of user speech with emotional management strategies to select the most appropriate response state and ultimately generate a targeted and meaningful response. Emotional response generation methods in related technologies typically rely on emotional tags to generate fixed-pattern responses. For example, when a user types in "I lost my keys while walking down the street today," the system detects the emotional tag as "sad" and generates a soothing response such as "Don't be sad, things will be fine." The limitation of this method is that the emotional responses it generates are relatively simple and cannot effectively address diverse emotional needs. Another improvement is to combine manual templates with deep learning models to optimize the diversity of responses and avoid stereotyped responses. However, traditional emotion generation models often neglect to deeply integrate user emotional information into the responses, resulting in the generated responses lacking emotional depth and appearing rather bland.

[0177] Large language models demonstrate significant advantages in emotional response. Due to their powerful conversational understanding capabilities, large language models can fully capture the information contained in the input text and generate more diverse emotional responses that are closer to user needs. Therefore, in order to more accurately evaluate the performance of different large language models in processing emotional subtasks, this application has designed a benchmark system specifically for evaluating emotional responses. This benchmark evaluates the emotional response capabilities of large language models from two evaluation dimensions:

[0178] (1) Response Scenarios: This includes both personal and social aspects. At the personal level, emotional responses involve responses to emotional issues, life difficulties, personal development, and other topics (specific aspects can include but are not limited to the examples above). At the social level, emotional responses focus on emotional needs in interpersonal communication, career and academic challenges, and social and cultural dynamics.

[0179] (2) Action Scenarios: These also include personal and social aspects. Personal action scenarios cover the emotional needs that users may face in terms of emotional issues, life difficulties, and developmental barriers. Social action scenarios include emotional responses to topics such as interpersonal relationships, workplace issues, and social and cultural dynamics.

[0180] This benchmark system comprehensively covers the emotional needs users may face by constructing a variety of real-life problem scenarios. During the design process, the third-person perspective of a large language model was introduced, enabling it to provide effective solutions to specific emotional issues based on the emotional and contextual information provided by users. These solutions can range from emotionally charged verbal responses to specific action recommendations.

[0181] In the baseline design for the emotional response section, the data generation and answer construction process continues the methods of the previous section. The difference is that in this section, emotional experts design scenarios for the core characters in each emotional theme, ensuring that these scenarios reflect the specific emotional needs of the characters. Based on these needs, the experts generate the most appropriate emotional responses as the correct analysis results. These responses can be in the form of language or action suggestions. The design requirements for the correct analysis results of emotional responses can include: humor, appeal, interactivity, gentleness, and politeness. When constructing analysis results that differ from the correct analysis results, data annotators use semi-automatic technology to design three analysis results based on the correct analysis results that have reasonable logic but do not fully meet the emotional needs. This ensures that the dialogue state analysis result set corresponding to each statement to be analyzed has sufficient discrimination.

[0182] 4. Emotional response to specific scenarios

[0183] To better apply sentiment evaluation of large language models to specific business scenarios, this application has designed a benchmark specifically for evaluating the emotional response of large language models to user feedback in specific scenarios. This benchmark aims to assess the model's performance in handling emotional issues in specific scenarios, and is divided into two evaluation dimensions: emotional counseling and psychological guidance.

[0184] The emotional counseling section covers four major categories of issues: love, friendship, family, and other emotional issues. Within the love context, this is further divided into five specific scenarios: early stages of love, mid-term love, late-term love, love crisis, and heartbreak and return. Friendship focuses on three typical scenarios: making new friends, maintaining friendships, and friendship crises. The family section includes emotional counseling for parent-child relationships, sibling relationships, and other kinship relationships. Other emotional scenarios involve a wider range of social relationships, such as virtual relationships, cross-cultural relationships, professional relationships, and emotional counseling scenarios related to interests and hobbies.

[0185] The emotional and psychological counseling section focuses on three common stressors: financial stress, life stress, and work / study stress. Financial stress encompasses situations such as insufficient income, financial planning, mortgage and car loans, education investment, and retirement pressures. Life stress encompasses family issues, relationships with friends, social anxiety, health concerns, and other life challenges. Work / study stress addresses workplace competition, entrepreneurial pressures, academic pressures, career planning, and work-life balance.

[0186] Within these sub-scenarios, emotional counseling and psychological counseling questions are designed to assess the performance of large language models in complex emotional situations. Emotional counseling covers most everyday emotional issues, while psychological counseling focuses on the most common sources of stress in real life. These scenarios are more challenging and align closely with the demands of real-life emotional conversations, making the evaluation results more meaningful.

[0187] The data construction process is similar to the previous one, but each sub-category scenario in this section was collaboratively designed by multiple psychologists to ensure greater authenticity, logic, and detail. These scenarios include context and detailed information to create a smooth, logical flow. The psychologists identified the protagonists and designed a logical and specific dialogue plot. The protagonists could ask questions or engage in small talk, using a distinctly colloquial style to meet the needs of real-world conversations. When designing the analysis results, emotional intelligence experts and data annotators used semi-automated techniques to generate four different analysis results. All analysis results were required to demonstrate plausibility, high emotional intelligence, clarity, and colloquial language, ensuring easy communication and stimulating continued interaction. While each analysis result had its own plausibility, only one was unanimously determined by the experts and annotators to be the most appropriate and correct analysis that would maximize emotional satisfaction. The entire conversation scenario, plot, and options were required to maintain a natural, coherent, and realistic logic.

[0188] There are some subtle differences in the design of the emotional counseling and emotional psychological counseling sections. For emotional counseling, a set of three multiple-choice questions will be designed for each subcategory scenario, designed for users of different age groups, covering the teenage stage (15-22 years old, juvenile), the youth stage (23-30 years old, youth) and the middle-aged stage (over 30 years old). This design is closer to the actual scene needs. The emotional psychological counseling section designs two multiple-choice questions for each emotional scenario, and each question is customized for a specific age group and identity to ensure that it is logical and appropriately challenging. Through these designs, the benchmark is more targeted and challenging, thereby better evaluating the emotional response ability of large language models.

[0189] Therefore, this application proposes a complete evaluation framework covering emotion recognition, understanding, management, and response, covering the entire process of emotion perception ability evaluation, ensuring a systematic evaluation of model capabilities; and, a semi-automated benchmark construction method collaborated by multiple psychology or sociology experts ensures the diversity and logical rigor of the evaluation data; at the same time, it includes multi-dimensional scenarios such as individuals, society, and special events, which are close to real conversation needs and enhance the practical application value of the evaluation results; constructs an evaluation system from four dimensions: emotion source, expression, context, and dynamic change, and supports complex emotion scene testing; can accurately evaluate whether the model has deeper reasoning ability and the ability to flexibly respond to diverse emotion scenes; in addition, it constructs standardized labels based on emotion theory, and supports the dynamic expansion of new scenarios and emotion types to adapt to the needs of technological development.

[0190] In some embodiments, an evaluation data set corresponding to the evaluation phase is input into a pre-trained large language model, and a target prediction result corresponding to the sentence to be analyzed in the evaluation data set is output, including: for any sentence to be analyzed in the evaluation data set, any sentence to be analyzed and the corresponding dialogue state analysis result set are repeatedly input into the pre-trained large language model according to a first preset number of times, and the prediction result corresponding to any sentence to be analyzed under each input is output; the prediction result that appears the most times is used as the target prediction result corresponding to any sentence to be analyzed.

[0191] The first preset number of times refers to the preset number of data inputs required to determine the target prediction result corresponding to any statement to be analyzed.

[0192] In a specific implementation, when the computer device inputs the evaluation data set corresponding to the evaluation phase into the pre-trained large language model and outputs the target prediction result corresponding to the sentence to be analyzed in the evaluation data set, it can repeatedly input any sentence to be analyzed in the evaluation data set corresponding to the evaluation phase and the corresponding dialogue state analysis result set into the pre-trained large language model according to a first preset number of times, and output the prediction result corresponding to any sentence to be analyzed under each input, so that the prediction result that appears the most times is used as the target prediction result corresponding to any sentence to be analyzed.

[0193] It is understandable that, in the process of repeatedly inputting any sentence to be analyzed and the corresponding dialogue state analysis result set into the pre-trained large language model according to the first preset number of times, the result arrangement order of the dialogue state analysis result set corresponding to any sentence to be analyzed should be the same, and the result arrangement order is the arrangement order of the analysis results. For example, the dialogue state analysis result set corresponding to any sentence to be analyzed includes four analysis results: "Encourage him to stick to himself", "Discuss the expression of music", "Analyze the audience's preferences", and "Provide suggestions for professional music production". In the process of repeatedly inputting the dialogue state analysis result set corresponding to any sentence to be analyzed into the pre-trained large language model according to the first preset number of times, the result arrangement order corresponding to each input should be the same, for example, the result arrangement order can be: "A: [Encourage him to stick to himself], B: [Discuss the expression of music], C: [Analyze the audience's preferences], D: [Provide suggestions for professional music production]".

[0194] Continuing with the previous example, assuming that the first preset number of times is 5 times, if the prediction results of the 5 outputs include 3 outputs of "C: [Analyze the audience's preferences]", 1 output of "D: [Provide advice on professional music production]", and 1 output of "A: [Encourage them to stick to themselves]", since "C: [Analyze the audience's preferences]" is output the most times, "C: [Analyze the audience's preferences]" is used as the target prediction result output by the pre-trained large prediction model for any sentence to be analyzed.

[0195] The technical solution of this embodiment involves repeatedly feeding any sentence to be analyzed in the evaluation dataset and its corresponding dialogue state analysis results set into a pre-trained large language model a first preset number of times, outputting the prediction result corresponding to each sentence to be analyzed. The prediction result with the most occurrences is used as the target prediction result for the sentence to be analyzed. In this way, by repeatedly feeding any sentence to be analyzed and its corresponding dialogue state analysis results set into the pre-trained large language model multiple times and using the prediction result with the most occurrences as the target prediction result, the reliability and statistical stability of the results are ensured.

[0196] In one embodiment, the number of times the evaluation data set corresponding to each evaluation link is input into the pre-trained large language model satisfies a second preset number, and the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set is different in each input; there are multiple target prediction results corresponding to the sentence to be analyzed in the evaluation data set, which respectively correspond to the analysis results selected by the pre-trained large language model in the dialogue state analysis result set for each input; the correct analysis results corresponding to the sentence to be analyzed in the evaluation data set and the corresponding target prediction results are compared to obtain evaluation information of the pre-trained large language model in each evaluation link, including: comparing the target prediction result output each time for the sentence to be analyzed in the evaluation data set with the corresponding correct analysis result to obtain the comparison result corresponding to the evaluation data set under each input; and determining the evaluation information of the pre-trained large language model in the evaluation link according to the comparison result corresponding to the evaluation data set under each input.

[0197] In a specific implementation, when a computer device compares the correct analysis result corresponding to a sentence to be analyzed in an evaluation data set with the corresponding target prediction result to obtain evaluation information of the pre-trained large language model in each evaluation link, since there are multiple target prediction results corresponding to the sentence to be analyzed in the evaluation data set, which respectively correspond to the analysis results selected by the pre-trained large language model in a dialogue state analysis result set with a different result arrangement order for each input, the computer device can compare the target prediction result output each time for the sentence to be analyzed in the evaluation data set with the corresponding correct analysis result to obtain the comparison result corresponding to each input of the evaluation data set, and determine the evaluation information of the pre-trained large language model in the evaluation link based on the comparison result corresponding to each input of the evaluation data set.

[0198] Furthermore, the second preset number of times refers to a preset number of statistical operations required to determine evaluation information of the pre-trained large language model in each evaluation phase; wherein the evaluation information can be represented by the average accuracy of the pre-trained large language model in the evaluation data set corresponding to the evaluation phase, wherein the number of statistical operations refers to the number of accuracy statistics required to calculate the average accuracy of the pre-trained large language model in the evaluation data set corresponding to the evaluation phase. The accuracy is calculated through the comparison results.

[0199] For each evaluation phase, the sentences to be analyzed and the corresponding dialogue state analysis result set can be input into a pre-trained large language model to determine the target prediction result output by the pre-trained large language model for the sentences to be analyzed. The method for determining the target prediction result for any sentence to be analyzed has been described in the above embodiments and is not specifically limited here. For each evaluation phase, after determining the target prediction result for each sentence to be analyzed in the evaluation data set, the target prediction result for each sentence to be analyzed can be compared with the corresponding correct analysis result. Based on the comparison results, the accuracy of the pre-trained large language model for the result order currently corresponding to that evaluation phase can be calculated. Then, the result order of the dialogue state analysis result set in the evaluation data set corresponding to that evaluation phase can be reordered, and the sentences to be analyzed and the reordered dialogue state analysis result set in the evaluation data set corresponding to that evaluation phase can be input into the pre-trained large language model. The accuracy of the pre-trained large language model for the result order currently corresponding to that evaluation phase can be again calculated. In this way, by repeating the second preset number of times, the average accuracy is calculated based on the accuracy statistics obtained each time under different result arrangement orders, thereby obtaining the evaluation information of the pre-trained large language model in any evaluation link.

[0200] For example, the accuracy of the pre-trained large language model under the current result arrangement order corresponding to any evaluation link is calculated, and then the result arrangement order corresponding to any evaluation link is re-sorted and re-input into the pre-trained large language model to recalculate the accuracy, and the execution is repeated four times (the second preset number of times is four), and the average accuracy rate is calculated based on the accuracy rates obtained from the four statistics as the evaluation information of the pre-trained large language model under any evaluation link.

[0201] In the technical solution of this embodiment, the number of times the evaluation data set corresponding to each evaluation phase is input into the pre-trained large language model satisfies a second preset number, the result arrangement order of the dialogue state analysis result set corresponding to the to-be-analyzed sentence in the evaluation data set is different for each input; there are multiple target prediction results corresponding to the to-be-analyzed sentence in the evaluation data set, each corresponding to the analysis result selected by the pre-trained large language model in the dialogue state analysis result set for each input; by comparing the target prediction result output each time for the to-be-analyzed sentence in the evaluation data set with the corresponding correct analysis result, a comparison result corresponding to the evaluation data set under each input is obtained; and based on the comparison result corresponding to the evaluation data set under each input, the evaluation information of the pre-trained large language model under the evaluation phase is determined. In this way, the evaluation data set corresponding to each evaluation phase is input into the pre-trained large language model multiple times, the result arrangement order of the dialogue state analysis result set corresponding to the to-be-analyzed sentence in the evaluation data set is different for each input, and the target prediction result output each time for the to-be-analyzed sentence in the evaluation data set is compared with the corresponding correct analysis result to obtain the comparison result corresponding to the evaluation data set under each input. This can eliminate the model's potential preference for the result arrangement order and improve the reliability of the evaluation information obtained based on the comparison result.

[0202] In one embodiment, the comparison result includes ratio information, and the target prediction result output each time for the sentence to be analyzed in the evaluation data set is compared with the corresponding correct analysis result to obtain the comparison result corresponding to each input of the evaluation data set, including: in the process of obtaining the ratio information corresponding to the evaluation data set under any input, the correct analysis result corresponding to the sentence to be analyzed in the evaluation data set is compared with the target prediction result corresponding to any input to determine a first quantity; the first quantity is the number of sentences to be analyzed whose corresponding target prediction result and the corresponding correct analysis result are the same under any input; the ratio information of the first quantity and the second quantity is obtained as the ratio information corresponding to the evaluation data set under any input; the second quantity is the total number of sentences to be analyzed in the evaluation data set.

[0203] In a specific implementation, the computer device compares the target prediction result outputted each time for the sentence to be analyzed in the evaluation dataset with the corresponding correct analysis result to obtain the comparison result corresponding to each input of the evaluation dataset. Furthermore, in the process of obtaining ratio information corresponding to any input of the evaluation dataset, the computer device may compare the correct analysis result corresponding to the sentence to be analyzed in the evaluation dataset with the target prediction result corresponding to any input to determine a first number; the first number is the number of sentences to be analyzed whose corresponding target prediction result and correct analysis result are the same under any input, and the ratio information of the first number to the second number is obtained as the ratio information corresponding to any input of the evaluation dataset; the second number is the total number of sentences to be analyzed in the evaluation dataset. The ratio information is the accuracy rate.

[0204] In the technical solution of this embodiment, the comparison result includes ratio information. By comparing the correct analysis result corresponding to the sentence to be analyzed in the evaluation dataset with the target prediction result corresponding to any input in the process of obtaining the ratio information corresponding to the evaluation dataset under any input, a first quantity is determined; the first quantity is the number of sentences to be analyzed whose target prediction result corresponding to any input is the same as the corresponding correct analysis result; the ratio information of the first quantity and the second quantity is obtained as the ratio information corresponding to the evaluation dataset under any input; the second quantity is the total number of sentences to be analyzed in the evaluation dataset. In this way, based on the ratio information corresponding to the evaluation dataset under any input, the accuracy of the pre-trained large language model corresponding to the result arrangement order corresponding to the any input can be determined.

[0205] It is understandable that in some embodiments, for any evaluation link, the corresponding accuracy rates can also be counted for different evaluation dimensions corresponding to the evaluation link. Specifically, for each evaluation dimension, the number of sentences to be analyzed whose corresponding target prediction results under the evaluation dimension are the same as the corresponding correct analysis results is counted to obtain the third number corresponding to each evaluation dimension, and then the total number of sentences to be analyzed under each evaluation dimension is determined to obtain the fourth number corresponding to each evaluation dimension. The ratio information between the third number corresponding to each evaluation dimension and the fourth number corresponding to each evaluation dimension is obtained as the accuracy rate of the pre-trained large language model under each evaluation dimension.

[0206] In one embodiment, the evaluation information of the pre-trained large language model in the evaluation link is determined based on the comparison results corresponding to each input of the evaluation data set, including: determining the average ratio information based on the ratio information corresponding to each input of the evaluation data set; and using the average ratio information as the evaluation information of the pre-trained large language model in the evaluation link.

[0207] Among them, the average ratio information is the average accuracy.

[0208] In a specific implementation, when a computer device determines the evaluation information of a pre-trained large language model in an evaluation link based on the comparison results corresponding to each input of the evaluation data set, it can determine the average ratio information based on the ratio information corresponding to each input of the evaluation data set; and use the average ratio information as the evaluation information of the pre-trained large language model in the evaluation link.

[0209] The technical solution of this embodiment determines average ratio information based on the ratio information corresponding to each input in the evaluation dataset; this average ratio information is then used as evaluation information for the pre-trained large language model during the evaluation phase. This allows the dialog state analysis result set to be arranged in a different order for each input. By calculating the average ratio information corresponding to each input, random fluctuations can be reduced, improving stability. Using this average information as evaluation information for the pre-trained large language model during the evaluation phase enhances its credibility.

[0210] In one embodiment, an evaluation data set corresponding to an evaluation phase is input into a pre-trained large language model, and a target prediction result corresponding to a sentence to be analyzed in the evaluation data set is output, including: inputting the sentence to be analyzed and the corresponding dialogue state analysis result set in the evaluation data set into the pre-trained large language model, and outputting the target prediction result corresponding to the sentence to be analyzed in the evaluation data set under this input; reordering the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set, using the reordered dialogue state analysis result set as a new dialogue state analysis result set, and returning to the step of inputting the sentence to be analyzed and the corresponding dialogue state analysis result set in the evaluation data set into the pre-trained large language model until a second preset number of times is met.

[0211] In a specific implementation, when a computer device inputs an evaluation dataset corresponding to an evaluation phase into a pre-trained large language model and outputs a target prediction result corresponding to a sentence to be analyzed in the evaluation dataset, the computer device may input the sentence to be analyzed and the corresponding dialogue state analysis result set in the evaluation dataset into the pre-trained large language model, output the target prediction result corresponding to the sentence to be analyzed in the evaluation dataset under this input, re-sort the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation dataset, use the re-sorted dialogue state analysis result set as a new dialogue state analysis result set, and return to the step of inputting the sentence to be analyzed and the corresponding dialogue state analysis result set in the evaluation dataset into the pre-trained large language model until a second preset number of times is met, thereby ensuring that the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation dataset is different in each input, thereby obtaining the target prediction result corresponding to the sentence to be analyzed in the evaluation dataset corresponding to the evaluation phase under each input.

[0212] The technical solution of this embodiment is to input the sentences to be analyzed in the evaluation dataset and the corresponding dialogue state analysis result set into a pre-trained large language model, output the target prediction result corresponding to the sentences to be analyzed in the evaluation dataset under the current input; re-sort the result arrangement order of the dialogue state analysis result set corresponding to the sentences to be analyzed in the evaluation dataset, use the re-sorted dialogue state analysis result set as the new dialogue state analysis result set, and return to the step of inputting the sentences to be analyzed in the evaluation dataset and the corresponding dialogue state analysis result set into the pre-trained large language model until a second preset number of times is met. In this way, it is possible to ensure that the result arrangement order of the dialogue state analysis result set corresponding to the sentences to be analyzed in the evaluation dataset is different in each input, and obtain the target prediction answer corresponding to the sentences to be analyzed under the dialogue state analysis result sets corresponding to different result arrangement orders, thereby eliminating the model's potential preference for the result arrangement order.

[0213] In some further embodiments, for any evaluation link, the evaluation dimensions under any evaluation link may correspond to different evaluation weights. Thus, in the process of calculating the accuracy of the pre-trained large language model under any evaluation link, the number of sentences to be analyzed whose corresponding target prediction results are the same as the corresponding correct analysis results can be counted according to different evaluation dimensions to obtain a third quantity corresponding to different evaluation dimensions, and then based on the fourth quantity corresponding to each evaluation dimension, the third quantity corresponding to each evaluation dimension and the evaluation weight corresponding to each evaluation dimension, the accuracy of the pre-trained large language model under different evaluation dimensions can be calculated.

[0214] In another embodiment, Figure 6 As shown, a flowchart of a method for outputting evaluation information of a large language model is provided, including the following steps:

[0215] Step S602: obtaining an evaluation data set corresponding to each evaluation step of a preset dialogue state evaluation process.

[0216] Step S604: For any sentence to be analyzed in the evaluation data set, any sentence to be analyzed and the corresponding dialogue state analysis result set are repeatedly input into the pre-trained large language model according to a first preset number of times, and the prediction result corresponding to any sentence to be analyzed under each input is output.

[0217] Step S606: The prediction result with the largest number of occurrences is used as the target prediction result corresponding to any sentence to be analyzed.

[0218] In step S608, the sentences to be analyzed in the evaluation data set and the corresponding dialogue state analysis result set are input into the pre-trained large language model, and the target prediction result corresponding to the sentences to be analyzed in the evaluation data set under this input is output.

[0219] Step S610: reorder the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set, use the reordered dialogue state analysis result set as a new dialogue state analysis result set, and return to the step of inputting the sentence to be analyzed in the evaluation data set and the corresponding dialogue state analysis result set into the pre-trained large language model until the second preset number of times is met.

[0220] Step S612, in the process of obtaining the ratio information corresponding to the evaluation data set under any input, the correct analysis result corresponding to the sentence to be analyzed in the evaluation data set is compared with the target prediction result corresponding to any input to determine the first quantity.

[0221] Step S614: obtaining ratio information of the first quantity to the second quantity as the ratio information corresponding to any input of the evaluation data set.

[0222] Step S616: Determine average ratio information based on the ratio information corresponding to each input of the evaluation data set.

[0223] In step S618, the average ratio information is used as evaluation information of the pre-trained large language model in the evaluation phase.

[0224] It should be noted that the specific limitations of the above steps can be found in the specific limitations of the evaluation information output method for a large language model mentioned above.

[0225] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily performed in sequence in the order indicated by the arrows. Unless clearly stated herein, the execution of these steps is not strictly limited in order, and these steps can be performed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above may include multiple steps or multiple stages, and these steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of the steps or stages in other steps or other steps. It is understandable that the various steps in different embodiments can be freely combined as needed, and the various non-contradictory schemes formed by the combination all fall within the scope of protection of this application.

[0226] Based on the same inventive concept, the embodiments of the present application also provide a device for constructing a large language model evaluation dataset for implementing the aforementioned method for constructing a large language model evaluation dataset. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of the embodiments of one or more devices for constructing a large language model evaluation dataset provided below can be found in the limitations of the method for constructing a large language model evaluation dataset above and will not be repeated here.

[0227] In an exemplary embodiment, Figure 7 As shown, a device for constructing an evaluation dataset of a large language model is provided, comprising: a sentence acquisition module 710, an expansion module 720, an optimization module 730 and a construction module 740, wherein:

[0228] The sentence acquisition module 710 is used to obtain dialogue state topic sentences corresponding to different evaluation dimensions; the dialogue state topic sentences contain dialogue state information and topic scene information; the evaluation dimensions are dimensions used to evaluate the dialogue state perception ability of the pre-trained large language model.

[0229] The expansion module 720 is configured to expand the conversation state topic sentence to obtain an expanded conversation state topic sentence; the expanded conversation state topic sentence includes preset key elements for analyzing the conversation state.

[0230] The optimization module 730 is configured to optimize the expanded dialog state subject sentence to obtain a sentence to be analyzed; the richness of the dialog state information of the sentence to be analyzed is higher than the richness of the dialog state information of the expanded dialog state subject sentence.

[0231] A construction module 740 is configured to obtain a dialog state analysis result set corresponding to the sentence to be analyzed, and use the sentence to be analyzed and the corresponding dialog state analysis result set as an evaluation dataset; the dialog state analysis result set includes a correct analysis result and at least one analysis result different from the correct analysis result; the evaluation dataset is used to evaluate the dialog state perception capability of the pre-trained large language model.

[0232] In one embodiment, the correct analysis result includes a first correct sub-result and a second correct sub-result, and each analysis result different from the correct analysis result includes a first sub-result and a second sub-result. The construction module 740 is specifically used to obtain the first correct sub-result and the corresponding second correct sub-result corresponding to the statement to be analyzed; the first correct sub-result represents the correct dialogue state category corresponding to the statement to be analyzed; the second correct sub-result represents the reason for the correct dialogue state category; obtain at least one first sub-result different from the first correct sub-result; the first sub-result represents a dialogue state category different from the correct dialogue state category; obtain at least one second sub-result different from the second correct sub-result; the second sub-result represents the reason for the dialogue state category different from the correct dialogue state category.

[0233] In one embodiment, the device further includes: a result acquisition module, used to obtain a first analysis result set corresponding to the statement to be analyzed; the first analysis result set includes a first category label and at least one second category label different from the first category label; the first category label represents a correct dialogue state category corresponding to the statement to be analyzed; the second category label represents a dialogue state category different from the correct dialogue state category; the first category label is used as the first correct sub-result, and the second category label is used as the first sub-result.

[0234] In one embodiment, the acquisition module is specifically used to obtain a second analysis result set corresponding to the statement to be analyzed; the second analysis result set includes first reason information and at least one second reason information different from the first reason information; the first reason information represents the reason for the generation of the correct dialogue state category; the second reason information represents the reason for the generation of a dialogue state category different from the correct dialogue state category; the first reason information is used as the second correct sub-result, and the second reason information is used as the second sub-result.

[0235] In one embodiment, the sentence acquisition module 710 is specifically used to obtain a dialogue sentence set from a preset multi-round dialogue data set; the dialogue sentences in the dialogue sentence set contain the dialogue state information; the dialogue sentence set is screened to obtain a target dialogue sentence set; the target dialogue sentences in the target dialogue sentence set meet the preset dialogue quality conditions; the expanded target dialogue sentence is obtained, and the expanded target dialogue sentence is used as the dialogue state topic sentence; the expanded target dialogue sentence is obtained by subject scenario expansion of the target dialogue sentence according to the definition of the evaluation dimension.

[0236] In one embodiment, the optimization module 730 is specifically used to input the expanded dialogue state topic sentence into a pre-trained sentence optimization model and output the sentence to be analyzed; the pre-trained sentence optimization model adopts the following training method: obtaining a training sample set; the training sample set includes an original sentence and a corresponding label-optimized sentence; the richness of the dialogue state information of the label-optimized sentence is higher than the richness of the dialogue state information of the original sentence; inputting the original sentence into the sentence optimization model to be trained, and outputting the predicted optimized sentence corresponding to the original sentence; adjusting the model parameters of the sentence optimization model to be trained according to the difference between the predicted optimized sentence corresponding to the original sentence and the corresponding label-optimized sentence and continuing iterative training until the trained sentence optimization model meets the training end condition and stops training to obtain the pre-trained sentence optimization model.

[0237] Each module in the apparatus for constructing a large language model evaluation dataset can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0238] Based on the same inventive concept, embodiments of the present application also provide a large language model evaluation information output device for implementing the aforementioned large language model evaluation information output method. The implementation solution provided by this device is similar to the implementation solution described in the aforementioned method. Therefore, the specific limitations of one or more large language model evaluation information output device embodiments provided below can be found in the above-mentioned limitations of the large language model evaluation information output method, and will not be repeated here.

[0239] In an exemplary embodiment, Figure 8 As shown, a device for outputting evaluation information of a large language model is provided, comprising: a data acquisition module 810, a prediction module 820 and a comparison module 830, wherein:

[0240] Data acquisition module 810 is used to obtain an evaluation data set corresponding to each evaluation step of a preset dialogue state evaluation process; the dialogue state evaluation process is a process for evaluating the dialogue state perception capability of a pre-trained large language model; the evaluation data set corresponding to each evaluation step includes a statement to be analyzed under different evaluation dimensions and a corresponding dialogue state analysis result set; the statement to be analyzed contains dialogue state information; the dialogue state analysis result set corresponding to each statement to be analyzed includes a correct analysis result and at least one analysis result different from the correct analysis result.

[0241] The prediction module 820 is used to input the evaluation data set corresponding to the evaluation link into the pre-trained large language model in each evaluation link, and output the target prediction result corresponding to the sentence to be analyzed in the evaluation data set; the target prediction result is the analysis result selected by the pre-trained large language model in the dialogue state analysis result set corresponding to the input sentence to be analyzed.

[0242] The comparison module 830 is used to compare the correct analysis results corresponding to the sentences to be analyzed in the evaluation data set with the corresponding target prediction results to obtain the evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

[0243] In one embodiment, the prediction module 820 is specifically configured to repeatedly input any sentence to be analyzed in the evaluation data set and the corresponding dialogue state analysis result set into the pre-trained large language model according to a first preset number of times, and output the prediction result corresponding to the sentence to be analyzed under each input; and the prediction result that appears the most times is used as the target prediction result corresponding to the sentence to be analyzed.

[0244] In one embodiment, the number of times the evaluation data set corresponding to each evaluation link is input into the pre-trained large language model satisfies a second preset number, and the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set is different in each input; there are multiple target prediction results corresponding to the sentence to be analyzed in the evaluation data set, which respectively correspond to the analysis results selected by the pre-trained large language model in the dialogue state analysis result set for each input; the comparison module 830 is specifically used to compare the target prediction result output each time for the sentence to be analyzed in the evaluation data set with the corresponding correct analysis result to obtain the comparison result corresponding to the evaluation data set under each input; based on the comparison result corresponding to the evaluation data set under each input, the evaluation information of the pre-trained large language model in the evaluation link is determined.

[0245] In one embodiment, the comparison result includes ratio information, and the comparison module 830 is specifically used to compare the correct analysis result corresponding to the statement to be analyzed in the evaluation data set with the target prediction result corresponding to any input in the process of obtaining the ratio information corresponding to the evaluation data set under any input, and determine a first quantity; the first quantity is the number of statements to be analyzed whose target prediction result corresponding to any input is the same as the corresponding correct analysis result; obtain the ratio information of the first quantity and the second quantity as the ratio information corresponding to the evaluation data set under any input; the second quantity is the total number of statements to be analyzed in the evaluation data set.

[0246] In one embodiment, the comparison module 830 is specifically used to determine the average ratio information based on the ratio information corresponding to each input of the evaluation data set; and use the average ratio information as the evaluation information of the pre-trained large language model in the evaluation link.

[0247] In one embodiment, the prediction module 820 is specifically configured to input the sentence to be analyzed and the corresponding dialogue state analysis result set in the evaluation data set into a pre-trained large language model, and output a target prediction result corresponding to the sentence to be analyzed in the evaluation data set under this input; reorder the result arrangement order of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set, use the reordered dialogue state analysis result set as a new dialogue state analysis result set, and return to the step of inputting the sentence to be analyzed and the corresponding dialogue state analysis result set in the evaluation data set into the pre-trained large language model until the second preset number of times is met.

[0248] Each module in the aforementioned large language model evaluation information output device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0249] In an exemplary embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. Figure 9As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, it implements a method for outputting evaluation information of a large language model and / or constructing an evaluation data set of a large language model. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0250] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0251] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0252] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0253] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0254] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0255] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile memory and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable logic unit (PLC), a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, and the like.

[0256] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0257] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for outputting evaluation information of a large language model, characterized in that: The method comprises: Obtaining an evaluation dataset corresponding to each evaluation step of a preset dialogue state evaluation process; the dialogue state evaluation process is a process for evaluating the dialogue state perception capability of a pre-trained large language model; the evaluation dataset corresponding to each evaluation step includes sentences to be analyzed under different evaluation dimensions and corresponding dialogue state analysis result sets; the sentences to be analyzed contain dialogue state information; and the dialogue state analysis result sets corresponding to each sentence to be analyzed include a correct analysis result and at least one analysis result different from the correct analysis result; In each evaluation step, the evaluation data set corresponding to the evaluation step is input into the pre-trained large language model, and a target prediction result corresponding to the sentence to be analyzed in the evaluation data set is output; the target prediction result is the analysis result selected by the pre-trained large language model from the dialogue state analysis result set corresponding to the input sentence to be analyzed; The correct analysis results corresponding to the sentences to be analyzed in the evaluation data set are compared with the corresponding target prediction results to obtain evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

2. The method according to claim 1, characterized in that The step of inputting the evaluation data set corresponding to the evaluation phase into the pre-trained large language model and outputting the target prediction result corresponding to the sentence to be analyzed in the evaluation data set includes: For any sentence to be analyzed in the evaluation dataset, repeatedly inputting the sentence to be analyzed and the corresponding dialogue state analysis result set into the pre-trained large language model a first preset number of times, and outputting a prediction result corresponding to the sentence to be analyzed under each input; The prediction result with the largest number of occurrences is used as the target prediction result corresponding to any of the sentences to be analyzed.

3. The method according to claim 1, characterized in that The number of times the evaluation data set corresponding to each evaluation step is input into the pre-trained large language model satisfies a second preset number, and the order of results of the dialogue state analysis result set corresponding to the to-be-analyzed sentence in the evaluation data set is different in each input; there are multiple target prediction results corresponding to the to-be-analyzed sentence in the evaluation data set, each corresponding to an analysis result selected by the pre-trained large language model in the dialogue state analysis result set for each input; The correct analysis results corresponding to the sentences to be analyzed in the evaluation data set are compared with the corresponding target prediction results to obtain evaluation information of the pre-trained large language model in each evaluation link, including: Comparing the target prediction result outputted each time for the sentence to be analyzed in the evaluation data set with the corresponding correct analysis result, to obtain the comparison result corresponding to each input of the evaluation data set; According to the comparison results corresponding to each input of the evaluation data set, the evaluation information of the pre-trained large language model in the evaluation link is determined.

4. The method according to claim 3, characterized in that The comparison result includes ratio information. The target prediction result outputted for each sentence to be analyzed in the evaluation dataset is compared with the corresponding correct analysis result to obtain the comparison result corresponding to each input of the evaluation dataset, including: In the process of obtaining the ratio information corresponding to any input of the evaluation data set, comparing the correct analysis result corresponding to the sentence to be analyzed in the evaluation data set with the target prediction result corresponding to the any input to determine a first number; the first number is the number of sentences to be analyzed for which the target prediction result corresponding to the any input is the same as the corresponding correct analysis result; The ratio information of the first quantity to the second quantity is obtained as the ratio information corresponding to the evaluation data set under any input; the second quantity is the total number of statements to be analyzed in the evaluation data set.

5. The method according to claim 4, characterized in that Determining evaluation information of the pre-trained large language model in the evaluation phase according to the comparison result corresponding to each input of the evaluation data set includes: Determining average ratio information based on the ratio information corresponding to each input of the evaluation data set; The average ratio information is used as evaluation information of the pre-trained large language model in the evaluation step.

6. The method according to claim 3, characterized in that The step of inputting the evaluation data set corresponding to the evaluation phase into the pre-trained large language model and outputting the target prediction result corresponding to the sentence to be analyzed in the evaluation data set includes: Input the sentences to be analyzed in the evaluation dataset and the corresponding dialogue state analysis result set into the pre-trained large language model, and output the target prediction result corresponding to the sentences to be analyzed in the evaluation dataset under this input; Rearrange the order of the results of the dialogue state analysis result set corresponding to the sentence to be analyzed in the evaluation data set, use the reordered dialogue state analysis result set as a new dialogue state analysis result set, and return to the step of inputting the sentence to be analyzed in the evaluation data set and the corresponding dialogue state analysis result set into the pre-trained large language model until the second preset number of times is met.

7. A device for outputting evaluation information of a large language model, characterized in that: The device comprises: A data acquisition module is configured to acquire an evaluation dataset corresponding to each evaluation step of a preset dialogue state evaluation process; the dialogue state evaluation process is a process for evaluating the dialogue state perception capability of a pre-trained large language model; the evaluation dataset corresponding to each evaluation step includes sentences to be analyzed under different evaluation dimensions and corresponding dialogue state analysis result sets; the sentences to be analyzed contain dialogue state information; and the dialogue state analysis result sets corresponding to each sentence to be analyzed include a correct analysis result and at least one analysis result different from the correct analysis result. A prediction module is configured to input, in each evaluation step, an evaluation dataset corresponding to the evaluation step into the pre-trained large language model, and output a target prediction result corresponding to the sentence to be analyzed in the evaluation dataset; the target prediction result is an analysis result selected by the pre-trained large language model from the dialogue state analysis result set corresponding to the input sentence to be analyzed; A comparison module is used to compare the correct analysis results corresponding to the sentences to be analyzed in the evaluation data set with the corresponding target prediction results to obtain evaluation information of the pre-trained large language model in each evaluation link; the evaluation information is used to characterize the dialogue state perception ability of the pre-trained large language model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images