Social big language model evaluation method, system and device and storage medium

Abstract

The invention discloses a social contact big language model evaluation method, system and device and a storage medium. The method comprises the steps that according to a service scene, social contact big language model capacity is disassembled, and an objective evaluation data set is collected; for various questions in the objective evaluation data set, constructing prompts with different person-set backgrounds, and carrying out objective evaluation on the social contact big language model through the objective evaluation data set to obtain an objective evaluation result of the social contact big language model; performing a simulation session with the social big language model, and performing evaluation in the aspects of model emotion satisfaction degree, human setup consistency, answer coherence and style significance according to session output of the social big language model to obtain a subjective evaluation result of the social big language model; and outputting an evaluation report according to the objective evaluation result and the subjective evaluation result of the social big language model. The technical problems that in the prior art, evaluation content is excessively concentrated in the knowledge understanding aspect of the model, user experience is not concerned, and the evaluation content is not comprehensive are solved.

Description

Technical Field

[0001] The present application relates to the technical field of social large language model evaluation, and in particular to a social large language model evaluation method, system, device and storage medium. Background Art

[0002] In recent years, with the rapid development of big data and computing power, as well as the advancement of deep learning algorithms, the conditions for the development of large models (especially large language models) have fully matured. These large models usually have hundreds of millions to hundreds of billions of parameters, can perform well in handling complex tasks, and the technology is becoming more and more mature. At the same time, with the popularization of artificial intelligence technology, the demand for intelligence and automation in various fields continues to grow. Due to its powerful knowledge representation and transfer learning capabilities, domain large models have become an important means to meet these needs, thus giving rise to the development of innovative applications. The evaluation method for the capabilities of large models in vertical fields has become the focus of research.

[0003] In order to ensure the quality and performance of domain big models, countries and regions have established corresponding evaluation systems. These evaluation systems usually include performance evaluation, stability testing, security verification and other aspects to ensure that big models can meet the needs in practical applications. Existing technologies focus on the evaluation of general big model capabilities. The evaluation content is overly focused on knowledge understanding and does not pay attention to user experience. In addition, the current evaluation scheme only covers objective knowledge question and answer content, lacks subjective multi-round dialogue content evaluation, and the evaluation content is not comprehensive. Summary of the invention

[0004] The present application provides a social large language model evaluation method, system, device and storage medium, which are used to improve the technical problems that the evaluation content of the existing technology is overly focused on the knowledge understanding of the model, does not pay attention to the user experience, and the evaluation content is not comprehensive.

[0005] In view of this, the first aspect of the present application provides a social large language model evaluation method, including:

[0006] Disassemble the capabilities of the social big language model according to its business scenarios, and collect objective evaluation data sets for each type of capability;

[0007] Construct prompts with different personal backgrounds for various topics in the objective evaluation data set, input the conversation context of various topics in the objective evaluation data set into the social big language model for prediction, and obtain the conversation output of the social big language model for various topics;

[0008] Using a scoring model, similarity scores are scored between the conversation output of the social big language model for each type of topic and the original conversation output of each type of topic in the objective evaluation dataset to obtain objective evaluation results of each type of capability of the social big language model;

[0009] Conduct a simulated conversation with the social big language model, evaluate the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, and obtain the subjective evaluation result of the social big language model;

[0010] An evaluation report is output based on the objective evaluation results of various types of capabilities of the social big language model and the subjective evaluation results.

[0011] Optionally, the prompt includes the personality's gender, personality's age, personality's hobbies, and personality's professional background.

[0012] Optionally, the ability types of the social big language model include basic knowledge, application ability and safety ethics.

[0013] Optionally, the simulated conversation with the social big language model is conducted, and the model emotional satisfaction, personality consistency, answer coherence, and style significance are evaluated according to the conversation output of the social big language model to obtain the subjective evaluation results of the social big language model, including:

[0014] Use the chat content of existing users to simulate conversations with the social large language model to build a conversation group;

[0015] Construct the qwen-32b model, and constrain it in the prompt according to the subjective scoring level and add examples; the subjective scoring level is divided according to the model's emotional satisfaction, personality consistency, answer coherence, and style significance;

[0016] Each conversation group is scored using the qwen-32b model, and the scoring results and scoring reasons are output to obtain the subjective evaluation results of the social big language model.

[0017] The second aspect of the present application provides a social large language model evaluation system, including:

[0018] The data collection module is used to disassemble the capabilities of the social big language model according to the business scenarios of the social big language model and collect objective evaluation data sets for each type of capability;

[0019] An objective evaluation module is used to construct prompts with different personal backgrounds for various topics in the objective evaluation data set, input the conversation context of various topics in the objective evaluation data set into the social big language model for prediction, and obtain the conversation output of the social big language model for various topics; use a scoring model to score the similarity between the conversation output of the social big language model for various topics and the original conversation output of various topics in the objective evaluation data set, and obtain objective evaluation results of various types of capabilities of the social big language model;

[0020] A subjective evaluation module is used to conduct a simulated conversation with the social big language model, evaluate the model's emotional satisfaction, personality consistency, answer coherence, and style significance according to the conversation output of the social big language model, and obtain the subjective evaluation result of the social big language model;

[0021] The report output module is used to output an evaluation report based on the objective evaluation results of various types of capabilities of the social large language model and the subjective evaluation results.

[0022] Optionally, the prompt includes the personality's gender, personality's age, personality's hobbies, and personality's professional background.

[0023] Optionally, the ability types of the social big language model include basic knowledge, application ability and safety ethics.

[0024] Optionally, the subjective assessment module is specifically used for:

[0025] Use the chat content of existing users to simulate conversations with the social large language model to build a conversation group;

[0026] Construct the qwen-32b model, and constrain it in the prompt according to the subjective scoring level and add examples; the subjective scoring level is divided according to the model's emotional satisfaction, personality consistency, answer coherence, and style significance;

[0027] Each conversation group is scored using the qwen-32b model, and the scoring results and scoring reasons are output to obtain the subjective evaluation results of the social big language model.

[0028] A third aspect of the present application provides an electronic device, the device comprising a processor and a memory;

[0029] The memory is used to store program code and transmit the program code to the processor;

[0030] The processor is used to execute the social large language model evaluation method described in any one of the first aspects according to the instructions in the program code.

[0031] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, it implements the social large language model evaluation method described in any one of the first aspects.

[0032] It can be seen from the above technical solutions that this application has the following advantages:

[0033] The social big language model evaluation method provided in the present application objectively evaluates the social big language model by collecting an objective evaluation data set, conducts a simulated conversation with the social big language model, evaluates the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, obtains the subjective evaluation results of the social big language model, and finally outputs an evaluation report based on the subjective evaluation results and the objective evaluation results, thereby realizing the combination of subjective and objective evaluations, making the evaluation content more comprehensive, and increasing the accuracy and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0035] Figure 1 A flowchart of a social large language model evaluation method provided in an embodiment of the present application;

[0036] Figure 2 An example diagram of a test data set provided in an embodiment of the present application;

[0037] Figure 3 An example diagram of a subjective scoring value range provided in an embodiment of the present application;

[0038] Figure 4 A structural diagram of a social large language model evaluation system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0040] For easier understanding, please refer to Figure 1 In a first aspect, the present application provides a social large language model evaluation method, comprising:

[0041] Step 110: Disassemble the capabilities of the social big language model according to the business scenarios of the social big language model, and collect objective evaluation data sets for each type of capability.

[0042] The corresponding model capabilities can be disassembled according to the business scenario and model positioning. For the social large language model, it is necessary to focus on three aspects: basic knowledge, application capabilities, and security ethics. Basic knowledge includes daily college entrance examination subjects, psychology, computer, education, organizational behavior, sociology, etc.; application capabilities include emotional counseling, psychological consultation, interest questions and answers, sports and entertainment, etc., and security ethics include bias discrimination, identity privacy, prompt leakage, illegal instructions, instruction induction, etc. Evaluation data sets are collected for specific capabilities. For example, 100 questions are collected for emotional counseling, 100 questions are collected for psychological consultation, and 100 questions are collected for each type of capability branch. The specific format can be referred to. Figure 2 .

[0043] Step 120: construct prompts with different personal backgrounds for each type of topic in the objective evaluation dataset, input the conversation context of each type of topic in the objective evaluation dataset into the social big language model for prediction, and obtain the conversation output of the social big language model for each type of topic.

[0044] Construct prompts with different character backgrounds for various topics in the objective evaluation dataset, including character gender, character age, character hobbies, and character professional background. Input the conversation context (input) of various topics in the objective evaluation dataset obtained in the above steps into the social big language model for prediction, and obtain the output (output) given by the social big language model under the given input (input), so as to obtain the conversation output of the social big language model for various topics.

[0045] Step 130: Use a scoring model to score the similarity between the conversation output of the social big language model for each type of topic and the original conversation output of each type of topic in the objective evaluation dataset, and obtain objective evaluation results of various types of capabilities of the social big language model.

[0046] The scoring model is used to score the similarity between the conversation output of the social big language model for various types of questions and the original conversation output of various types of questions in the objective evaluation dataset. The more similar the output results are, the higher the similarity score is. The qwen-32b model can be used as a scoring model, and constraints can be added to the system prompt for evaluation. The similarity score is based on a ten-point system. The similarity scores of the social big language model for various types of questions in all evaluation datasets are obtained, and the final scores are averaged for each ability type to obtain the objective evaluation results of various types of abilities of the social big language model, such as basic knowledge scores of 81.23, application ability of 75.58, and safety ethics of 55.60.

[0047] Step 140: Conduct a simulated conversation with the social big language model, evaluate the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, and obtain the subjective evaluation results of the social big language model.

[0048] Conduct simulated conversations based on the ability type of the social language big model, and conduct subjective experience tests on the social language big model. During the test, it is necessary to consider the four aspects of model emotional satisfaction, personality consistency, answer coherence, and style significance for evaluation. Among them, emotional satisfaction is to identify the emotional state of the social language big model's reply, whether it has changed from negative to positive, or from positive to negative, where negative emotions include disgust, anger, sadness, etc., and positive emotions include likes, happiness, etc. Personality consistency refers to the degree to which the social language big model follows the set personality background prompt, such as basic age, gender, hobbies, etc., whether they will be forgotten or changed during the chat. Answer coherence refers to whether there is a mismatch between questions and answers or incomplete responses when the social language big model replies. Style significance refers to whether the constraints on the personality style can be strictly followed. For example, if the style requires gentleness and gracefulness, then the model needs to reply as required. For the subjective evaluation of emotional satisfaction, the specific score level can be referred to Figure 3 .

[0049] It should be noted that the model can be used to simulate people's subjective evaluation, and the chat content of existing users is used to simulate conversations with the social big language model to build conversation groups; the qwen-32b model is built, and the prompt is constrained according to the subjective scoring level and examples are added; the subjective scoring level is obtained according to the model's emotional satisfaction, personality consistency, answer coherence and style significance; each conversation group is scored through the qwen-32b model, and the scoring results and scoring reasons are output to obtain the subjective evaluation results of the social big language model.

[0050] Step 150: Output an evaluation report based on the objective evaluation results and subjective evaluation results of various types of capabilities of the social big language model.

[0051] Finally, an evaluation report is output based on the subjective and objective evaluation results of the social big language model. The evaluation results of the social big model include two parts. One part is the result of subjective experience. It is necessary to indicate the experience scores of emotional satisfaction, personality consistency, answer coherence, and style significance in the evaluation report, and explain in detail the obvious deficiencies. The other part includes objective evaluation results, giving objective scores for basic knowledge, application ability, and safety ethics, analyzing the low-scoring aspects, and giving suggestions for iterative optimization.

[0052] The social big language model evaluation method provided in the present application objectively evaluates the social big language model by collecting an objective evaluation data set, conducts a simulated conversation with the social big language model, evaluates the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, obtains the subjective evaluation results of the social big language model, and finally outputs an evaluation report based on the subjective evaluation results and the objective evaluation results, thereby realizing the combination of subjective and objective evaluations, making the evaluation content more comprehensive, and increasing the accuracy and reliability.

[0053] For easier understanding, please refer to Figure 4 The second aspect of the present application provides a social large language model evaluation system, including:

[0054] The data collection module is used to disassemble the capabilities of the social big language model according to the business scenarios of the social big language model and collect objective evaluation data sets for each type of capability;

[0055] The objective evaluation module is used to construct prompts with different personal backgrounds for various topics in the objective evaluation dataset, input the conversation context of various topics in the objective evaluation dataset into the social big language model for prediction, and obtain the conversation output of the social big language model for various topics; the scoring model is used to score the similarity between the conversation output of the social big language model for various topics and the original conversation output of various topics in the objective evaluation dataset, and obtain the objective evaluation results of various types of capabilities of the social big language model;

[0056] The subjective evaluation module is used to simulate conversations with the social big language model, evaluate the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, and obtain the subjective evaluation results of the social big language model;

[0057] The report output module is used to output an evaluation report based on the objective and subjective evaluation results of various types of capabilities of the social big language model.

[0058] As a further improvement, the prompt includes the character's gender, age, hobbies and professional background.

[0059] As a further improvement, the ability types of the social big language model include basic knowledge, application ability and security ethics.

[0060] As a further improvement, the subjective assessment module is specifically used to:

[0061] Use the chat content of existing users and the social language model to simulate conversations and build conversation groups;

[0062] Construct the qwen-32b model, and constrain it in the prompt according to the subjective scoring level and add examples; the subjective scoring level is divided according to the model's emotional satisfaction, personality consistency, answer coherence, and style significance;

[0063] Each conversation group is scored using the qwen-32b model, and the scoring results and reasons are output to obtain the subjective evaluation results of the social big language model.

[0064] The social big language model evaluation system provided in the present application objectively evaluates the social big language model by collecting objective evaluation data sets, conducting simulated conversations with the social big language model, evaluating the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, obtaining the subjective evaluation results of the social big language model, and finally outputting an evaluation report based on the subjective evaluation results and the objective evaluation results, thereby realizing the combination of subjective and objective evaluations, and making the evaluation content more comprehensive, accurate, and reliable.

[0065] A third aspect of the present application provides an electronic device, the device comprising a processor and a memory;

[0066] The memory is used to store the program code and transmit the program code to the processor;

[0067] The processor is used to execute the social large language model evaluation method of the first aspect according to the instructions in the program code.

[0068] In a fourth aspect, the present application provides a computer-readable storage medium, which is used to store program code. When the program code is executed by a processor, the social large language model evaluation method of the first aspect is implemented.

[0069] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0070] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0071] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0075] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions for executing all or part of the steps of the method described in each embodiment of the present application through a computer device (which can be a personal computer, server, or network device, etc.). The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0076] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A social large language model evaluation method, characterized in that: include: Disassemble the capabilities of the social big language model according to its business scenarios, and collect objective evaluation data sets for each type of capability; Construct prompts with different personal backgrounds for various topics in the objective evaluation data set, input the conversation context of various topics in the objective evaluation data set into the social big language model for prediction, and obtain the conversation output of the social big language model for various topics; Using a scoring model, similarity scores are scored between the conversation output of the social big language model for each type of topic and the original conversation output of each type of topic in the objective evaluation dataset to obtain objective evaluation results of each type of capability of the social big language model; Conduct a simulated conversation with the social big language model, evaluate the model's emotional satisfaction, personality consistency, answer coherence, and style significance based on the conversation output of the social big language model, and obtain the subjective evaluation result of the social big language model; An evaluation report is output based on the objective evaluation results of various types of capabilities of the social big language model and the subjective evaluation results.

2. The social large language model evaluation method according to claim 1 is characterized in that: The prompt includes the character's gender, age, hobbies and professional background.

3. The social large language model evaluation method according to claim 1 is characterized in that: The ability types of the social big language model include basic knowledge, application ability and safety ethics.

4. The social large language model evaluation method according to claim 1 is characterized in that: The simulated conversation with the social big language model is performed, and the model emotion satisfaction, personality consistency, answer coherence and style significance are evaluated according to the conversation output of the social big language model, and the subjective evaluation result of the social big language model is obtained, including: Use the chat content of existing users to simulate conversations with the social large language model to build a conversation group; Construct the qwen-32b model, and constrain it in the prompt according to the subjective scoring level and add examples; the subjective scoring level is divided according to the model's emotional satisfaction, personality consistency, answer coherence, and style significance; Each conversation group is scored using the qwen-32b model, and the scoring results and scoring reasons are output to obtain the subjective evaluation results of the social big language model.

5. A social large language model evaluation system, characterized in that: include: The data collection module is used to disassemble the capabilities of the social big language model according to the business scenarios of the social big language model and collect objective evaluation data sets for each type of capability; An objective evaluation module is used to construct prompts with different personal backgrounds for various topics in the objective evaluation data set, input the conversation context of various topics in the objective evaluation data set into the social big language model for prediction, and obtain the conversation output of the social big language model for various topics; use a scoring model to score the similarity between the conversation output of the social big language model for various topics and the original conversation output of various topics in the objective evaluation data set, and obtain objective evaluation results of various types of capabilities of the social big language model; A subjective evaluation module is used to conduct a simulated conversation with the social big language model, evaluate the model's emotional satisfaction, personality consistency, answer coherence, and style significance according to the conversation output of the social big language model, and obtain the subjective evaluation result of the social big language model; The report output module is used to output an evaluation report based on the objective evaluation results of various types of capabilities of the social large language model and the subjective evaluation results.

6. The social large language model evaluation system according to claim 5, characterized in that: The prompt includes the character's gender, age, hobbies and professional background.

7. The social large language model evaluation method according to claim 5 is characterized in that: The ability types of the social big language model include basic knowledge, application ability and safety ethics.

8. The social large language model evaluation system according to claim 5, characterized in that: The subjective assessment module is specifically used for: Use the chat content of existing users to simulate conversations with the social large language model to build a conversation group; Construct the qwen-32b model, and constrain it in the prompt according to the subjective scoring level and add examples; the subjective scoring level is divided according to the model's emotional satisfaction, personality consistency, answer coherence, and style significance; Each conversation group is scored using the qwen-32b model, and the scoring results and scoring reasons are output to obtain the subjective evaluation results of the social big language model.

9. An electronic device, characterized in that: The device comprises a processor and a memory; The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the social large language model evaluation method described in any one of claims 1-4 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and when the program code is executed by the processor, it implements the social large language model evaluation method described in any one of claims 1-4.

Images