Chat service evaluation method and device, computer program product and electronic equipment

Abstract

The invention discloses a chat service evaluation method and device, a computer program product and electronic equipment, and relates to the technical field of artificial intelligence. The chat service evaluation method comprises the steps of determining chat test data of a target intelligent role through question and answer interaction between a user and the target intelligent role; and based on a preset evaluation dimension, guiding the large language model to evaluate the chat test data to obtain an intelligent evaluation result of the chat test data. According to the method, the big language model is guided to intelligently evaluate the chat data of the target intelligent role from the preset evaluation dimension, so that the comprehensiveness, accuracy and efficiency of chat service evaluation can be improved, and the time and labor costs of chat service evaluation are reduced.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of artificial intelligence technology. More specifically, the embodiments of the present disclosure relate to a chat service evaluation method, a chat service evaluation device, a computer program product, and an electronic device. Background Art

[0002] This section is intended to provide a background or context to embodiments of the invention that are recited in the claims. No description herein is admitted to be prior art by inclusion in this section.

[0003] With the development of artificial intelligence technology, various artificial intelligence characters, namely chatbots, have emerged, establishing a more direct and efficient connection between "platform content" and "users". In order to intuitively reflect the performance level of artificial intelligence characters, it is necessary to evaluate the output content of artificial intelligence characters.

[0004] In the related technologies, the method of manually evaluating AI roles by experts is inefficient and requires high labor and time costs. The method of evaluating AI roles by building an evaluation model with the help of machine learning requires high model training costs and business adaptation costs.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a chat service evaluation method, a chat service evaluation device, a computer program product and an electronic device, thereby at least to a certain extent solving the problem of high cost and low efficiency of artificial intelligence role evaluation in related technologies.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to a first aspect of the present disclosure, a chat service evaluation method is provided, the method comprising: determining chat test data of the target intelligent character through question-answer interaction between a user and the target intelligent character; based on a preset evaluation dimension, guiding a large language model to evaluate the chat test data to obtain an intelligent evaluation result of the chat test data.

[0009] In an exemplary embodiment of the present disclosure, determining the chat test data of the target intelligent character through question-and-answer interaction between the user and the target intelligent character includes: acquiring user chat data, and inputting the user chat data into the target intelligent character to obtain chat reply data of the target intelligent character; and determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character.

[0010] In an exemplary embodiment of the present disclosure, determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character includes: acquiring the character characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character; determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character, the character characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character.

[0011] In an exemplary embodiment of the present disclosure, based on the preset evaluation dimension, guiding the large language model to evaluate the chat test data to obtain the intelligent evaluation result of the chat test data, including: constructing model prompt words based on the preset evaluation dimension, the role feature data of the target intelligent character, and the chat background feature data of the target intelligent character; based on the model prompt words and the model evaluation prompt samples, guiding the large language model to evaluate the chat reply data in the chat test data to obtain the intelligent evaluation result of the chat test data.

[0012] In an exemplary embodiment of the present disclosure, the preset evaluation dimensions include a quality dimension, a role dimension, a scenario dimension and a risk dimension.

[0013] In an exemplary embodiment of the present disclosure, the method further includes: verifying the validity of the intelligent evaluation result of the chat test data to obtain a validity verification result of the intelligent evaluation result of the chat test data, and the validity verification result is used to optimize the large language model.

[0014] In an exemplary embodiment of the present disclosure, the verification of the validity of the intelligent evaluation result of the chat test data to obtain the validity verification result of the intelligent evaluation result of the chat test data includes: obtaining the manual evaluation result of the chat test data; performing consistency verification on the manual evaluation result of the chat test data and the intelligent evaluation result to obtain a consistency verification result; and determining the validity verification result of the intelligent evaluation result of the chat test data based on the consistency verification result.

[0015] According to a second aspect of the present disclosure, a chat service evaluation device is disclosed, comprising: a data acquisition module, used to determine the chat test data of the target intelligent role through question-answering interaction between a user and the target intelligent role; an intelligent evaluation module, used to guide a large language model to evaluate the chat test data based on preset evaluation dimensions, and obtain intelligent evaluation results of the chat test data.

[0016] According to a third aspect of an embodiment of the present disclosure, a computer program product is disclosed, including a computer program, which, when executed by a processor, implements the chat service evaluation method of the first aspect and possible implementation methods thereof.

[0017] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is disclosed, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute the chat service evaluation method of the above-mentioned first aspect and its possible implementation method by executing the executable instructions.

[0018] The technical solution disclosed in this disclosure has the following beneficial effects:

[0019] In the above chat service evaluation process, the chat test data of the target intelligent role is determined through the question-and-answer interaction between the user and the target intelligent role; based on the preset evaluation dimension, the large language model is guided to evaluate the chat test data to obtain the intelligent evaluation result of the chat test data. The present disclosure guides the large language model to perform intelligent evaluation on the chat data of the target intelligent role from the preset evaluation dimension, which can improve the comprehensiveness, accuracy and efficiency of the chat service evaluation, reduce the time and manpower cost of the chat service evaluation, and can better adapt to diverse and complex business evaluation scenarios.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 A schematic diagram showing a flow chart of a chat service evaluation method in this exemplary embodiment;

[0023] Figure 2 A flowchart of obtaining chat test data of a target intelligent character in this exemplary embodiment is shown;

[0024] Figure 3 A schematic diagram of an evaluation system for generative artificial intelligence in this exemplary embodiment is shown;

[0025] Figure 4 A flow chart of an intelligent evaluation in this exemplary embodiment is shown;

[0026] Figure 5 A structural block diagram of a chat service evaluation device in this exemplary embodiment is shown;

[0027] Figure 6 An electronic device for implementing the above-mentioned chat service evaluation method in this exemplary embodiment is shown.

[0028] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0029] The principles and spirit of the present disclosure will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided only to enable those skilled in the art to better understand and implement the present disclosure, and are not intended to limit the scope of the present disclosure in any way. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0030] Those skilled in the art will appreciate that the embodiments of the present disclosure may be implemented as a device, apparatus, method or computer program product. Therefore, the present disclosure may be specifically implemented in the following forms, namely: complete hardware, complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0031] According to the embodiments of the present disclosure, a chat service evaluation method, a chat service evaluation device, a computer program product and an electronic device are proposed.

[0032] Any number of elements in the drawings is for illustration and not limitation, and any naming is for distinction only and does not have any limiting meaning.

[0033] The principle and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

[0034] In the related technologies disclosed in the present invention, the method of manually evaluating artificial intelligence roles by experts is inefficient and requires high labor and time costs. The method of evaluating artificial intelligence roles by building an evaluation model with the help of machine learning requires high model training costs and business adaptation costs. In addition, since artificial intelligence role evaluation exhibits test prediction characteristics, building an evaluation model with the help of machine learning is difficult to match the specific needs of large model business evaluation when facing large model business evaluation, and cannot be adapted to diverse and complex business scenarios.

[0035] Based on the above problems, the present invention proposes a chat business evaluation method, which guides the large language model to perform intelligent evaluation on the chat data of the target intelligent role from the preset evaluation dimension, which can improve the comprehensiveness, accuracy and efficiency of the chat business evaluation, reduce the time and manpower cost of the chat business evaluation, and can better adapt to diverse and complex business evaluation scenarios.

[0036] Application Scenario Overview

[0037] It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in this respect. On the contrary, the embodiments of the present disclosure can be applied to any applicable scenario. For example, it can be applied to chat business evaluation scenarios of various artificial intelligence roles, such as artificial intelligence love assistants, artificial intelligence matchmakers, artificial intelligence customer service and other artificial intelligence roles, and the present disclosure does not specifically limit this.

[0038] Exemplary Methods

[0039] In combination with the above application scenarios, refer to Figure 1 To describe a chat service evaluation method according to an exemplary embodiment of the present disclosure.

[0040] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a chat service evaluation method according to an example implementation of the present disclosure. Figure 1 As shown, the chat service evaluation method may include:

[0041] Step S110, determining chat test data of the target intelligent character through question-answer interaction between the user and the target intelligent character;

[0042] Step S120: Based on the preset evaluation dimension, guide the large language model to evaluate the chat test data to obtain the intelligent evaluation result of the chat test data.

[0043] Implementation Figure 1The chat business evaluation method shown guides the large language model to perform intelligent evaluation on the chat data of the target intelligent role from the preset evaluation dimension, which can improve the comprehensiveness, accuracy and efficiency of the chat business evaluation, reduce the time and manpower cost of the chat business evaluation, and can better adapt to diverse and complex business evaluation scenarios.

[0044] These steps are described in detail below.

[0045] In step S110, chat test data of the target intelligent character is determined through question-answering interaction between the user and the target intelligent character.

[0046] The user here can be a real chat user or a pre-built simulated chat user, and the present disclosure does not specifically limit this. For example, a model chat user can be constructed based on some preset user characteristics, such as personality characteristics, emotional characteristics, hobby characteristics, etc.

[0047] Among them, the target intelligent role refers to the intelligent robot to be evaluated for chat business, which can be an intelligent robot that can realize specific chat business, such as artificial intelligence love assistant, artificial intelligence matchmaker, artificial intelligence customer service and other intelligent robots. This disclosure does not make specific limitations on this.

[0048] Optionally, the question-and-answer interaction between the same user and the target intelligent character may be one round of question-and-answer interaction or multiple rounds of question-and-answer interaction, which is not specifically limited in the present disclosure.

[0049] Among them, chat test data refers to test data used to evaluate the performance level of the target intelligent character, and may include but is not limited to the chat response data of the target intelligent character.

[0050] In an optional implementation, the above-mentioned determination of the chat test data of the target intelligent character through question-and-answer interaction between the user and the target intelligent character can be achieved through the following steps: obtaining user chat data, and inputting the user chat data into the target intelligent character to obtain chat reply data of the target intelligent character; and determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character.

[0051] Exemplarily, the user chat data may be chat data input by a real chat user in the chat input box of the target intelligent character, or chat data constructed by a simulated chat user according to the corresponding user characteristics, which is not specifically limited in the present disclosure.

[0052] The chat reply data refers to the reply data obtained by the target intelligent character through language reasoning on the user chat data. For example, the user inputs the user chat data: "Good morning! I plan to go to the library to study", and the target intelligent character outputs the chat reply data: "Wow, great! How is the environment there?"

[0053] After obtaining the chat response data of the target intelligent character, chat test data can be constructed based on the chat response data of the target intelligent character to provide data support for the evaluation of the target intelligent character. This can be applicable to the construction of test data for thousands of rounds of conversations, and most of these chat test data can be reused after fine-tuning, which can improve data utilization.

[0054] In an optional implementation, the above-mentioned determination of the chat test data of the target intelligent character based on the chat reply data of the target intelligent character can be achieved through the following steps: obtaining the role characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character; determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character, the role characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character.

[0055] Among them, the role characteristic data of the target intelligent character refers to the personality characteristics assigned to the target intelligent character, and may include multi-dimensional role characteristic data.

[0056] Exemplarily, the role characteristic data of the target intelligent character may include: characteristic data of basic information dimension (such as age, education, major, personality and other basic data that can identify individual characteristics), characteristic data of social relationship dimension (such as emotional relationship, family relationship, work relationship, etc. that reflects the complex information of the individual's position and association in the social network), characteristic data of interest and hobby dimension (such as characteristic data that can reflect the preferences and tendencies of the target intelligent character in his spare time), characteristic data of business function strategy dimension (such as characteristic data related to the action logic and operation method followed by the character in a specific business scenario), etc. The present disclosure does not make specific limitations on this.

[0057] Exemplarily, the character characteristic data of the target intelligent character may also include: language style characteristics, empathy ability characteristics, etc., which are not specifically limited in the present disclosure.

[0058] Among them, the chat background feature data of the target intelligent character refers to the background features when the target intelligent character outputs chat reply data, such as historical chat records, chat topics, etc., and the present disclosure does not make specific limitations on this.

[0059] Based on the chat reply data of the target intelligent character, the character characteristic data of the target intelligent character, and the chat background characteristic data of the target intelligent character, the chat test data of the target intelligent character is determined to ensure that the performance of the target intelligent character during the interaction process can more accurately fit the pre-set personality framework and chat background.

[0060] It should be noted that in actual application, when determining the chat test data of the target intelligent character, feature data of other dimensions can be flexibly added to make the performance of the target intelligent character in the interaction process more in line with the character characteristics. This disclosure does not make specific limitations on this.

[0061] For example, Figure 2 As shown, a flowchart for obtaining chat test data of a target intelligent character is provided, which may specifically include the following steps:

[0062] Step S201, obtaining user chat data, and inputting the user chat data into a target intelligent character to obtain chat reply data of the target intelligent character;

[0063] Step S202, obtaining character feature data of the target intelligent character and chat background feature data of the target intelligent character;

[0064] Step S203, determining chat test data of the target intelligent character based on the chat reply data of the target intelligent character, the character feature data of the target intelligent character, and the chat background feature data of the target intelligent character.

[0065] After obtaining the chat test data of the target intelligent character, step S120 may be continued.

[0066] In step S120, based on the preset evaluation dimension, the large language model is guided to evaluate the chat test data to obtain an intelligent evaluation result of the chat test data.

[0067] Among them, the preset evaluation dimension refers to the pre-set evaluation dimension for the target intelligent role, which may include but is not limited to one or more of the quality dimension, role dimension, scenario dimension and risk dimension.

[0068] Optionally, the preset evaluation dimensions can be further refined into multiple granularity levels and expanded to multiple sub-granularity indicators, so as to conduct a more comprehensive, accurate and systematic comprehensive evaluation and analysis of the intelligent chat service, and provide quantitative basis and decision-making support for subsequent chat service optimization and quality improvement.

[0069] For example, Figure 3 The figure shows a schematic diagram of a generative artificial intelligence evaluation system (GAI-EVAL), which further refines the quality dimension, role dimension, scenario dimension and risk dimension into multiple evaluation granularity levels.

[0070] Among them, quality dimensions may include but are not limited to: grammar, fluency, semantic relevance, logical consistency, language redundancy, contradiction, ambiguity and other evaluation granularities.

[0071] Taking grammar as an example, it can be further divided into multiple dimensions such as characters, words, sentences and paragraphs. Specifically, in the character dimension, it can be evaluated whether there are errors such as wrong characters, similar-sounding characters and similar-shaped characters; in terms of morphology, it can be evaluated whether there are errors in the use of word forms, word part collocation, etc.; at the syntactic level, it can be evaluated whether there are problems with sentence components, sentence structure and grammatical semantics; at the paragraph level, it can be evaluated whether there are errors in punctuation marks, etc. Optionally, at the grammatical level, it can also be evaluated whether there are problems with the use of emoticons, which is not specifically limited in this disclosure.

[0072] Taking logical consistency as an example, in a long conversation with the target intelligent character, it often involves switching between multiple chat topics. It can be evaluated whether there are problems such as the reply content being different from the previous topic, the logic being inconsistent, and the expressions being contradictory.

[0073] Taking language redundancy as an example, we can evaluate whether there are problems such as the reply repeating the previous content, repeating the previous content and the reply being redundant. It should be noted that language redundancy is the amount of information in a language that exceeds the minimum amount required for transmission. In language communication, people can use less information to convey messages, but use more information to express them, resulting in redundant information. Redundant expressions will confuse readers and even lead to misunderstandings of information. For example, "I'll be at your house at 6 o'clock tomorrow night. 6 o'clock tomorrow night!", the latter sentence contains redundant information.

[0074] Taking semantic ambiguity as an example, it can be evaluated whether the reply content does not understand the intention of the previous text, the content semantics is unclear and ambiguous.

[0075] Among them, role dimensions may include but are not limited to: role setting, interpersonal relationships, language style, empathy, spoken style, communication skills and other evaluation granularities.

[0076] Taking role setting as an example, in the context of AI chat services, each role can be given a clear and specific personality setting in order to evaluate whether the performance of the role in the interaction process can accurately fit the pre-set personality setting framework. Specifically, it is manifested in the following key aspects: First, in the specific knowledge scope involved in the reply content, such as the basic information of the role, the age, education and other information presented must be highly consistent with the established role setting, and no deviation or contradiction should occur. Second, for complex role information such as emotional relationships, family relationships, and work relationships, the role should show accurate understanding and grasp when replying, and even in the face of potential misleading or interference from the above information, it can accurately respond according to the personality setting to ensure the accuracy and coherence of the information. Third, the reply content should deeply reflect its inherent understanding of its own set outlook on life, values ​​and world view, and be compatible with the life experience of the role, so that the reply is highly reasonable and authentic at the semantic, emotional and logical levels, so as to achieve the goal of high-quality and immersive interaction with users. Optionally, a persona may cover multi-dimensional information elements, including basic information dimensions, such as age, education, profession, personality, and other basic data that can identify individual characteristics; social relationship dimensions, such as emotional relationships, family relationships, work relationships, and other complex information that reflects the individual's position and connections in the social network; interest and hobby dimensions, which reflect the character's preferences and tendencies in his or her spare time; and business function strategy dimensions, which are related to the action logic and operating methods followed by the character in specific business scenarios.

[0077] Taking language style as an example, we can evaluate whether the reply content can reflect delicate emotions, moods and intonation, whether it can reflect one's own language style, word habits and unique expression habits, etc. Language style refers to the different language materials and methods used by people in communication according to different communication occasions, purposes, tasks and the disposition and quality of the communicators. In the intelligent chat role, each role can be given a unique language style.

[0078] Taking empathy as an example, we can evaluate whether the reply content can fully understand the user's emotional needs, show empathy and conduct effective emotional guidance. Among them, empathy refers to the ability to put oneself in the shoes of others and experience their situation, so as to feel and understand their emotions. Empathy is not only about empathy, but also about assisting the other party in dealing with emotions, helping the other party to get out of emotions, and improving the ability to deal with similar situations in the future.

[0079] Among them, the scenario dimension may include but is not limited to: scenario-related, in-depth scenario, scenario reasonable, reasonable divergence and other evaluation granularities. For example, the general scenarios can be divided into multiple categories, such as hobbies, health and fitness, gossip, festivals and celebrations, work and occupation, food and cooking, news and current affairs, games and entertainment, personal development and learning, fashion and beauty, cars, movies and TV series, family life and friends, travel and culture, technology and innovation, history and culture, psychology and emotions, pets and animals, politics and society, science and technology, etc. For example, the multiple major categories can be further refined into multiple secondary categories. Taking "pets and animals" as an example, it can be further refined into multiple secondary categories such as "cute pet selection", "pet experience", "pet maintenance" and "pet knowledge", so as to evaluate whether the reply content of the target intelligent role is closely centered on the relevant scenarios.

[0080] Among them, the risk dimension may include but is not limited to: knowledge reserve, privacy and security and other evaluation granularities. Since the target intelligent role has boundary risks, such as data security and privacy risks, adversarial attack risks, algorithm bias and controllability risks, content compliance risks, factual error risks, etc., this exemplary embodiment classifies the relevant risks into three granularities: knowledge reserve, privacy and security, so as to evaluate from the risk dimension based on these three granularities. This disclosure does not make specific limitations on this. Exemplarily, as shown in Table 1, some evaluation granularities under the risk dimension are provided.

[0081] Table 1

[0082]

[0083]

[0084] It should be noted that, in actual application, the evaluation indicators of each dimension can be set according to the actual chat service type, and this disclosure does not make specific limitations on this.

[0085] In an optional implementation, when the chat test data of the target intelligent character includes the chat reply data of the target intelligent character, the character feature data of the target intelligent character, and the chat background feature data of the target intelligent character, the above-mentioned based on the preset evaluation dimension, guiding the large language model to evaluate the chat test data to obtain the intelligent evaluation result of the chat test data can be achieved through the following steps: based on the preset evaluation dimension, the character feature data of the target intelligent character, and the chat background feature data of the target intelligent character, constructing a model prompt word; based on the model prompt word and the model evaluation prompt sample, guiding the large language model to evaluate the chat reply data in the chat test data to obtain the intelligent evaluation result of the chat test data.

[0086] Specifically, the preset evaluation dimensions, the role characteristic data of the target intelligent character, and the chat background characteristic data of the target intelligent character may be input into a preset prompt word template to generate a model prompt word.

[0087] Among them, model prompt words refer to input instructions or key words provided when interacting with the large language model, which are used to specify the tasks that the large language model should perform and generate corresponding outputs.

[0088] Among them, model evaluation prompt samples refer to providing a small number of examples in the prompts to help the large language model better understand the objectives and output requirements of the task.

[0089] Optionally, the intelligent evaluation result may be represented by an evaluation score. A larger evaluation score indicates a better chat service performance effect; a smaller evaluation score indicates a worse chat service performance effect.

[0090] Exemplarily, as shown in Table 2, intelligent evaluation results using logical consistency as an evaluation indicator are provided.

[0091] Table 2

[0092]

[0093]

[0094] Exemplarily, as shown in Table 3, intelligent evaluation results using language redundancy as an evaluation indicator are provided.

[0095] Table 3

[0096]

[0097] Exemplarily, as shown in Table 4, intelligent evaluation results using semantic ambiguity as an evaluation indicator are provided.

[0098] Table 4

[0099]

[0100]

[0101] Exemplarily, as shown in Table 5, intelligent evaluation results using language style as an evaluation indicator are provided.

[0102] Table 5

[0103]

[0104] Exemplarily, as shown in Table 6, intelligent evaluation results with empathy as the evaluation indicator are provided.

[0105] Table 6

[0106]

[0107] By adopting a strategic approach that combines prompt words with model evaluation prompt samples, we can effectively drive large models to implement intelligent and automated evaluation processes for smart chat content, which can improve evaluation efficiency and reduce evaluation costs to a certain extent.

[0108] Optionally, based on each evaluation dimension, the large language model can be guided to evaluate the chat test data, obtain the intelligent evaluation results under each evaluation dimension of the chat test data, and determine the comprehensive intelligent evaluation results of the chat test data according to the intelligent evaluation results under each evaluation dimension of the chat test data. Specifically, the intelligent evaluation results under each evaluation dimension of the chat test data can be weighted to obtain the comprehensive intelligent evaluation results of the chat test data.

[0109] In an optional implementation, the following steps may also be performed: verifying the validity of the intelligent evaluation results of the chat test data to obtain a validity verification result of the intelligent evaluation results of the chat test data, and using the validity verification result to optimize the large language model.

[0110] Through validity verification, the accuracy, reliability and effectiveness of the evaluation of large language models can be ensured, and technical support and guarantee can be provided for the accurate evaluation of the business effects of intelligent chat applications.

[0111] Exemplarily, if the validity verification fails, the model prompt words, preset evaluation dimensions, prompt word templates, model evaluation prompt samples and other parameter information of the large language model can be adjusted to optimize the evaluation effect of the large language model. This disclosure does not make specific limitations on this.

[0112] In an optional implementation, the above-mentioned verification of the validity of the intelligent evaluation results of the chat test data to obtain the validity verification results of the intelligent evaluation results of the chat test data can be achieved through the following steps: obtaining the manual evaluation results of the chat test data; performing consistency verification on the manual evaluation results of the chat test data and the intelligent evaluation results to obtain a consistency verification result; and determining the validity verification results of the intelligent evaluation results of the chat test data based on the consistency verification results.

[0113] The manual evaluation results may be the manual evaluation results of experts on the chat test data.

[0114] For example, Kappa consistency can be used to verify the consistency between the intelligent evaluation results and the manual evaluation results. After multiple rounds of adjustments, the consistency value can be maintained above the preset threshold (for example, 0.4), which can ensure the effectiveness of the evaluation system. It should be noted that Kappa consistency is defined as follows: a statistical indicator used to measure the degree of consistency of two or more evaluators (people or classification systems) on a classification task, mainly used to evaluate the reliability of classification, especially when there is a certain degree of classification difficulty or subjectivity.

[0115] By using the Kappa coefficient as a consistency verification tool, the degree of agreement and consistency between the intelligent evaluation results and the evaluation results of multiple experts in professional fields can be verified and quantitatively analyzed, which helps to continuously optimize the evaluation capabilities of large language models and improve evaluation accuracy.

[0116] For example, Figure 4 As shown, a flowchart of intelligent evaluation is provided, which may specifically include the following steps:

[0117] Step S401, determining chat test data of the target intelligent character through question-answer interaction between the user and the target intelligent character; the chat test data includes chat reply data of the target intelligent character, character feature data of the target intelligent character, and chat background feature data of the target intelligent character;

[0118] Step S402, constructing a model prompt word based on the preset evaluation dimension, the role characteristic data of the target intelligent character, and the chat background characteristic data of the target intelligent character;

[0119] Step S403, based on the model prompt words and the model evaluation prompt samples, guiding the large language model to evaluate the chat reply data in the chat test data, and obtaining the intelligent evaluation result of the chat test data;

[0120] Step S404, obtaining manual evaluation results of the chat test data, and performing consistency verification between the manual evaluation results of the chat test data and the intelligent evaluation results to obtain consistency verification results;

[0121] Step S405, determining the validity verification result of the intelligent evaluation result of the chat test data according to the consistency verification result, and the validity verification result is used to optimize the large language model.

[0122] From the evaluation effect, the disclosure can accurately and effectively provide clear direction guidance and strategic suggestions for the continuous optimization and positive development of the intelligent chat business, which has effectively promoted the substantial improvement and all-round optimization of the overall quality of the business, and has had a positive and far-reaching impact on the technological progress and business development of the intelligent chat business. In addition, the disclosure promotes the continuous evolution of the evaluation capabilities of large language models based on human feedback, allowing artificial intelligence to quickly master human experience to ensure the accuracy of the evaluation results. For example, the target intelligent character scored 4.19 in the first evaluation. After multiple optimizations, the evaluation result of the target intelligent character reached 6.29 points, which can improve the user's chat experience.

[0123] Exemplary Devices

[0124] After introducing the method of the exemplary embodiment of the present disclosure, next, refer to Figure 5 A chat service evaluation device according to an exemplary embodiment of the present disclosure is described.

[0125] See also Figure 5 , Figure 5 FIG. 2 is a block diagram of a chat service evaluation device according to an exemplary embodiment of the present disclosure. Figure 5 As shown, a chat service evaluation device 500 according to an exemplary embodiment of the present disclosure includes: a data acquisition module 510, an intelligent evaluation module 520, wherein:

[0126] The data acquisition module 510 is used to determine the chat test data of the target intelligent character through the question-answer interaction between the user and the target intelligent character;

[0127] The intelligent evaluation module 520 is used to guide the large language model to evaluate the chat test data based on the preset evaluation dimension to obtain the intelligent evaluation result of the chat test data.

[0128] In an optional implementation, based on the aforementioned scheme, the data acquisition module 510 includes: a reply data acquisition module, used to acquire user chat data, and input the user chat data into the target intelligent character to obtain the chat reply data of the target intelligent character; a test data acquisition module, used to determine the chat test data of the target intelligent character based on the chat reply data of the target intelligent character.

[0129] In an optional implementation, based on the aforementioned scheme, the test data acquisition module can be configured to: obtain the role characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character; determine the chat test data of the target intelligent character based on the chat reply data of the target intelligent character, the role characteristic data of the target intelligent character, and the chat background characteristic data of the target intelligent character.

[0130] In an optional implementation, based on the aforementioned scheme, the intelligent evaluation module 520 can be configured to: construct model prompt words based on preset evaluation dimensions, role feature data of the target intelligent character, and chat background feature data of the target intelligent character; based on the model prompt words and model evaluation prompt samples, guide the large language model to evaluate the chat reply data in the chat test data to obtain the intelligent evaluation results of the chat test data.

[0131] In an optional implementation, based on the aforementioned solution, the preset evaluation dimensions include quality dimension, role dimension, scenario dimension and risk dimension.

[0132] In an optional implementation, based on the aforementioned solution, the chat service evaluation device further includes: a verification module, which is used to verify the validity of the intelligent evaluation results of the chat test data, and obtain the validity verification results of the intelligent evaluation results of the chat test data, and the validity verification results are used to optimize the large language model.

[0133] In an optional implementation, based on the aforementioned scheme, the verification module can be configured to: obtain manual evaluation results of chat test data; perform consistency verification on the manual evaluation results of the chat test data and the intelligent evaluation results to obtain consistency verification results; and determine the validity verification results of the intelligent evaluation results of the chat test data based on the consistency verification results.

[0134] By implementing the embodiments of the present disclosure, a large language model is guided to perform intelligent evaluation on the chat data of the target intelligent character from a preset evaluation dimension, which can improve the comprehensiveness, accuracy and efficiency of the chat business evaluation, reduce the time and manpower cost of the chat business evaluation, and better adapt to diverse and complex business evaluation scenarios.

[0135] It should be noted that although several modules or units of the chat service evaluation device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units for embodiment.

[0136] Exemplary Program Products

[0137] After introducing the apparatus according to the exemplary embodiment of the present disclosure, next, a program product according to the exemplary embodiment of the present disclosure will be described.

[0138] The exemplary embodiment of the present disclosure also provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the above-mentioned chat service evaluation method is implemented.

[0139] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0140] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0141] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0142] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure. Exemplarily, the following steps may be included:

[0143] Determine chat test data of the target intelligent character through question-and-answer interaction between the user and the target intelligent character;

[0144] Based on the preset evaluation dimensions, the large language model is guided to evaluate the chat test data to obtain the intelligent evaluation results of the chat test data.

[0145] In an optional implementation, based on the aforementioned scheme, the above-mentioned determination of the chat test data of the target intelligent character through question-and-answer interaction between the user and the target intelligent character can be achieved through the following steps: obtaining user chat data, and inputting the user chat data into the target intelligent character to obtain the chat reply data of the target intelligent character; determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character.

[0146] In an optional implementation, based on the aforementioned scheme, the above-mentioned determination of the chat test data of the target intelligent character based on the chat reply data of the target intelligent character can be achieved through the following steps: obtaining the character characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character; determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character, the character characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character.

[0147] In an optional implementation, based on the aforementioned scheme, the above-mentioned based on the preset evaluation dimension, guiding the large language model to evaluate the chat test data to obtain the intelligent evaluation result of the chat test data can be achieved through the following steps: based on the preset evaluation dimension, the role feature data of the target intelligent role, and the chat background feature data of the target intelligent role, constructing the model prompt words; based on the model prompt words and the model evaluation prompt samples, guiding the large language model to evaluate the chat reply data in the chat test data to obtain the intelligent evaluation result of the chat test data.

[0148] In an optional implementation, based on the aforementioned solution, the preset evaluation dimensions include quality dimension, role dimension, scenario dimension and risk dimension.

[0149] In an optional implementation, based on the above-mentioned scheme, the following steps may also be performed: verifying the validity of the intelligent evaluation results of the chat test data to obtain the validity verification results of the intelligent evaluation results of the chat test data, and using the validity verification results to optimize the large language model.

[0150] In an optional implementation, based on the aforementioned scheme, the above-mentioned verification of the validity of the intelligent evaluation results of the chat test data to obtain the validity verification results of the intelligent evaluation results of the chat test data can be achieved through the following steps: obtaining the manual evaluation results of the chat test data; performing consistency verification on the manual evaluation results of the chat test data and the intelligent evaluation results to obtain consistency verification results; and determining the validity verification results of the intelligent evaluation results of the chat test data based on the consistency verification results.

[0151] By implementing the embodiments of the present disclosure, a large language model is guided to perform intelligent evaluation on the chat data of the target intelligent character from a preset evaluation dimension, which can improve the comprehensiveness, accuracy and efficiency of the chat business evaluation, reduce the time and manpower cost of the chat business evaluation, and better adapt to diverse and complex business evaluation scenarios.

[0152] Exemplary Electronic Devices

[0153] After introducing the chat service evaluation method, apparatus, and program product according to the exemplary embodiment of the present disclosure, next, an electronic device according to another exemplary embodiment of the present disclosure is introduced.

[0154] The exemplary embodiment of the present disclosure also provides an electronic device capable of implementing the above-mentioned chat service evaluation method. The electronic device may include a processor and a memory. The memory stores executable instructions of the processor, such as program code. The processor executes the method in the exemplary embodiment by executing the executable instructions. In addition, the electronic device may also include a display for displaying a graphical user interface.

[0155] Reference below Figure 6 , the electronic device is exemplarily described in the form of a general-purpose computing device. It should be understood that Figure 6 The electronic device 600 shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0156] like Figure 6 As shown, the electronic device 600 may include: a processor 610 , a memory 620 , a bus 630 , an I / O (input / output) interface 640 , a network adapter 650 , and a display 660 .

[0157] The memory 620 may include a volatile memory, such as a RAM 621, a cache unit 622, and may also include a non-volatile memory, such as a ROM 623. The memory 620 may also include one or more program modules 624, such program modules 624 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or a combination thereof may include the implementation of a network environment. For example, the program module 624 may include each module in the above-mentioned device.

[0158] The processor 610 may include one or more processing units, for example, the processor 610 may include an AP (Application Processor), a modem processor, a GPU (Graphics Processing Unit), an ISP (Image Signal Processor), a controller, an encoder, a decoder, a DSP (Digital Signal Processor), a baseband processor and / or an NPU (Neural-Network Processing Unit), etc.

[0159] The processor 610 may be configured to execute executable instructions stored in the memory 620 , such as executing any one or more method steps in this exemplary embodiment.

[0160] Exemplarily, the processor 610 may perform the following steps:

[0161] Determine chat test data of the target intelligent character through question-and-answer interaction between the user and the target intelligent character;

[0162] Based on the preset evaluation dimensions, the large language model is guided to evaluate the chat test data to obtain the intelligent evaluation results of the chat test data.

[0163] In an optional implementation, based on the aforementioned scheme, the above-mentioned determination of the chat test data of the target intelligent character through question-and-answer interaction between the user and the target intelligent character can be achieved through the following steps: obtaining user chat data, and inputting the user chat data into the target intelligent character to obtain the chat reply data of the target intelligent character; determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character.

[0164] In an optional implementation, based on the aforementioned scheme, the above-mentioned determination of the chat test data of the target intelligent character based on the chat reply data of the target intelligent character can be achieved through the following steps: obtaining the character characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character; determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character, the character characteristic data of the target intelligent character and the chat background characteristic data of the target intelligent character.

[0165] In an optional implementation, based on the aforementioned scheme, the above-mentioned based on the preset evaluation dimension, guiding the large language model to evaluate the chat test data to obtain the intelligent evaluation result of the chat test data can be achieved through the following steps: based on the preset evaluation dimension, the role feature data of the target intelligent role, and the chat background feature data of the target intelligent role, constructing the model prompt words; based on the model prompt words and the model evaluation prompt samples, guiding the large language model to evaluate the chat reply data in the chat test data to obtain the intelligent evaluation result of the chat test data.

[0166] In an optional implementation, based on the aforementioned solution, the preset evaluation dimensions include quality dimension, role dimension, scenario dimension and risk dimension.

[0167] In an optional implementation, based on the above-mentioned scheme, the following steps may also be performed: verifying the validity of the intelligent evaluation results of the chat test data to obtain the validity verification results of the intelligent evaluation results of the chat test data, and using the validity verification results to optimize the large language model.

[0168] In an optional implementation, based on the aforementioned scheme, the above-mentioned verification of the validity of the intelligent evaluation results of the chat test data to obtain the validity verification results of the intelligent evaluation results of the chat test data can be achieved through the following steps: obtaining the manual evaluation results of the chat test data; performing consistency verification on the manual evaluation results of the chat test data and the intelligent evaluation results to obtain consistency verification results; and determining the validity verification results of the intelligent evaluation results of the chat test data based on the consistency verification results.

[0169] By implementing the embodiments of the present disclosure, a large language model is guided to perform intelligent evaluation on the chat data of the target intelligent character from a preset evaluation dimension, which can improve the comprehensiveness, accuracy and efficiency of the chat business evaluation, reduce the time and manpower cost of the chat business evaluation, and better adapt to diverse and complex business evaluation scenarios.

[0170] The bus 630 is used to realize the connection between different components of the electronic device 600, and may include a data bus, an address bus, and a control bus.

[0171] The electronic device 600 can communicate with one or more external devices 700 (eg, a keyboard, a mouse, an external controller, etc.) through the I / O interface 640 .

[0172] The electronic device 600 can communicate with one or more networks through the network adapter 650. For example, the network adapter 650 can provide mobile communication solutions such as 3G / 4G / 5G, or provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. The network adapter 650 can communicate with other modules of the electronic device 600 through the bus 630.

[0173] The electronic device 600 may display a graphical user interface and the like through the display 660 .

[0174] although Figure 6 Not shown, other hardware and / or software modules may also be provided in the electronic device 600, including but not limited to: a display, a microcode, a device driver, a redundant processor, an external disk drive array, a RAID (Redundant Arrays of Independent Disks) system, a tape drive, and a data backup storage system.

[0175] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the exemplary embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0176] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module" or "system". Those skilled in the art will readily think of other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0177] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A chat service evaluation method, characterized in that: The method comprises: Determining chat test data of the target intelligent character through question-answer interaction between the user and the target intelligent character; Based on the preset evaluation dimension, the large language model is guided to evaluate the chat test data to obtain an intelligent evaluation result of the chat test data.

2. The method according to claim 1, characterized in that The determining of the chat test data of the target intelligent character through the question-and-answer interaction between the user and the target intelligent character includes: Acquire user chat data, and input the user chat data into a target intelligent character to obtain chat reply data of the target intelligent character; Based on the chat reply data of the target intelligent character, chat test data of the target intelligent character is determined.

3. The method according to claim 2, characterized in that The determining the chat test data of the target intelligent character based on the chat reply data of the target intelligent character comprises: Acquire character feature data of the target intelligent character and chat background feature data of the target intelligent character; Based on the chat reply data of the target intelligent character, the character feature data of the target intelligent character, and the chat background feature data of the target intelligent character, the chat test data of the target intelligent character is determined.

4. The method according to claim 3, characterized in that The step of guiding the large language model to evaluate the chat test data based on the preset evaluation dimension to obtain the intelligent evaluation result of the chat test data includes: Constructing a model prompt word based on the preset evaluation dimension, the character feature data of the target intelligent character, and the chat background feature data of the target intelligent character; Based on the model prompt words and the model evaluation prompt samples, the large language model is guided to evaluate the chat reply data in the chat test data to obtain an intelligent evaluation result of the chat test data.

5. The method according to claim 1, characterized in that The preset evaluation dimensions include quality dimension, role dimension, scenario dimension and risk dimension.

6. The method according to claim 1, characterized in that The method further comprises: The validity of the intelligent evaluation result of the chat test data is verified to obtain a validity verification result of the intelligent evaluation result of the chat test data, and the validity verification result is used to optimize the large language model.

7. The method according to claim 6, characterized in that The validating the intelligent evaluation result of the chat test data to obtain the validating result of the intelligent evaluation result of the chat test data includes: Obtaining manual evaluation results of the chat test data; Performing consistency verification on the manual evaluation result of the chat test data and the intelligent evaluation result to obtain a consistency verification result; According to the consistency verification result, the validity verification result of the intelligent evaluation result of the chat test data is determined.

8. A chat service evaluation device, characterized in that: The device comprises: A data acquisition module, used to determine chat test data of the target intelligent character through question-answer interaction between the user and the target intelligent character; The intelligent evaluation module is used to guide the large language model to evaluate the chat test data based on a preset evaluation dimension to obtain an intelligent evaluation result of the chat test data.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images