Model evaluation method and device and electronic equipment

Abstract

The invention discloses a model evaluation method, system and device, electronic equipment and a computer readable storage medium. The method comprises the following steps: acquiring a game application preset for at least one evaluation dimension; controlling the non-player character to execute the first game task through the to-be-evaluated model; a second game task is provided for each player character, and the second game task comprises the step of evaluating the non-player character under each evaluation dimension according to the execution condition of the non-player character on the first game task; and collecting evaluation data of each player character on the non-player character under each evaluation dimension, and calculating an evaluation result of the to-be-evaluated model under each evaluation dimension according to the evaluation data. The method solves the technical problem in the prior art that the evaluation result lacks objectivity and credibility due to finiteness of users participating in evaluation, subjectivity of the users and other factors.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model evaluation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology, especially the significant improvement of computing power and the advent of the big data era, neural network models (hereinafter referred to as "models") have gradually become an indispensable technical tool in many fields, which has also led to an increasing number of models. Different models have different advantages. Model callers usually evaluate multiple models based on their own business scenarios to select models that meet business needs.

[0003] The large language model is an advanced artificial intelligence model. The large number of model parameters enables the large language model to support a wider range of application scenarios. It has become a model type that is currently popular among application developers in various fields. The evaluation of large language models involves many aspects, among which user experience evaluation is mainly based on human evaluation or user surveys. These evaluation methods often lead to a lack of objectivity and credibility in the evaluation results due to factors such as the limited number of users participating in the evaluation and the subjectivity of the users.

[0004] Therefore, there is an urgent need for a model evaluation method that can overcome the influencing factors that affect the evaluation results in the existing user experience evaluation, so that the model evaluation results can be objective and highly credible. Summary of the invention

[0005] The present application provides a model evaluation method, device, electronic device, and computer-readable storage medium to solve the technical problem that the evaluation results of existing user experience evaluation methods lack objectivity and credibility due to factors such as the limited number of users participating in the evaluation and the subjectivity of the users.

[0006] In a first aspect, an embodiment of the present application provides a model evaluation method, the method comprising: obtaining a game application preset for at least one evaluation dimension, the evaluation dimension being used to characterize preset indicators for evaluating a model to be evaluated, the game application comprising at least one non-player character and multiple player characters; controlling the non-player character to perform a first game task in an application process of the game application through the model to be evaluated; in response to completion of the execution of the first game task, providing a second game task to each player character, the second game task comprising evaluating the non-player character under each evaluation dimension according to the execution of the first game task by the non-player character; collecting evaluation data of the non-player character under each evaluation dimension by each player character, and calculating the evaluation result of the model to be evaluated under each evaluation dimension according to the evaluation data.

[0007] In a second aspect, an embodiment of the present application provides a model evaluation system, the system comprising: a model supply module, a first model evaluation module, and a second model evaluation module; the model supply module is used to encapsulate multiple models to be evaluated into a unified interface, and provide the interface to the first model evaluation module; the first model evaluation module is used to obtain a game application preset for at least one evaluation dimension, the evaluation dimension is used to characterize the preset evaluation indicators for evaluating the model to be evaluated, the game application comprising at least one non-player character and multiple player characters; the non-player character is controlled by the model to be evaluated to execute a first game task in the application process of the game application; in response to the completion of the first game task, a second game task is provided to each player character, the second game task comprising: The non-player character is evaluated in each evaluation dimension based on the execution of the first game task; the evaluation data of each player character on the non-player character in each evaluation dimension is collected, and the evaluation results of the model to be evaluated in each evaluation dimension are calculated based on the evaluation data; the second model evaluation module is used to connect the non-player character controlled by the model to be evaluated to a crowdsourcing platform, so that users of the crowdsourcing platform evaluate the non-player character in each evaluation dimension based on the execution of the first game task by the non-player character; the evaluation data of the crowdsourcing platform users on the non-player character in each evaluation dimension is collected, and the crowdsourcing evaluation results of the model to be evaluated in each evaluation dimension are calculated based on the evaluation data; the evaluation results are compared with the crowdsourcing evaluation results to provide credibility verification of the evaluation results.

[0008] In a third aspect, an embodiment of the present application provides a model evaluation device, the device comprising: a game application acquisition unit, a game task execution unit, a game task providing unit, and an evaluation result calculation unit; the game application acquisition unit is used to acquire a game application preset for at least one evaluation dimension, the evaluation dimension is used to characterize a preset indicator for evaluating a model to be evaluated, and the game application comprises at least one non-player character and multiple player characters; the game task execution unit is used to control the non-player character to perform a first game task in an application process of the game application through the model to be evaluated; the game task providing unit is used to provide a second game task to each player character in response to completion of the execution of the first game task, the second game task comprising evaluating the non-player character under each evaluation dimension according to the execution of the first game task by the non-player character; the evaluation result calculation unit is used to collect evaluation data of each player character on the non-player character under each evaluation dimension, and calculate the evaluation result of the model to be evaluated under each evaluation dimension according to the evaluation data.

[0009] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; the processor is used to execute the one or more computer instructions to implement the above method.

[0010] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having one or more computer instructions stored thereon, which, when executed by a processor, executes the above method.

[0011] Compared with the prior art, the model evaluation method provided by the present application includes: obtaining a game application preset for at least one evaluation dimension, the evaluation dimension is used to characterize the preset indicators for evaluating the model to be evaluated, and the game application includes at least one non-player character and multiple player characters; controlling the non-player character to perform a first game task in the application process of the game application through the model to be evaluated; in response to the completion of the first game task, providing a second game task to each player character, the second game task including evaluating the non-player character under each evaluation dimension according to the execution of the first game task by the non-player character; collecting the evaluation data of each player character on the non-player character under each evaluation dimension, and calculating the evaluation result of the model to be evaluated under each evaluation dimension according to the evaluation data. First, the method integrates model evaluation into the game application, issues game tasks to non-player characters and player characters, so that the player characters can evaluate the execution of the game tasks (first game tasks) by non-player characters in the process of executing the game tasks (second game tasks), so that players can participate in model evaluation without perception, and improve the objectivity of the evaluation results; second, the method integrates model evaluation into the game application. Since the number of players participating in the game application is large and the portraits are diverse, it undoubtedly overcomes the defect of a limited number of users participating in the evaluation, and reduces the influence of personal factors such as user subjectivity, personal bias, self-selection bias, and personal understanding bias on the evaluation results, and comprehensively improves the credibility of the evaluation results. Therefore, the model evaluation method provided by the present application is a method for realizing imperceptible evaluation of the model based on integrating model evaluation into the game application, which solves the technical problem that the evaluation results lack objectivity and credibility due to factors such as the limited number of users participating in the evaluation and the subjectivity of the users in the existing user experience evaluation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is an application system diagram of the model evaluation method provided in the embodiment of the present application;

[0013] Figure 2 is a flow chart of the model evaluation method provided in the first embodiment of the present application;

[0014] Figure 3is a schematic diagram of a model evaluation system provided in the second embodiment of the present application;

[0015] Figure 4 is a schematic diagram of the structure of a model evaluation device provided in the third embodiment of the present application;

[0016] Figure 5 It is a schematic diagram of the structure of an electronic device provided in the fourth embodiment of the present application. DETAILED DESCRIPTION

[0017] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0018] With the rapid development of computer technology, especially the significant improvement in computing power and the advent of the big data era, neural network models (hereinafter referred to as "models") have gradually become an indispensable technical tool in many fields. A model is a computer neural network that imitates the structure and function of the human brain. By learning complex patterns and relationships in data, it can perform tasks such as classification, regression, clustering, and prediction. With the demand for models in various fields, the types and number of models are increasing. Different types of models have different advantages, which gives model callers more choices. Model callers usually combine their own business scenarios, connect to multiple models, evaluate each model, and select one or more models that meet business needs, so that they can schedule models according to different user needs and provide model services during the business application stage.

[0019] Large Language Model (LLM) is an advanced artificial intelligence model built using deep learning technology. Large language models mainly learn language grammar, semantics, style and other characteristics through training with a large amount of text data, so as to achieve the understanding and generation of natural language. The core of large language models lies in their huge scale, with millions or even billions of model parameters. The huge scale allows large language models to capture more complex language structures and patterns, and thus support a wider range of application scenarios, such as natural language understanding, natural language generation, dialogue, translation, etc. Therefore, large language models have become a model type that is currently very popular among application developers in various fields.

[0020] The evaluation of large language models involves many aspects, such as model performance, robustness, fairness, efficiency, user experience, etc. Among them, user experience evaluation is a model evaluation content that comprehensively considers the language logic and comprehension, response speed, accuracy, etc. of the large language model, which can enable the model caller to fully understand the performance of the large language model in practical applications. Therefore, user experience evaluation occupies an important position in model evaluation. At present, user experience evaluation is mainly based on human evaluation or user surveys. The human evaluation method is to conduct subjective evaluation of the output content of the model by human reviewers, and the user survey method is to collect user feedback on model performance and experience through questionnaires and other methods. These two evaluation methods can provide model callers with user experience evaluation results to a certain extent, but both have some shortcomings. For example, human evaluation methods have the following shortcomings: First, they are highly subjective. Human reviewers are highly subjective, and there may be large differences between different reviewers, which often leads to inconsistent evaluation results. Second, they are costly and require a lot of manpower and time, especially when large-scale evaluations are required, the cost will be very high. Third, they are not repeatable. Due to the subjectivity and human factors of human reviewers, it is difficult to ensure that the evaluation can be repeated, and subsequent evaluations may not be able to completely replicate the conditions and results of the initial evaluation. Fourth, there is a risk of bias. The personal preferences and background of human reviewers may affect the evaluation results, increasing the risk of bias in the evaluation results. Fifth, they are limited in scale. Due to human resource constraints, they can usually only evaluate a limited number of models and scenarios, making it difficult to cover all possible usage scenarios. Sixth, they are time-consuming. The entire evaluation process may be very time-consuming, which limits the frequency and timeliness of model evaluation. For another example, the user survey method has the following shortcomings: First, the response rate is low. Users may not be willing to spend time completing the questionnaire, resulting in a low response rate for the model evaluation, which affects the representativeness and reliability of the evaluation results; second, self-selection bias. Only users who have a strong positive or negative experience with the model are more inclined to participate in the survey, which will undoubtedly lead to a self-selection bias in the evaluation results; third, the quality of the feedback data. Users may not be serious when filling out the questionnaire, or give random answers, which affects the quality of the feedback data; fourth, understanding bias. Different users may have different understandings of the questions in the questionnaire, which will lead to problems with the accuracy and consistency of the answers to the questions; fifth, quantitative analysis is difficult. It is difficult to conduct quantitative analysis on users' subjective feedback, especially open-ended questions, which are difficult to systematically summarize and generalize; sixth, feedback delay. User surveys are often conducted after users experience the model, which cannot reflect the user experience in real time and have timeliness issues; seventh, limited depth and breadth. Questionnaire surveys may not be able to explore all possible user experience issues in depth and it is difficult to fully cover the diverse needs and feedback of users. In summary, the existing evaluation methods have the problem that the evaluation results lack representativeness and credibility due to factors such as the limited number of users participating in the evaluation and the subjectivity of the users.

[0021] In view of this, the present application provides a model evaluation method, which is a method for realizing imperceptible evaluation of the model based on integrating model evaluation into game applications. First, the method integrates model evaluation into game applications, and issues game tasks to non-player characters and player characters, so that the player characters evaluate the execution of game tasks (first game tasks) by non-player characters in the process of executing game tasks (second game tasks), thereby enabling players to participate in model evaluation without perception, thereby improving the objectivity of the evaluation results; second, the method integrates model evaluation into game applications. Since the number of players participating in game applications is large and their portraits are diverse, it undoubtedly overcomes the defect of a limited number of users participating in the evaluation, and greatly reduces the influence of personal factors such as user subjectivity, personal bias, self-selection bias, and personal understanding bias on the evaluation results, thereby comprehensively improving the credibility of the evaluation results.

[0022] The model evaluation method, device, electronic device, and computer-readable storage medium described in this application are further described in detail below in conjunction with specific embodiments and drawings.

[0023] Figure 1 : is an application system diagram of the model evaluation method provided in the embodiment of the present application. Figure 1 As shown, the system includes a first user terminal 101, a second user terminal 102, a first server terminal 103 and a second server terminal 104. The first user terminal 101 and the second user terminal 102 can be any device such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a personal digital assistant (PDA), etc. Specifically, the first user terminal 101 is a terminal device corresponding to a business application developer (i.e., a model caller), and the second user terminal 102 is a terminal device corresponding to a business application user (i.e., a user). The first server terminal 103 is deployed with multiple models, and the second server terminal 104 is deployed with business applications and model evaluation methods. When conducting model evaluation, the second server 104 connects the various models deployed in the first server 103 to the business application, and provides the business application to the second user terminal 102. The user generates usage data based on the use of the business application by the second user terminal 102. The second user terminal 102 sends the generated usage data to the second server 104. The second server 104 generates an evaluation result for the model based on the received usage data, and sends the evaluation result to the first user terminal 101, so that the business application developers can screen the models that meet the business application based on the evaluation results.

[0024] The first embodiment of the present application provides a model evaluation method, which is deployed in Figure 1The second server 104 is used to provide model evaluation services to application developers. The method provided in this embodiment is particularly suitable for user experience evaluation of large language models.

[0025] Figure 2 is a flow chart of the model evaluation method provided in this embodiment. Figure 2 The model evaluation method provided in this embodiment is described in detail. The embodiments described below are used to explain the technical solution of this application and are not intended to be used as limitations for actual use.

[0026] like Figure 2 As shown, the model evaluation method provided in this embodiment includes the following steps S210 to S240:

[0027] Step S210, obtaining a game application preset for at least one evaluation dimension, where the evaluation dimension is used to characterize a preset indicator for evaluating a model to be evaluated, and the game application includes at least one non-player character and multiple player characters.

[0028] The evaluation dimensions can be understood as indicators used to evaluate the performance of the model. A series of indicators are set from multiple angles or aspects, and the model is evaluated for each indicator. This can comprehensively measure the performance of the model and help the model caller to have a more comprehensive understanding of the advantages and disadvantages of the model. User experience evaluation of large language models can usually include the following evaluation dimensions: first, the model dialogue ability dimension, to evaluate the coherence, fluency, consistency, and scalability of the model dialogue; second, the model personality consistency dimension, to evaluate the model speech consistency (i.e., language style, expression habits consistency), model behavior consistency, model knowledge consistency, model knowledge illusion consistency, etc.; third, the model scenario consistency dimension, to evaluate the model personality adaptability to the scenario in the context of conflict between the model personality and the application scenario; fourth, the model worldview consistency dimension, to evaluate the consistency between the current dialogue and the worldview background or the worldview background assigned by the user; fifth, the expressiveness dimension of the model output content, to evaluate the theme, decision-making, plot diversity, personality expressiveness, content expressiveness, etc. of the model output content; sixth, the model subjective feeling dimension, to evaluate the model anthropomorphism, human-like performance, model communication skills, subjective comfort, sense of companionship, empathy, etc.; seventh, the safety dimension of the model output content, to evaluate whether the model output content has no bad guidance, etc.

[0029] In an optional implementation, the evaluation dimensions described in this embodiment include at least one of the following dimensions: language interaction ability, language and role consistency, language and scene consistency, language content expressiveness, role anthropomorphism, and language content security.

[0030] The game application is preset by the model caller (e.g., game developer) for one or more evaluation dimensions. Specifically, the model caller designs a game application that can obtain evaluation data of the evaluation dimension according to the evaluation dimension that the model caller wants to know. For example, if the model caller wants to know the intelligence level of the model, he will design a game application similar to guessing riddles, so as to obtain the evaluation data of the correctness of the answers output by the evaluation model based on the game application, so as to understand the intelligence level of the model to be evaluated.

[0031] In this embodiment, the game application includes at least one preset non-player character and multiple player characters, wherein the non-player character is used to interact with the evaluation model, and the player character is used to interact with real players participating in the game application. The purpose of obtaining the game application is to provide an evaluation scenario for model evaluation.

[0032] Step S220: controlling the non-player character to execute a first game task in the application process of the game application through the model to be evaluated.

[0033] In an optional implementation, controlling a non-player character to perform a first game task in an application process of a game application through the model to be evaluated may specifically include: in response to a start instruction for the game application, connecting the model to be evaluated to the game application, so that the model to be evaluated controls the non-player character to perform the first game task in the application process of the game application.

[0034] The opening instructions can be understood as the player's Figure 1 The second user terminal 102 shown sends the instruction information for starting the game match to the second server terminal 104. The instruction information may be generated by the second user terminal 102 based on the player's opening operation. Exemplarily, the second user terminal 102 generates a game opening instruction in response to the player opening the game application and clicking the game opening control, and sends the game opening instruction to the second server terminal 104. In response to receiving the game opening instruction, the second server terminal 104 runs the game application and connects the model to be evaluated to the game application based on the model evaluation method to control the non-player character in the game application.

[0035] In an optional implementation, the to-be-evaluated model includes multiple models, based on which the game application also includes multiple non-player characters.

[0036] Controlling a non-player character to perform a first game task in an application process of a game application by using a model to be evaluated may specifically include: connecting multiple models to be evaluated to the game application, and establishing corresponding relationships with multiple non-player characters, so that each model to be evaluated controls a non-player character to perform the first game task in the application process of the game application.

[0037] When there are multiple models to be evaluated, the multiple models to be evaluated can be connected to the game application together, and the multiple models to be evaluated can be matched one-to-one with multiple non-player characters in the game application, so that one model to be evaluated controls one non-player character to perform game tasks in the application process.

[0038] In an optional implementation, the correspondence between the multiple models to be evaluated and the multiple non-player characters is randomly established. Specifically, the multiple models to be evaluated are connected to the game application, and a non-player character is randomly assigned to each model to be evaluated, and the correspondence between the model to be evaluated and the assigned non-player character is established. In another optional implementation, the correspondence between the multiple models to be evaluated and the multiple non-player characters is established according to a preset mapping table. Specifically, a mapping table recording the binding relationship between the multiple models to be evaluated and the multiple non-player characters is pre-constructed. When the multiple models to be evaluated are connected to the game application, the information is read from the mapping table, each model to be evaluated is bound to a non-player character with a binding relationship, and the correspondence between the model to be evaluated and the bound non-player character is established.

[0039] In an optional implementation, multiple models to be evaluated have a unified interface. The interface defines a set of rules and specifications to enable interaction and communication between different software components or systems. In this embodiment, the interface refers to the rules and specifications for accessing the model to be evaluated to the game application. When there are multiple models to be evaluated, the multiple models to be evaluated will be encapsulated as a unified interface in advance to make model access more convenient. Optionally, multiple open source models and closed source models are encapsulated into a unified interface, and model parameters such as the character description, maximum model reply length, reply diversity, memory rounds, and knowledge base of each model are exposed to maximize the customization requirements of the model caller for model calling. Optionally, the encapsulated model unified interface can be further encapsulated as a model unified interface installation package to enhance the reusability of the model unified interface. Optionally, the encapsulated model unified interface installation package can be further converted into an AOP SDK (Software Development Kit) through AOP (Agent Oriented Programming) for model callers to call.

[0040] Based on this, multiple models to be evaluated are connected to the game application, and corresponding relationships are established with multiple non-player characters, which may specifically include: controlling the game application to call multiple models to be evaluated based on a unified interface, and binding a non-player character to each model to be evaluated. In this embodiment, each model to be evaluated can control the non-player character bound to it to perform the first game task in the application process of the game application.

[0041] The first game task can be understood as a game task provided for a non-player character, which can be preset in the game application development stage or input by the player through the second user terminal 102. The model to be evaluated interacts with the player character by controlling the non-player character to perform the first game task, so that the player perceives the performance level of the model to be evaluated from the execution of the first game task to provide evaluation data. In an optional implementation, the first game task can be a natural language interaction with the player character. For example, the player character gives a riddle, the model to be evaluated parses the riddle, and controls the non-player character to give the answer. For another example, the player character gives a prompt word, the model to be evaluated parses the prompt word according to the prompt word and the dialogue context, and controls the non-player character to give the supplementary content of the prompt word.

[0042] Step S230, in response to the first game task being completed, providing each player character with a second game task, the second game task including evaluating the non-player character under various evaluation dimensions according to the non-player character's execution of the first game task.

[0043] The second game task can be understood as a game task provided for the player character, and the game task corresponds to the first game task. The player character provides evaluation data for the non-player character through the second game task. Specifically, the second game task is to evaluate the non-player character under various evaluation dimensions according to the execution of the first game task by the non-player character, and the evaluation data is the basis for the later calculation of the evaluation results of the model to be evaluated. In an optional implementation, the first game task can be a natural language interaction with the player character, then the second game task is to evaluate the non-player character under various evaluation dimensions according to the interaction of the non-player character and the player character in natural language. For example: the first game task is for the non-player character to guess the riddle given by the player character, and the second game task can include the player character evaluating the correctness of the answer given by the non-player character. For another example: the first game task is for the non-player character to supplement the prompt word given by the player character, and the second game task can include the player character evaluating the anthropomorphism of the supplementary content of the prompt word given by the non-player character.

[0044] Optionally, the execution of the first game task by the non-player character may include not only the execution result of the first game task by the non-player character, but also the execution process of the first game task by the non-player character, which is determined according to the preset evaluation dimension. Exemplarily, the evaluation dimension is the expressiveness dimension of the model output content, then the basis for evaluating the non-player character is the execution result of the first game task by the non-player character, such as whether the answer given by the non-player character is correct. Exemplarily, the evaluation dimension is the model reaction speed dimension, then the basis for evaluating the non-player character is the execution process of the first game task by the non-player character, such as the time taken by the non-player character to give the answer after the player character gives the riddle.

[0045] Optionally, the evaluation of the non-player characters may be to score the non-player characters in each evaluation dimension according to the performance of the non-player characters in performing the first game task. For example, the non-player characters may be scored 8 points in the dimension of the expressiveness of the model output content, and 5 points in the dimension of the model reaction speed. Optionally, the evaluation of the non-player characters may also be to rank the multiple non-player characters in each evaluation dimension according to the performance of the multiple non-player characters in performing the first game task. For example, there are 5 non-player characters, corresponding to 5 models to be evaluated, and the result of ranking the non-player characters in the dimension of the expressiveness of the model output content is [non-player character 2, non-player character 5, non-player character 1, non-player character 3, non-player character 4], and the result of ranking the non-player characters in the dimension of the model reaction speed is [non-player character 2, non-player character 1, non-player character 4, non-player character 3, non-player character 5]. The specific implementation method is not limited here.

[0046] Step S240, collecting evaluation data of each player character on the non-player character in each evaluation dimension, and calculating the evaluation result of the model to be evaluated in each evaluation dimension based on the evaluation data.

[0047] In an optional implementation, various operations and operation results of non-player characters and player characters in executing game tasks in the application process of the game application are recorded in the game log. Based on this, the evaluation data of each player character on the non-player character under each evaluation dimension can be collected from the game log corresponding to the player account controlling each player character. The evaluation data can be the scoring data of the non-player character under a certain evaluation dimension, or the ranking data of multiple non-player characters under a certain evaluation dimension, without specific limitation.

[0048] Calculating the evaluation results of the model to be evaluated under each evaluation dimension according to the evaluation data may be to perform summing processing, average processing, etc. on multiple evaluation data under each evaluation dimension. In this embodiment, an evaluation data obtained after processing multiple evaluation data under a certain evaluation dimension is used as the evaluation result of the model to be evaluated under the evaluation dimension. Exemplarily, there are 10 player characters in the game application, and the scores of these 10 player characters for non-player characters under the dimension of expressiveness of the model output content are 8 points, 7 points, 8 points, 10 points, 5 points, 7 points, 6 points, 8 points, 9 points, and 9 points respectively. Then, the evaluation result of the model to be evaluated that controls the non-player character under the dimension of expressiveness of the model output content is 7.7 points (the average of the above ten scores).

[0049] In an optional implementation, after calculating the evaluation results of the model to be evaluated under each evaluation dimension, the comprehensive evaluation result of the model to be evaluated can also be calculated by summing, averaging, weighted sum, etc. For example, the evaluation result of the model to be evaluated in the dimension of model dialogue ability is 8 points, the evaluation result in the dimension of model personality consistency is 5 points, the evaluation result in the dimension of expressiveness of model output content is 10 points, and the evaluation result in the dimension of security of model output content is 10 points. Then, the comprehensive evaluation result of the model to be evaluated is 8.25 points (the average of the above four evaluation results).

[0050] In an optional implementation, when the evaluation model includes multiple evaluation models, the evaluation data of each player character on the non-player character under each evaluation dimension is collected, and the evaluation results of the model to be evaluated under each evaluation dimension are calculated based on the evaluation data, which may specifically include the following steps S241 to S243:

[0051] Step S241, for each model to be evaluated, collect evaluation data of each player character on the non-player character corresponding to the model to be evaluated under each evaluation dimension.

[0052] Step S242, based on the evaluation data corresponding to each model to be evaluated, calculate the evaluation results of each model to be evaluated in each evaluation dimension.

[0053] Step S243, sorting the multiple models to be evaluated according to the evaluation results of each model to be evaluated under each evaluation dimension, and generating ranking results of the multiple models to be evaluated under each evaluation dimension.

[0054] When there are multiple models to be evaluated, the ranking results of the multiple models to be evaluated under each evaluation dimension can also be calculated, that is, the ranking list of the multiple models to be evaluated under each evaluation dimension, so that the model caller can filter and call the model that better suits the application scenario according to the ranking list.

[0055] Exemplarily, the game application set for the two model evaluation dimensions of model dialogue ability and model output content expression is a supplementary content game, in which 5 player characters and 4 non-player characters are set, the 5 player characters are referees, and the 4 non-player characters are contestants. In the application process of the game application, the 5 player characters give prompts in turn, and the 4 non-player characters supplement the relevant content according to the prompts synchronously. Then, the 5 player characters score the 4 non-player characters in the two evaluation dimensions of model dialogue ability and model output content expression according to the supplement of the prompts by the 4 non-player characters. After the game process ends, the game logs of the player accounts corresponding to the 5 player characters are obtained, and the scoring data of the 4 non-player characters under the two evaluation dimensions are collected from the game logs, and the evaluation results of the 4 models to be evaluated corresponding to the 4 non-player characters under the two evaluation dimensions are calculated based on the scoring data. Assume that under the calculated model dialogue capability dimension, the evaluation result of model 1 to be evaluated is 5.5 points, the evaluation result of model 2 to be evaluated is 8.5 points, the evaluation result of model 3 to be evaluated is 3.7 points, and the evaluation result of model 4 to be evaluated is 9.0 points; under the model output content expression dimension, the evaluation result of model 1 to be evaluated is 8.5 points, the evaluation result of model 2 to be evaluated is 6.5 points, the evaluation result of model 3 to be evaluated is 5.7 points, and the evaluation result of model 4 to be evaluated is 8.0 points. Then, the ranking of the four models to be evaluated under the model dialogue capability dimension is [model 4 to be evaluated, model 2 to be evaluated, model 1 to be evaluated, model 3 to be evaluated], and the ranking under the model output content expression dimension is [model 1 to be evaluated, model 4 to be evaluated, model 2 to be evaluated, model 3 to be evaluated]. Based on the above rankings, when the model caller expects to use a model with strong conversational capabilities, the model to be evaluated 4 can be called; when the model caller expects to use a model with strong output content expression, the model to be evaluated 1 can be called.

[0056] In an optional implementation, the method provided in this embodiment may further include the following steps S11 to S13:

[0057] Step S11, connecting the non-player character controlled by the model to be evaluated to the crowdsourcing platform, so that users of the crowdsourcing platform can evaluate the non-player character under various evaluation dimensions according to the execution of the first game task by the non-player character.

[0058] Step S12, collecting evaluation data of non-player characters from users of the crowdsourcing platform in each evaluation dimension, and calculating crowdsourcing evaluation results of the model to be evaluated in each evaluation dimension based on the evaluation data.

[0059] Step S13, comparing the crowdsourcing evaluation results with the evaluation results to provide credibility verification of the evaluation results.

[0060] The crowdsourcing platform is an Internet-based collaborative model that uses a large amount of human resources to complete specific tasks. These tasks can be data annotation, translation, design, programming, market research, creative creation, etc. The core idea is to outsource the work originally completed by internal employees or specific teams to an uncertain number of the public. In this embodiment, the specific task is the evaluation task of the evaluation model to be evaluated, and the users of the crowdsourcing platform can be high-level players and / or professionals in model development and evaluation.

[0061] In an optional implementation, the first game task and the execution of the first game task by the non-player character can be sent to the crowdsourcing platform, so that the crowdsourcing platform users can evaluate the model to be evaluated corresponding to the non-player character in each evaluation dimension based on the execution of the first game task by the non-player character, and the evaluation method is consistent with the method in which the player character performs the second game task in the game application. Exemplarily, the first game task in the game application is that the non-player character guesses the riddle given by the player character, and the second game task is that the player character scores the time taken by the non-player character to give the answer and the correctness of the answer. After obtaining the scoring data of the player character on the non-player character, and calculating the evaluation results of the model to be evaluated in the model reaction speed dimension and the model output content expression dimension based on the scoring data, the riddle given by the player character and the answer given by the non-player character and the time taken to give the answer will be sent to the crowdsourcing platform, so that the crowdsourcing platform users can score the time taken by the non-player character to give the answer and the correctness of the answer.

[0062] In another optional implementation, the non-player character in the game application that is controlled by the model to be evaluated can be connected to the page of the crowdsourcing platform, so that the users of the crowdsourcing platform can perform the same operations as the player character in the game application from the perspective of the player character, such as asking riddles to the non-player character controlled by the model to be evaluated, and scoring the model to be evaluated under various evaluation dimensions according to the answers given by the non-player character and the process of giving the answers.

[0063] After the crowdsourcing platform users have completed the evaluation, the evaluation data of the crowdsourcing platform users on the model to be evaluated under each evaluation dimension will be collected, and the crowdsourcing evaluation results of the model to be evaluated under each evaluation dimension will be calculated based on the evaluation data. The calculation method of the crowdsourcing evaluation results is the same as the calculation method of the player evaluation results (i.e., the evaluation results calculated based on the evaluation scores given by the players), so it will not be repeated here.

[0064] In this embodiment, the purpose of calculating the crowdsourcing evaluation results of the model to be evaluated is to serve as a comparison with the player evaluation results, on the one hand to verify the correctness of the player evaluation results, and on the other hand to prove to the model caller that the player evaluation results are credible.

[0065] In an optional implementation, the model evaluation method provided in this embodiment may further include the following steps S21 to S22:

[0066] Step S21, screening training samples according to the evaluation data of each player character on the non-player character under each evaluation dimension, the training samples including the first game task and the execution results of the first game task by the non-player character whose evaluation data is higher than a preset score threshold.

[0067] Step S22: training the model to be trained based on the training samples.

[0068] The above steps serve as downstream projects of the model evaluation method provided in this embodiment, and their purpose is to screen out high-quality model output data as training data, and to optimize the training of other models that are in the research and development stage or have insufficient performance.

[0069] In an optional implementation, the training data includes the first game task and a high-quality execution result. Optionally, the high quality is screened based on the evaluation data of the player character. A score threshold can be preset as a high-quality measurement standard. When the evaluation data of the first game task execution result of the player character on the non-player character is higher than the preset score threshold, the execution result is regarded as a high-quality execution result and combined with the first game task as training data. Exemplarily, the player character scores the correctness of the answer of non-player character 1 as 9 points, and scores the correctness of the answer of non-player character 2 as 3 points. The preset score threshold is 8 points, then the answer given by non-player character 1 is a high-quality execution result, and the riddle given by the player character and the answer given by non-player character 1 can be combined as a training sample. Training the model to be trained with the training sample can improve the output quality of the model and optimize the model performance.

[0070] Optionally, training the model to be trained based on the training sample may include the following steps: first, inputting the first game task in the training sample into the model to be trained, so that the model to be trained parses the first game task based on the current model parameters and outputs the predicted execution result for the first game task; second, comparing the predicted execution result with the execution result in the training sample, and calculating the loss function between the two; third, using the gradient descent method, based on minimizing the loss function, updating the current model parameters of the model to be trained; fourth, iteratively training the model to be trained through the above steps until the loss function between the predicted execution result output by the model and the execution result in the training sample is reduced to the target value, completing the training of the model to be trained. Since the training sample is a high-quality sample obtained through evaluation, training the model to be trained based on the training sample can make the output quality of the model higher and the robustness of the model stronger.

[0071] In an optional implementation, the game application provided in this embodiment is a sub-application embedded in a main game application, and the main game application includes a massively multiplayer online role-playing game. The method provided in this embodiment may also include: in response to the completion of the second game task, issuing rewards to the player accounts corresponding to each player character, and exiting the game application and entering the main game application, so that the player account uses the rewards in the main game application. In this way, players participating in the main game application can be encouraged to participate in the sub-application, further ensuring the number of users participating in the model evaluation.

[0072] In an optional implementation, the method provided in this embodiment may further include the following steps: broadcasting the first game task, the execution of the first game task by the non-player character, and the evaluation data of the player character on the non-player character under each evaluation dimension. Optionally, the execution of the game task by the player character and the non-player character is published on the game social platform in the form of communication information, which can attract more players to participate in the game application to further increase the number of users participating in the model evaluation, and reversely supervise the evaluation behavior of the players to avoid players from giving abnormal evaluation data, thereby further enhancing the accuracy of the evaluation results.

[0073] The first embodiment above provides an optional model evaluation method. First, the method integrates model evaluation into the game application, issues game tasks to non-player characters and player characters, so that the player characters evaluate the execution of the game tasks (first game tasks) performed by the non-player characters in the process of performing the game tasks (second game tasks), and realizes that players participate in model evaluation without perception, thereby improving the objectivity of the evaluation results; second, the method integrates model evaluation into the game application. Since there are a large number of players participating in the game application and their portraits are diverse, it undoubtedly overcomes the defect of a limited number of users participating in the evaluation, and greatly reduces the influence of personal factors such as user subjectivity, personal bias, self-selection bias, and personal understanding bias on the evaluation results, thereby comprehensively improving the credibility of the evaluation results; third, the method provides high-quality training samples for the model to be trained, thereby improving the ability of the model to be trained to master knowledge. In short, the model evaluation method provided in this embodiment is a method for realizing non-perceptual evaluation of the model based on integrating model evaluation into the game application, thereby ensuring the objectivity, accuracy, and credibility of the evaluation results.

[0074] It should be noted that the examples in the first embodiment are only for explaining the method described in this application and are not intended to be limiting for actual use. The model evaluation method provided in this application includes but is not limited to the method described in the first embodiment.

[0075] The second embodiment of the present application provides a model evaluation system. Figure 330 is a schematic diagram of a model evaluation system provided in this embodiment. The system includes: a model supply module 301 , a first model evaluation module 302 , and a second model evaluation module 303 .

[0076] The model supply module 301 is used to encapsulate multiple models to be evaluated into a unified interface and provide the interface to the first model evaluation module.

[0077] Optionally, multiple open source models and closed source models are encapsulated into a unified interface, and the model parameters such as the character description, maximum response length, response diversity, memory rounds, knowledge base, etc. of each model are exposed to maximize the customization requirements of the model caller for the model call. Optionally, the encapsulated model unified interface can be further encapsulated into a model unified interface installation package to enhance the reusability of the model unified interface. Optionally, the encapsulated model unified interface installation package can be further converted into an AOP SDK (Software Development Kit) through AOP (Agent Oriented Programming).

[0078] The first model evaluation module 302 is used to obtain a preset game application for at least one evaluation dimension, where the evaluation dimension is used to characterize a preset evaluation indicator for evaluating the model to be evaluated, and the game application includes at least one non-player character and multiple player characters; the non-player character is controlled to execute a first game task in the application process of the game application through the model to be evaluated; in response to the completion of the first game task, a second game task is provided to each player character, and the second game task includes evaluating the non-player character under each evaluation dimension according to the execution of the first game task by the non-player character; the evaluation data of each player character on the non-player character under each evaluation dimension is collected, and based on the evaluation data, the evaluation result of the model to be evaluated under each evaluation dimension is calculated.

[0079] The first model evaluation module 302 is mainly used to evaluate the evaluation model based on game users (players) in the game application to obtain player evaluation results. This part of the content has been described in detail in the first embodiment. The specific content can be referred to the first embodiment of the present application, and will not be repeated here.

[0080] In an optional implementation, the first model evaluation module 302 is connected to a game application system, and the game application system is deployed with game applications for evaluating the model to be evaluated, such as: a player-proposed game, a prompt word supplement game, etc. The game application system is also deployed with support modules corresponding to the game application, such as: a riddle question bank, a model and non-player character binding module, a player character scoring module, an evaluation score synchronization module, etc., to support the operation of the game application and model evaluation.

[0081] The second model evaluation module 303 is used to connect the non-player character controlled by the model to be evaluated to the crowdsourcing platform, so that the users of the crowdsourcing platform can evaluate the non-player character under various evaluation dimensions according to the execution of the first game task by the non-player character; collect the evaluation data of the non-player character by the users of the crowdsourcing platform under various evaluation dimensions, and calculate the crowdsourcing evaluation results of the model to be evaluated under various evaluation dimensions based on the evaluation data; compare the evaluation results with the crowdsourcing evaluation results to provide credibility verification of the evaluation results.

[0082] The second model evaluation module 303 is mainly used to connect with the crowdsourcing platform to evaluate the evaluation model based on the crowdsourcing platform users (high-level players and / or professionals in model development and evaluation) in the crowdsourcing platform, obtain crowdsourcing evaluation results, and use the results as credibility verification of the player evaluation results.

[0083] The third embodiment of the present application provides a model evaluation device. Figure 4 Schematic diagram of the structure of the model evaluation device provided in this embodiment.

[0084] like Figure 4 As shown, the model evaluation device provided in this embodiment includes: a game application acquisition unit 401, a game task execution unit 402, a game task providing unit 403, and an evaluation result calculation unit 404.

[0085] The game application acquisition unit 401 is used to acquire a game application preset for at least one evaluation dimension, where the evaluation dimension is used to characterize a preset indicator for evaluating a model to be evaluated, and the game application includes at least one non-player character and multiple player characters.

[0086] The game task execution unit 402 is used to control the non-player character to execute a first game task in the application process of the game application through the model to be evaluated.

[0087] Optionally, controlling the non-player character to perform a first game task in the application process of the game application through the model to be evaluated includes:

[0088] In response to a start instruction for the game application, the model to be evaluated is connected to the game application, so that the model to be evaluated controls the non-player character to perform the first game task in the application process of the game application.

[0089] Optionally, the to-be-evaluated model includes multiple ones, and the game application includes multiple non-player characters;

[0090] The step of controlling the non-player character to perform a first game task in the application process of the game application by using the model to be evaluated includes:

[0091] A plurality of the models to be evaluated are connected to the game application, and corresponding relationships are established with a plurality of the non-player characters, so that each of the models to be evaluated controls one of the non-player characters to perform the first game task in the application process of the game application.

[0092] Optionally, the multiple models to be evaluated have a unified interface;

[0093] The step of connecting the plurality of models to be evaluated to the game application and establishing corresponding relationships with the plurality of non-player characters includes:

[0094] The game application is controlled to call a plurality of the models to be evaluated based on the unified interface, and one of the non-player characters is bound to each of the models to be evaluated.

[0095] The game task providing unit 403 is used to provide a second game task to each player character in response to the completion of the first game task. The second game task includes evaluating the non-player character under various evaluation dimensions according to the execution of the first game task by the non-player character.

[0096] The evaluation result calculation unit 404 is used to collect evaluation data of each player character on the non-player character in each evaluation dimension, and calculate the evaluation result of the model to be evaluated in each evaluation dimension according to the evaluation data.

[0097] Optionally, collecting evaluation data of each player character on the non-player character in each evaluation dimension, and calculating the evaluation result of the model to be evaluated in each evaluation dimension according to the evaluation data, includes:

[0098] For each of the models to be evaluated, collecting evaluation data of each player character on the non-player character corresponding to the model to be evaluated under each evaluation dimension;

[0099] Calculate the evaluation results of each model to be evaluated under each evaluation dimension according to the evaluation data corresponding to each model to be evaluated;

[0100] According to the evaluation results of each of the models to be evaluated under each evaluation dimension, the multiple models to be evaluated are sorted to generate ranking results of the multiple models to be evaluated under each evaluation dimension.

[0101] Optionally, the device further comprises a crowdsourcing evaluation unit; the crowdsourcing evaluation unit is used to:

[0102] Connecting the non-player character controlled by the model to be evaluated to a crowdsourcing platform, so that users of the crowdsourcing platform can evaluate the non-player character under various evaluation dimensions according to the performance of the non-player character on the first game task;

[0103] Collecting evaluation data of the non-player characters from users of the crowdsourcing platform under various evaluation dimensions, and calculating crowdsourcing evaluation results of the model to be evaluated under various evaluation dimensions based on the evaluation data;

[0104] The crowdsourcing evaluation result is compared with the evaluation result to provide credibility verification of the evaluation result.

[0105] Optionally, the device further includes a model training unit; the model training unit is used to:

[0106] According to the evaluation data of each player character on the non-player character under each evaluation dimension, screening training samples, the training samples including the first game task and the execution result of the first game task by the non-player character whose evaluation data is higher than a preset score threshold;

[0107] The model to be trained is trained based on the training samples.

[0108] Optionally, the game application is a sub-application embedded in a main game application, and the main game application includes a massively multiplayer online role-playing game; the device further includes a reward issuing unit; and the reward issuing unit is used to:

[0109] In response to the completion of the execution of the second game task, rewards are issued to the player accounts corresponding to each player character, and the game application is exited and the main game application is entered so that the player account can use the rewards in the main game application.

[0110] Optionally, the first game task is to perform natural language interaction with the player character, and the second game task is to evaluate the non-player character under various evaluation dimensions according to the natural language interaction between the non-player character and the player character.

[0111] Optionally, the evaluation dimension includes at least one of the following dimensions:

[0112] Language interaction ability, consistency between language and role, consistency between language and scene, expressiveness of language content, personification of roles, and security of language content.

[0113] Optionally, the device further includes a data broadcast unit; the data broadcast unit is used to:

[0114] Broadcast the first game task, the execution status of the first game task by the non-player character, and the evaluation data of the non-player character by the player character under each evaluation dimension.

[0115] A fourth embodiment of the present application provides an electronic device, Figure 5 It is a schematic diagram of the structure of the electronic device provided in this embodiment.

[0116] like Figure 5 As shown, the electronic device provided in this embodiment includes: a memory 501 and a processor 502;

[0117] The memory 501 is used to store computer instructions for executing the model evaluation method;

[0118] The processor 502 is configured to execute computer instructions stored in the memory 501 to perform the following operations:

[0119] Acquire a game application preset for at least one evaluation dimension, where the evaluation dimension is used to characterize a preset indicator for evaluating a model to be evaluated, and the game application includes at least one non-player character and a plurality of player characters;

[0120] Controlling the non-player character to execute a first game task in the application process of the game application through the model to be evaluated;

[0121] In response to the first game task being completed, providing each player character with a second game task, wherein the second game task includes evaluating the non-player character under various evaluation dimensions according to the performance of the first game task by the non-player character;

[0122] The evaluation data of each player character on the non-player character under each evaluation dimension is collected, and based on the evaluation data, the evaluation result of the model to be evaluated under each evaluation dimension is calculated.

[0123] Optionally, controlling the non-player character to perform a first game task in the application process of the game application through the model to be evaluated includes:

[0124] In response to a start instruction for the game application, the model to be evaluated is connected to the game application, so that the model to be evaluated controls the non-player character to perform the first game task in the application process of the game application.

[0125] Optionally, the to-be-evaluated model includes multiple ones, and the game application includes multiple non-player characters;

[0126] The step of controlling the non-player character to perform a first game task in the application process of the game application by using the model to be evaluated includes:

[0127] A plurality of the models to be evaluated are connected to the game application, and corresponding relationships are established with a plurality of the non-player characters, so that each of the models to be evaluated controls one of the non-player characters to perform the first game task in the application process of the game application.

[0128] Optionally, collecting evaluation data of each player character on the non-player character in each evaluation dimension, and calculating the evaluation result of the model to be evaluated in each evaluation dimension according to the evaluation data, includes:

[0129] For each of the models to be evaluated, collecting evaluation data of each player character on the non-player character corresponding to the model to be evaluated under each evaluation dimension;

[0130] Calculate the evaluation results of each model to be evaluated under each evaluation dimension according to the evaluation data corresponding to each model to be evaluated;

[0131] According to the evaluation results of each of the models to be evaluated under each evaluation dimension, the multiple models to be evaluated are sorted to generate ranking results of the multiple models to be evaluated under each evaluation dimension.

[0132] Optionally, also execute:

[0133] Connecting the non-player character controlled by the model to be evaluated to a crowdsourcing platform, so that users of the crowdsourcing platform can evaluate the non-player character under various evaluation dimensions according to the performance of the non-player character on the first game task;

[0134] Collecting evaluation data of the non-player characters from users of the crowdsourcing platform under various evaluation dimensions, and calculating crowdsourcing evaluation results of the model to be evaluated under various evaluation dimensions based on the evaluation data;

[0135] The crowdsourcing evaluation result is compared with the evaluation result to provide credibility verification of the evaluation result.

[0136] Optionally, the multiple models to be evaluated have a unified interface;

[0137] The step of connecting the plurality of models to be evaluated to the game application and establishing corresponding relationships with the plurality of non-player characters includes:

[0138] The game application is controlled to call a plurality of the models to be evaluated based on the unified interface, and one of the non-player characters is bound to each of the models to be evaluated.

[0139] Optionally, also execute:

[0140] According to the evaluation data of each player character on the non-player character under each evaluation dimension, screening training samples, the training samples including the first game task and the execution result of the first game task by the non-player character whose evaluation data is higher than a preset score threshold;

[0141] The model to be trained is trained based on the training samples.

[0142] Optionally, the game application is a sub-application embedded in a main game application, and the main game application includes a massively multiplayer online role-playing game; and further executing:

[0143] In response to the completion of the execution of the second game task, rewards are issued to the player accounts corresponding to each player character, and the game application is exited and the main game application is entered so that the player account can use the rewards in the main game application.

[0144] Optionally, the first game task is to perform natural language interaction with the player character, and the second game task is to evaluate the non-player character under various evaluation dimensions according to the natural language interaction between the non-player character and the player character.

[0145] Optionally, the evaluation dimension includes at least one of the following dimensions:

[0146] Language interaction ability, consistency between language and role, consistency between language and scene, expressiveness of language content, personification of roles, and security of language content.

[0147] Optionally, also execute:

[0148] Broadcast the first game task, the execution status of the first game task by the non-player character, and the evaluation data of the non-player character by the player character under each evaluation dimension.

[0149] The fifth embodiment of the present application provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed by a processor, they are used to implement the methods described in the embodiments of the present application.

[0150] It should be noted that the relational terms such as "first" and "second" in this article are only used to distinguish one entity or operation from another entity or operation, and do not require or imply any actual relationship or order between these entities or operations. In addition, the words "include", "have", "include" and "includes" and other similar forms are the same in meaning, and the end of any one or more items after any of the above words is open-ended, and any of the above nouns does not mean that the one or more items have been exhaustively listed or are limited to the one or more items listed.

[0151] As used herein, unless expressly stated otherwise, the term "or" includes all possible combinations, except those that are not feasible. For example, if it is expressed that a database may include A or B, then unless otherwise specifically stated or not feasible, it may include database A, or B, or A and B. As a second example, if it is expressed that a database may include A, B, or C, then unless otherwise specifically stated or not feasible, the database may include database A, or B, or C, or A and B, or A and C, or B and C, or A, B, and C.

[0152] It is worth noting that the above embodiments can be implemented by hardware or software (program code), or a combination of hardware and software. If implemented by software, it can be stored in the above-mentioned computer-readable medium. When the software is executed by a processor, it can execute the above-mentioned disclosed method. The computing unit and other functional units described in the present disclosure can be implemented by hardware or software, or a combination of hardware and software. Those of ordinary skill in the art will also understand that the above-mentioned multiple modules / units can be combined into one module / unit, and each of the above-mentioned modules / units can be further divided into multiple sub-modules / sub-units.

[0153] In the above detailed description, the embodiments have been described with reference to many specific details, which may vary from implementation to implementation. Certain adaptations and modifications may be made to the embodiments. For those skilled in the art, other embodiments may be readily apparent from the specific embodiments disclosed in the present application. This specification and examples are for exemplary purposes only, and the true scope and essence of the present application are described by the claims. The sequence of steps shown in the diagrams is also for the purpose of explanation only and is not intended to be limited to any particular steps or sequence. Therefore, those skilled in the art will appreciate that these steps may be performed in different orders when implementing the same method.

[0154] In the drawings and detailed description of the present application, exemplary embodiments are disclosed. However, many variations and modifications may be made to these embodiments. Accordingly, although specific terms are used, these terms are only general and descriptive and not for the purpose of limitation.

Claims

1. A model evaluation method, characterized in that: The method comprises: Acquire a game application preset for at least one evaluation dimension, where the evaluation dimension is used to characterize a preset indicator for evaluating a model to be evaluated, and the game application includes at least one non-player character and a plurality of player characters; Controlling the non-player character to execute a first game task in the application process of the game application through the model to be evaluated; In response to the first game task being completed, providing each player character with a second game task, wherein the second game task includes evaluating the non-player character under various evaluation dimensions according to the performance of the first game task by the non-player character; The evaluation data of each player character on the non-player character under each evaluation dimension is collected, and based on the evaluation data, the evaluation result of the model to be evaluated under each evaluation dimension is calculated.

2. The method according to claim 1, characterized in that The models to be evaluated include multiple ones, and the game application includes multiple non-player characters; The step of controlling the non-player character to perform a first game task in the application process of the game application by using the model to be evaluated includes: A plurality of the models to be evaluated are connected to the game application, and corresponding relationships are established with a plurality of the non-player characters, so that each of the models to be evaluated controls one of the non-player characters to perform the first game task in the application process of the game application.

3. The method according to claim 2, characterized in that The collecting of evaluation data of each player character on the non-player character in each evaluation dimension, and calculating the evaluation result of the model to be evaluated in each evaluation dimension according to the evaluation data, includes: For each of the models to be evaluated, collecting evaluation data of each player character on the non-player character corresponding to the model to be evaluated under each evaluation dimension; Calculate the evaluation results of each model to be evaluated under each evaluation dimension according to the evaluation data corresponding to each model to be evaluated; According to the evaluation results of each of the models to be evaluated under each evaluation dimension, the multiple models to be evaluated are sorted to generate ranking results of the multiple models to be evaluated under each evaluation dimension.

4. The method according to claim 1, characterized in that: The method further comprises: Connecting the non-player character controlled by the model to be evaluated to a crowdsourcing platform, so that users of the crowdsourcing platform can evaluate the non-player character under various evaluation dimensions according to the performance of the non-player character on the first game task; Collecting evaluation data of the non-player characters from users of the crowdsourcing platform under various evaluation dimensions, and calculating crowdsourcing evaluation results of the model to be evaluated under various evaluation dimensions based on the evaluation data; The crowdsourcing evaluation result is compared with the evaluation result to provide credibility verification of the evaluation result.

5. The method according to claim 2, characterized in that: The multiple models to be evaluated have a unified interface; The step of connecting the plurality of models to be evaluated to the game application and establishing corresponding relationships with the plurality of non-player characters includes: The game application is controlled to call a plurality of the models to be evaluated based on the unified interface, and one of the non-player characters is bound to each of the models to be evaluated.

6. The method according to claim 1, characterized in that The method further comprises: According to the evaluation data of each player character on the non-player character under each evaluation dimension, screening training samples, the training samples including the first game task and the execution result of the first game task by the non-player character whose evaluation data is higher than a preset score threshold; The model to be trained is trained based on the training samples.

7. The method according to claim 1, characterized in that The game application is a sub-application embedded in a main game application, and the main game application includes a massively multiplayer online role-playing game; the method further includes: In response to the completion of the execution of the second game task, rewards are issued to the player accounts corresponding to each player character, and the game application is exited and the main game application is entered so that the player account can use the rewards in the main game application.

8. The method according to claim 1, characterized in that The first game task is to perform natural language interaction with the player character, and the second game task is to evaluate the non-player character under various evaluation dimensions according to the natural language interaction between the non-player character and the player character.

9. The method according to claim 8, characterized in that The evaluation dimension includes at least one of the following dimensions: Language interaction ability, consistency between language and role, consistency between language and scene, expressiveness of language content, personification of roles, and security of language content.

10. The method according to claim 1, characterized in that The method further comprises: Broadcast the first game task, the execution status of the first game task by the non-player character, and the evaluation data of the non-player character by the player character under each evaluation dimension.

11. The method according to claim 1, characterized in that: The step of controlling the non-player character to perform a first game task in the application process of the game application by using the model to be evaluated includes: In response to a start instruction for the game application, the model to be evaluated is connected to the game application, so that the model to be evaluated controls the non-player character to perform the first game task in the application process of the game application.

12. A model evaluation system, characterized in that: The system comprises: a model supply module, a first model evaluation module, and a second model evaluation module; The model supply module is used to encapsulate multiple models to be evaluated into a unified interface and provide the interface to the first model evaluation module; The first model evaluation module is used to obtain a preset game application for at least one evaluation dimension, wherein the evaluation dimension is used to characterize a preset evaluation index for evaluating the model to be evaluated, wherein the game application includes at least one non-player character and multiple player characters; the non-player character is controlled to perform a first game task in the application process of the game application through the model to be evaluated; in response to the completion of the execution of the first game task, a second game task is provided to each player character, wherein the second game task includes evaluating the non-player character under each evaluation dimension according to the execution of the first game task by the non-player character; the evaluation data of each player character on the non-player character under each evaluation dimension is collected, and the evaluation result of the model to be evaluated under each evaluation dimension is calculated according to the evaluation data; The second model evaluation module is used to connect the non-player character controlled by the model to be evaluated to the crowdsourcing platform, so that users of the crowdsourcing platform can evaluate the non-player character under various evaluation dimensions according to the execution of the first game task by the non-player character; collect evaluation data of the non-player character by users of the crowdsourcing platform under various evaluation dimensions, and calculate crowdsourcing evaluation results of the model to be evaluated under various evaluation dimensions based on the evaluation data; and compare the evaluation results with the crowdsourcing evaluation results to provide credibility verification of the evaluation results.

13. A model evaluation device, characterized in that: The device comprises: a game application acquisition unit, a game task execution unit, a game task providing unit, and an evaluation result calculation unit; The game application acquisition unit is used to acquire a game application preset for at least one evaluation dimension, where the evaluation dimension is used to characterize a preset indicator for evaluating the evaluation model, and the game application includes at least one non-player character and multiple player characters; The game task execution unit is used to control the non-player character to execute a first game task in the application process of the game application through the model to be evaluated; The game task providing unit is configured to provide a second game task to each player character in response to the completion of the execution of the first game task, wherein the second game task includes evaluating the non-player character under various evaluation dimensions according to the execution of the first game task by the non-player character; The evaluation result calculation unit is used to collect evaluation data of each player character on the non-player character in each evaluation dimension, and calculate the evaluation result of the model to be evaluated in each evaluation dimension according to the evaluation data.

14. An electronic device, characterized in that: include: Memory, processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to implement the method according to any one of claims 1-11.

15. A computer-readable storage medium having one or more computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the method according to any one of claims 1 to 11 is performed.

Images