Model evaluation methods, equipment, media and products

Abstract

This application discloses a model evaluation method, apparatus, medium, and product, belonging to the field of computer science. The model evaluation method of this application identifies initial data to obtain first candidate data and the corresponding data type; filters the first candidate data based on the corresponding data type; inputs the filtered data into at least two target models to obtain second candidate data output by each target model; and evaluates the corresponding target models based on the second candidate data output by each target model to generate evaluation results. This application improves the accuracy of data selection and reduces labor costs by identifying initial data and filtering first candidate data; inputting the filtered data into multiple target models and evaluating the corresponding target models based on the second candidate data output by each target model enables a more comprehensive evaluation of the overall capabilities of the target models, making the evaluation data more realistic.

Description

Technical Field

[0001] This application relates to the field of computers, and in particular to model evaluation methods, devices, media, and products. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have become an important research direction in the field of natural language processing. These models have demonstrated powerful capabilities in various application scenarios, such as text generation, question answering systems, and content summarization. However, how to effectively evaluate the performance of these models, especially their performance in practical applications, has become an urgent problem to be solved.

[0003] Currently, model evaluation methods mainly rely on domain-specific test sets and human evaluation. For example, some researchers have proposed obtaining the original dataset, filtering the model based on the data for multiple rounds of training, and determining the target data based on the negative examples in the test results, thereby selecting a representative training dataset. While this method can obtain a representative data sample set for training complex network models, it primarily focuses on data filtering rather than model evaluation itself.

[0004] However, existing large language model evaluation methods suffer from the following problems: First, most existing evaluation methods focus on specialized fields, judging whether the model meets output expectations by setting exam questions, which often differs from the subjective experience (feeling) of humans using the model in daily life; second, the professional ability to solve problems cannot be intuitively translated into the ability to help humans solve problems; finally, existing solutions mainly rely on manual human evaluation, which is not only time-consuming and labor-intensive but also difficult to scale. These problems make it difficult for existing model evaluation methods to comprehensively and objectively reflect the model's performance in practical applications, and thus fail to provide effective guidance for model optimization.

[0005] Therefore, there is an urgent need for a model evaluation method that can comprehensively consider multiple factors, has a high degree of automation, and can objectively reflect the actual performance of the model in practical applications, so as to improve the efficiency and accuracy of model evaluation. Summary of the Invention

[0006] To address the technical challenges of existing large language model evaluation methods, such as their focus on specific professional fields, inconsistency with human daily use experience, and difficulty in large-scale promotion, and to achieve comprehensive, accurate, automated, and robust technical effects, this application provides an automated model evaluation method, device, medium, and product that is consistent with human experience.

[0007] This application provides a model evaluation method, including:

[0008] Get initial data;

[0009] The initial data is identified to obtain first candidate data and the data type corresponding to the first candidate data;

[0010] The first candidate data is filtered based on the data type corresponding to the first candidate data, and the filtered data is input into at least two target models to obtain the second candidate data output by each of the target models.

[0011] The corresponding target models are evaluated based on the second candidate data output by each target model, and evaluation results are generated.

[0012] Optionally, the step of identifying the initial data and obtaining the first candidate data and the data type corresponding to the first candidate data includes:

[0013] The initial data is classified based on the first model, and data belonging to the target type is identified and extracted from the initial data as the first candidate data.

[0014] The data type of the first candidate data is the target type.

[0015] Optionally, the first model may employ a large language model.

[0016] Optionally, the step of filtering the first candidate data based on the data type corresponding to the first candidate data, inputting the filtered data into at least two target models, and obtaining the second candidate data output by each of the target models includes:

[0017] Based on the data type corresponding to the first candidate data, the first candidate data is filtered to determine data belonging to the data type as the third candidate data;

[0018] The third candidate data is input into at least two target models to obtain the second candidate data output by each target model.

[0019] Optionally, the step of filtering the first candidate data based on the data type corresponding to the first candidate data to determine data belonging to the data type as the third candidate data includes:

[0020] The first candidate data is identified using a second model based on the first preset rules and the data type to obtain first identified data, which includes first analysis data and first result data.

[0021] Based on the first analysis data and the first result data, data belonging to the data type is determined from the first candidate data as the third candidate data.

[0022] Optionally, the step of filtering the first candidate data based on the data type corresponding to the first candidate data, inputting the filtered data into at least two target models, and obtaining the second candidate data output by each of the target models includes:

[0023] Based on the data type corresponding to the first candidate data, the first candidate data is filtered to determine the data belonging to the data type as the fourth candidate data;

[0024] The fourth candidate data is input into at least two prediction models for prediction, and the prediction results of each prediction model are obtained.

[0025] Calculate the similarity between each of the prediction results, and based on all the similarities, select the data in the fourth candidate data that meets the preset conditions as the fifth candidate data;

[0026] The fifth candidate data is input into at least two target models to obtain the second candidate data output by each target model.

[0027] Optionally, the step of evaluating the corresponding target model based on the second candidate data output by each target model and generating evaluation results includes:

[0028] A third model is used to evaluate each of the target models based on the second preset rules, all the second candidate data, and the first candidate data corresponding to the second candidate data, and to generate the evaluation results corresponding to each of the target models.

[0029] This application provides an electronic device comprising: one or more processors; and a memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the model evaluation method described above.

[0030] This application provides a computer-readable medium having computer program instructions stored thereon, which can be executed by a processor to implement the model evaluation method described above.

[0031] This application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the model evaluation method described above.

[0032] The beneficial effects of the above technical solution are as follows: The model evaluation method of this application identifies the initial data to obtain first candidate data and the data type corresponding to the first candidate data; filters the first candidate data based on the data type corresponding to the first candidate data; inputs the filtered data into at least two target models to obtain second candidate data output by each target model; and evaluates the corresponding target models based on the second candidate data output by each target model to generate evaluation results. This application improves the accuracy of data selection and reduces labor costs by identifying the initial data and filtering the first candidate data; inputting the filtered data into multiple target models and evaluating the corresponding target models based on the second candidate data output by each target model enables a more comprehensive evaluation of the overall capabilities of the target models, making the evaluation data more realistic and significantly improving evaluation efficiency and accuracy. Attached Figure Description

[0033] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0034] Figure 1 This is a flowchart illustrating one embodiment of the model evaluation method described in this application.

[0035] Figure 2 A flowchart illustrating an embodiment of this application of filtering first candidate data and inputting the filtered data into a target model;

[0036] Figure 3 A flowchart illustrating another embodiment of the method for filtering first candidate data and inputting the filtered data into the target model in accordance with this application;

[0037] Figure 4 This is an exemplary structural diagram of the electronic device of this application. Detailed Implementation

[0038] The advantages of this application are further illustrated below with reference to the accompanying drawings and specific embodiments.

[0039] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0040] The terminology used in this disclosure is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The singular forms “a,” “the,” and “the” as used in this disclosure and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0041] It should be understood that although the terms first, second, third, etc., may be used in this disclosure to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this disclosure, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0042] In the description of this application, it should be understood that the numerical labels before the steps do not indicate the order of the steps, but are only used to facilitate the description of this application and to distinguish each step, and therefore should not be construed as a limitation of this application.

[0043] In practical applications, the model evaluation method of this application can be applied to scenarios such as human-computer question answering and casual conversation. For example, this model evaluation method can be directly applied to products such as intelligent question answering platforms and personalized casual conversation companion robots. Specifically, the method first collects a full corpus of user-robot interactions in real-world scenarios as initial data. Then, it identifies and extracts core target data types (i.e., first-line candidate data) such as user inquiries, emotional expressions, and everyday casual conversation. After filtering this data, it simultaneously inputs two or more question-answering / casual conversation models for comparison, obtaining the responses from each model. Finally, based on key indicators such as the matching degree between the responses and user needs, the fluency and naturalness of the language, and the emotional resonance, the performance of different models is evaluated and scored, outputting clear evaluation results. This process efficiently supports the product side in selecting the optimal model or specifically optimizing model weaknesses to improve the user interaction experience.

[0044] See Figure 1 The model evaluation method of this embodiment includes the following steps:

[0045] S1. Obtain initial data;

[0046] In this embodiment, the initial data can be text data, image data, audio data, etc.; the initial data belongs to the category of question data, which can include eight target types: mathematical calculation (mainly in Chinese and English), logical reasoning, comprehensive question answering, coding, role-playing, multilingual translation, creative writing, and natural language understanding. The initial data is used for subsequent model evaluation. These datasets may contain a large amount of information, covering different topics and content, to ensure the comprehensiveness and accuracy of the model evaluation. The initial data can be several thousand data points that have been manually categorized.

[0047] S2. Identify the initial data to obtain first candidate data and the data type corresponding to the first candidate data;

[0048] Further, step S2 may include: classifying the initial data based on the first model, and identifying and extracting data belonging to the target type from the initial data as the first candidate data;

[0049] Wherein, the data type of the first candidate data is the target type.

[0050] It should be noted that the first model adopts a large language model.

[0051] In a preferred embodiment, the first model can be an advanced Large Language Model (LLM) such as the Qwen2.5-14B-Instruct model or the GPT-4 model, capable of understanding complex text content and performing accurate classification. The Qwen2.5-14B-Instruct model excels in Chinese language capabilities, being specifically optimized for Chinese corpora and demonstrating outstanding performance in Chinese understanding, generation, and reasoning. The Qwen2.5-14B-Instruct model is open-source and free, with open-source model weights, making it suitable for local deployment, academic research, or enterprise secondary development. With fine-tuning of instructions, it is suitable for interactive scenarios such as dialogue, question answering, and task execution. This model is lightweight and efficient, with a moderate 14B parameter scale, resulting in low deployment costs and suitability for small to medium-scale applications. The GPT-4 model excels in multilingual tasks, including English and Chinese; it possesses strong logical reasoning capabilities, demonstrating exceptional performance in complex reasoning, mathematical, and code generation tasks; it supports image input (requires a subscription to GPT-4 Turbo or a higher version); and it boasts a robust plugin system, API ecosystem, and commercial support.

[0052] The last layer of the first model is a classification layer with an output dimension of 8. The first model is used to perform preliminary screening of massive initial data and select the first candidate data related to the target type. Step S1 aims to reduce the amount of initial data and improve the efficiency of subsequent processing.

[0053] S3. Filter the first candidate data based on the data type corresponding to the first candidate data, input the filtered data into at least two target models, and obtain the second candidate data output by each of the target models;

[0054] Step S3 can be achieved in two different ways:

[0055] The first method of reference Figure 2 Step S3 may include the following steps:

[0056] S311. Based on the data type corresponding to the first candidate data, filter the first candidate data to determine data belonging to the data type as the third candidate data;

[0057] Specifically, step S311 may include:

[0058] The second model is used to identify the first candidate data based on the first preset rule and the data type to obtain the first identification data, which includes the first analysis data and the first result data. Based on the first analysis data and the first result data, data belonging to the data type is determined from the first candidate data as the third candidate data.

[0059] In this embodiment, the first result data indicates whether the first candidate data belongs to the corresponding data type; for example, it belongs to / does not belong to, or / not, etc.; the first analysis data indicates the analysis reason for whether the first candidate data belongs to the corresponding data type.

[0060] In this embodiment, the first candidate number is further identified to determine whether the obtained third candidate data truly belongs to the target type, and to detect whether there are any misjudgments. Step S311 improves the accuracy and reliability of the data.

[0061] It should be noted that the second model can adopt a large language model (LLM) such as the Qwen2.5-14B-Instruct model or the GPT-4 model. The first preset rules may include evaluation criteria across multiple dimensions such as data integrity, data quality, and data relevance, as well as a first prompt word template. This first prompt word template is used as input to the second model so that the second model can generate the corresponding response by understanding the first prompt word template.

[0062] The first prompt template may include three parts: character setting, task breakdown, and output format requirements.

[0063] As an example, and not a limitation, the template for the first prompt word is as follows:

[0064] [Character Setting]

[0065] You are a classification consistency analysis assistant, focused on determining whether the "topic" and "classification result" conform to the classification rules, and outputting structured conclusions including "analysis reasons" and "final results".

[0066] [Task Breakdown]

[0067] 1. Define the classification rules: First, confirm the category system (it must belong to one of the following eight categories: mathematical calculation, logical reasoning, comprehensive question and answer, code, role-playing, multilingual translation, creative writing, and natural language understanding).

[0068] 2. Analyze question characteristics: Extract core information of the question (keywords, semantic tendency, domain attributes, etc.) and match the judgment criteria corresponding to the classification rules.

[0069] 3. Compare the classification results: Compare the classification results with the **expected category** derived from the "question features + classification rules".

[0070] Output format requirements

[0071] Returns a JSON format containing two fields:

[0072] - "Analyze the reasons": Explain in words "why they are consistent / inconsistent" (this needs to be combined with the classification rules to break down the matching / mismatch points between the question features and the classification results).

[0073] - "Final result": string, outputs only "yes" (consistent) or "no" (inconsistent).

[0074] Using the first candidate as: Input: Title = "Let's play a role-playing game, you be the doctor and I'll be the patient, let's begin the consultation", Classification result = "Role-playing"

[0075] - Output:

[0076] json

[0077] {

[0078] Analysis of the Reasons: The phrase "Let's play a role-playing game" in the question signals the start of the role-playing activity; "You be the doctor, I'll be the patient" clarifies the role assignment; and "Let's begin the consultation" constructs the role-playing scenario. This perfectly matches the core criteria of "role-playing" (initiating role assignment + clarifying the role-playing scenario), therefore the classification result is consistent with the question.

[0079] Final result: Yes

[0080] }

[0081] S312. Input the third candidate data into at least two target models to obtain the second candidate data output by each of the target models.

[0082] In step S312, the filtered third candidate data are input into multiple target models for prediction to obtain the corresponding second candidate data.

[0083] In this embodiment, the target model is the model to be evaluated. The target model also belongs to the category of large language models, and can be a Qwen2.5-14B-Instruct model or a GPT-4 model, etc.

[0084] The second method is to refer to Figure 3 Step S3 may include the following steps:

[0085] S321. Based on the data type corresponding to the first candidate data, filter the first candidate data to determine data belonging to the data type as the fourth candidate data;

[0086] Specifically, step S321 may include:

[0087] The first candidate data is identified using a fourth model based on a third preset rule and the data type to obtain second identification data, which includes second analysis data and second result data. Based on the second analysis data and the second result data, data belonging to the data type is determined from the first candidate data as the fourth candidate data.

[0088] The third preset rule may include evaluation criteria across multiple dimensions such as data integrity, data quality, and data relevance, as well as a first prompt word template. This first prompt word template is used as input to the fourth model so that the fourth model can generate a corresponding response by understanding the first prompt word template.

[0089] The fourth model also belongs to the category of large language models, and can be the Qwen2.5-14B-Instruct model or the GPT-4 model, etc.

[0090] In this embodiment, the second result data indicates whether the second candidate data belongs to the corresponding data type; for example, whether it belongs to / does not belong to, or whether it does not. The second analysis data indicates the analysis reason for whether the second candidate data belongs to the corresponding data type.

[0091] In this embodiment, the first candidate number is further identified to determine whether the obtained fourth candidate data truly belongs to the target type, and to detect whether there are any misjudgments. Step S321 improves the accuracy and reliability of the data.

[0092] S322. Input the fourth candidate data into at least two prediction models for prediction, and obtain the prediction results of each prediction model;

[0093] S323. Calculate the similarity between each of the prediction results, and based on all the similarities, select the data in the fourth candidate data that meets the preset conditions as the fifth candidate data;

[0094] In this embodiment, the preset condition is that the similarity is within a threshold range, such as 0-0.5. The higher the similarity, the more similar the prediction results. The similarity can be calculated using n-gram similarity. n-gram similarity is a classic text similarity calculation method in Natural Language Processing (NLP), which measures similarity by comparing the degree of overlap of consecutive character or word sequences (i.e., n-grams) in two text segments.

[0095] In steps S322 and S323, to ensure that the selected data is discriminative across different models, three or more large language models (i.e., prediction models) from different sources (e.g., Yuewen, Qianwen, and Doubao) can be used to generate answers (i.e., prediction results) for the same question (i.e., the fourth candidate data). The discriminative nature of the question is then determined based on the n-gram similarity of these answers. These steps ensure the diversity and effectiveness of the evaluation data (i.e., the fifth candidate data).

[0096] The following is a sample question without discrimination:

[0097] Question: Tell me what the next line after "The sun sets beyond the mountains" is?

[0098] Answer 1:

[0099] The next line after "The sun sets beyond the mountains" is "The Yellow River flows into the sea." These two lines are from "Ascending the Stork Tower" by the Tang Dynasty poet Wang Zhihuan. The full poem is:

[0100] The sun sets behind the mountains, and the Yellow River flows into the sea.

[0101] To see a thousand miles further, climb one more story.

[0102] This poem depicts the magnificent scenery the poet saw when he climbed to a high place and looked into the distance. The first two lines describe the natural scenery, while the last two lines contain profound philosophical ideas and express a positive attitude towards life.

[0103] Answer 2:

[0104] The next line after "The sun sets beyond the mountains" is "The Yellow River flows into the sea." These two lines are from "Ascending the Stork Tower" by the Tang Dynasty poet Wang Zhihuan. The full poem is as follows:

[0105] The sun sets behind the mountains, and the Yellow River flows into the sea.

[0106] To see a thousand miles further, climb one more story.

[0107] This poem depicts the magnificent scenery the author witnessed upon ascending the Stork Tower, while also expressing his philosophical reflections on life. The first two lines concisely sketch a grand and majestic scene, while the last two lines use metaphor to express the author's pursuit and yearning for a higher realm of life. The entire poem possesses profound imagery and concise language, making it a gem among Tang Dynasty five-character quatrains.

[0108] Answer 3:

[0109] The next line after "The sun sets beyond the mountains" is "The Yellow River flows into the sea." These two lines are from Wang Zhihuan's poem "Ascending the Stork Tower." The complete poem is:

[0110] The sun sets behind the mountains.

[0111] The Yellow River flows into the sea.

[0112] To see a thousand miles further,

[0113] To reach a higher level.

[0114] This poem depicts the magnificent view from a high vantage point, expressing the poet's broad-mindedness and ambition to pursue higher goals.

[0115] All three answers were correct and explained the author and the scene depicted in the poem. The models did not differentiate between the correctness of the answers, the aesthetics of the format, or their helpfulness. In other words, the predictions from the three models were similar, failing to meet the preset conditions, and therefore the corresponding question could not be considered as the fifth candidate data.

[0116] S324. Input the fifth candidate data into at least two target models to obtain the second candidate data output by each of the target models.

[0117] In step S324, the filtered fifth candidate data are input into multiple target models for prediction to obtain the corresponding second candidate data.

[0118] In this embodiment, the target model is the model to be evaluated. The target model also belongs to the category of large language models, and can be a Qwen2.5-14B-Instruct model or a GPT-4 model, etc.

[0119] S4. Evaluate the corresponding target model based on the second candidate data output by each target model, and generate evaluation results.

[0120] Further, step S4 may include: using a third model to evaluate each of the target models based on the second preset rules, all of the second candidate data and the first candidate data corresponding to the second candidate data, and generating the evaluation results corresponding to each of the target models.

[0121] In this embodiment, the third model can be a large language model (LLM) such as the Qwen2.5-14B-Instruct model or the GPT-4 model. The second preset rule may include multiple evaluation indicators such as accuracy, consistency, and robustness, as well as a second prompt word template. This second prompt word template is used as input to the third model so that the third model can generate the corresponding response by understanding the second prompt word template.

[0122] As an example, and not a limitation, the template for the second prompt word is as follows:

[0123] You will evaluate two responses to a given user question (i.e., second candidate data) and determine which response wins or if they are tied. The following is related content:

[0124] [BEGIN DATA]

[0125] ***

[0126] [User Inquiry]:{}

[0127] ***

[0128] [Response 1]: {}

[0129] ***

[0130] [Response 2]: {}

[0131] ***

[0132] [END DATA]

[0133] Here is a guide to evaluating and comparing two responses:

[0134] 1. Identify the key factors that differentiate the two responses.

[0135] 2. Summarize your comparison and evaluation by providing your final decision on which option wins or whether both are tied. Your final decision statement should begin with "Therefore, the final decision is response 1 / response 2". Ensure your decision is consistent with the comprehensive evaluation and comparison you provided.

[0136] In this embodiment, the third model analyzes the second candidate data (response) output by each target model based on the first candidate data (question / problem) corresponding to the second candidate data (response / answer / respondence) and the second preset rules, and then outputs evaluation results to determine which model is the best among the target models. The evaluation results may include indicators such as fluency, relevance, information content, logical coherence, factual consistency, and security. Fluency measures grammatical correctness and naturalness of language; relevance measures the degree of connection between the output and the input instruction; information content measures the information density and value of the content; factual consistency measures whether the output conforms to known facts; logical coherence measures the logical connection between viewpoints; and security measures the detection of bias, offensive content, etc.

[0137] In a preferred embodiment, the model evaluation method may further include:

[0138] S5. Use the evaluation results to supervise the fine-tuning of the third model, and use the third model to generate more evaluation result data. By iteratively augmenting the data and improving the capabilities of the third model, continuous improvement and optimization of the model can be achieved.

[0139] In this embodiment, the model evaluation method identifies initial data to obtain first candidate data and the corresponding data type; filters the first candidate data based on the data type; inputs the filtered data into at least two target models to obtain second candidate data output by each target model; and evaluates the corresponding target models based on the second candidate data output by each target model to generate evaluation results. This application improves the accuracy of data selection and reduces labor costs by identifying initial data and filtering first candidate data. Inputting the filtered data into multiple target models and evaluating the corresponding target models based on the second candidate data output by each target model allows for a more comprehensive evaluation of the target models' overall capabilities, making the evaluation data more realistic and significantly improving evaluation efficiency and accuracy.

[0140] The model evaluation method in this embodiment comprehensively considers the model's performance in actual use, rather than just focusing on specific ability tests. Through a multi-dimensional, multi-scenario evaluation process, it can more comprehensively assess the overall capabilities of a large language model, including its performance in practical applications and its degree of satisfaction with user needs. By improving the accuracy of evaluation data selection and the realism of data distribution, and by constructing an accurate method for judging haptic consistency, the consistency between evaluation results and human haptic perception can be improved. This means that the evaluation results better reflect the model's performance in actual use, thus providing a more accurate basis for model improvement and optimization. The automation of data selection and third-party model training significantly reduces labor costs and improves evaluation efficiency and scalability. This makes large-scale, high-frequency evaluation possible, thus supporting rapid model iteration and optimization. Through iterative training and data augmentation, the model's generalization ability and robustness are maintained, preventing overfitting. This means that the model can maintain good performance and accuracy when facing new and unknown data, thereby improving its reliability and stability in practical applications.

[0141] Examples of application scenarios:

[0142] The third model was used to score the responses of Model A and Model B, and to score the responses of Model C and Model B:

[0143] Model A Model B 54

[0144] Model C Model B 65

[0145] We can conclude that Model C has a better user experience than Model B, and Model A has a slightly better user experience than Model B. Assuming Model B is the model from the previous release, this quantitative indicator of user experience can help provide a basis for selection in the next release.

[0146] This application aims to improve the efficiency and accuracy of data screening, ensuring the quality and diversity of evaluation data. Methods to improve the accuracy of screening models include selecting widely distributed data content, reducing contextual interference, and ensuring that the data is discriminatory against different models. This invention aims to improve the performance and reliability of screening models, thereby enhancing the accuracy and effectiveness of evaluation data.

[0147] The training process for the third model includes defining the scoring domain and rules, generating training data, checking the validity of the data, and iteratively training the model. The aim of this process is to build an accurate and reliable third model, improving the accuracy and consistency of the evaluation results.

[0148] Generating and expanding high-quality training data maintains the model's generalization ability and robustness, preventing overfitting. This invention aims to provide high-quality training data for third-party models, improving their performance and reliability.

[0149] This application combines automated data screening processes with third-party model training optimization to create a multi-dimensional, multi-scenario evaluation process that closely reflects real-world applications. It enables the construction of a comprehensive, accurate, and automated evaluation system, improving the efficiency and reliability of large language model evaluation. This application aims to address the weakness of existing automated evaluation benchmarks in assessing comprehensive sensory abilities by constructing robust data screening methods and models to improve the accuracy of evaluation data screening and the realism of data distribution. Simultaneously, it constructs an accurate sensory consistency judgment method to enhance the consistency between automated and human evaluations, achieving a scalable automated evaluation system. This will provide more accurate and reliable evaluation tools and methods for the development and application of large language models, promoting their widespread application and continuous improvement across various fields.

[0150] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0151] Furthermore, some embodiments of this application also provide an electronic device. The electronic device can be various forms of digital computer, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, etc. The electronic device can also be various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices.

[0152] The electronic device includes: one or more processors; and a memory storing computer program instructions that, when executed, cause the processor to perform the steps of the methods provided in any one or more of the above embodiments. Figure 4 An exemplary structural diagram of the electronic device is disclosed. For example... Figure 4As shown, the electronic device includes one or more processors 1101, a memory 1102, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). The components, their connections and relationships, and their functions shown herein are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0153] The electronic device may further include an input device 1103 and an output device 1104. The processor 1101, memory 1102, input device 1103, and output device 1104 may be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0154] Input device 1103 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the electronic device, such as a touch screen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 1104 may include a display device, auxiliary lighting device (e.g., LED), and haptic feedback device (e.g., vibration motor). The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.

[0155] To provide interaction with the user, the electronic device can be a computer. The computer has: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0156] In this embodiment, a computer-readable medium stores a computer program / instructions that, when executed by a processor, implement the steps of the methods provided in any one or more of the above embodiments. This computer-readable medium may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into that device. The aforementioned computer-readable medium carries one or more computer-readable instructions.

[0157] The memory 1102 can serve as a non-transitory computer-readable storage medium, used to store non-transitory software programs, non-transitory computer-executable programs, and modules. The processor 1101 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 1102, thereby implementing the program instructions / modules corresponding to the methods provided in any one or more of the embodiments described above in this application.

[0158] The memory 1102 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 1102 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 1102 may optionally include memory remotely located relative to the processor 1101, and these remote memories can be connected to the electronic device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0159] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0160] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0161] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0162] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. For example, it can be implemented using an application-specific integrated circuit (ASIC), a general-purpose computer, or any other similar hardware device. In some embodiments, the software program of this application can be executed by a processor to implement the steps or functions described above. Similarly, the software program of this application (including related data structures) can be stored in a computer-readable recording medium, such as RAM memory, magnetic or optical drives, floppy disks, or similar devices. Furthermore, some steps or functions of this application can be implemented in hardware, for example, as circuitry that works with a processor to perform the various steps or functions.

[0163] The computer program product provided in this application includes one or more computer programs / instructions. When executed by a processor, these computer programs / instructions generate, in whole or in part, the processes or functions described in this application. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0164] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0165] The scope of this application is defined by the appended claims rather than the foregoing description, and is therefore intended to encompass all variations falling within the meaning and scope of equivalents of the claims. No reference numerals in the claims should be construed as limiting the scope of the claims. Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device in software or hardware. Terms such as "first," "second," etc., are used only for distinguishing descriptions and do not indicate any particular order, nor should they be construed as indicating or implying relative importance.

[0166] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A model evaluation method, characterized in that, include: Get initial data; The initial data is identified to obtain first candidate data and the data type corresponding to the first candidate data; The first candidate data is filtered based on the data type corresponding to the first candidate data, and the filtered data is input into at least two target models to obtain the second candidate data output by each of the target models. The corresponding target models are evaluated based on the second candidate data output by each target model, and evaluation results are generated.

2. The model evaluation method according to claim 1, characterized in that, The step of identifying the initial data and obtaining first candidate data and the data type corresponding to the first candidate data includes: The initial data is classified based on the first model, and data belonging to the target type is identified and extracted from the initial data as the first candidate data. The data type of the first candidate data is the target type.

3. The model evaluation method according to claim 2, characterized in that, The first model uses a large language model.

4. The model evaluation method according to claim 1, characterized in that, The step of filtering the first candidate data based on the data type corresponding to the first candidate data, inputting the filtered data into at least two target models, and obtaining the second candidate data output by each of the target models includes: Based on the data type corresponding to the first candidate data, the first candidate data is filtered to determine the data belonging to the data type as the third candidate data; The third candidate data is input into at least two target models to obtain the second candidate data output by each target model.

5. The model evaluation method according to claim 4, characterized in that, The step of filtering the first candidate data based on the data type corresponding to the first candidate data, and determining data belonging to the data type as the third candidate data, includes: The first candidate data is identified using a second model based on the first preset rules and the data type to obtain first identified data, which includes first analysis data and first result data. Based on the first analysis data and the first result data, data belonging to the data type is determined from the first candidate data as the third candidate data.

6. The model evaluation method according to claim 1, characterized in that, The step of filtering the first candidate data based on the data type corresponding to the first candidate data, inputting the filtered data into at least two target models, and obtaining the second candidate data output by each of the target models includes: Based on the data type corresponding to the first candidate data, the first candidate data is filtered to determine the data belonging to the data type as the fourth candidate data; The fourth candidate data is input into at least two prediction models for prediction, and the prediction results of each prediction model are obtained. Calculate the similarity between each prediction result, and based on all the similarities, select the data in the fourth candidate data that meets the preset conditions as the fifth candidate data; The fifth candidate data is input into at least two target models to obtain the second candidate data output by each target model.

7. The model evaluation method according to claim 1, characterized in that, The evaluation of the corresponding target models based on the second candidate data output by each target model, and the generation of evaluation results, includes: A third model is used to evaluate each of the target models based on the second preset rules, all the second candidate data, and the first candidate data corresponding to the second candidate data, and to generate the evaluation results corresponding to each of the target models.

8. An electronic device, characterized in that, The electronic device includes: One or more processors; and A memory storing computer program instructions, which, when executed, cause the processor to perform the steps of the model evaluation method as described in any one of claims 1 to 7.

9. A computer-readable medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the model evaluation method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the model evaluation method according to any one of claims 1 to 7.

Images