A significant transition sample-based large model capability efficient evaluation method
By constructing a significant transition sample pool and a hierarchical storage mechanism, the problems of high computational cost and unstable results in large model evaluation are solved, realizing efficient and interpretable model capability evaluation, improving evaluation accuracy and adapting to the needs of rapid iteration.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING NORMAL UNIV AT ZHUHAI
- Filing Date
- 2026-04-27
- Publication Date
- 2026-07-31
AI Technical Summary
Existing large model evaluation methods are computationally expensive and inefficient, making it difficult to meet the needs of rapid iteration. Random sampling results are unstable, and manually constructed samples rely on experience and have poor generalization. Existing evaluation indicators lack interpretability and cannot clearly characterize the leap patterns of model capabilities.
By identifying stable transition points in the performance of models of different scales, a significant transition sample pool is constructed. A multi-scale inference module is used to generate inference result vectors, and a transition index identification module is used to allocate transition indices. The significant transition sample pool is constructed and stored hierarchically. A structured evaluation module is used to extract representative samples from the sample pool for evaluation, thereby reducing computational resource consumption and improving evaluation efficiency and interpretability.
It achieves efficient and lightweight model capability assessment, accurately captures model capability transition points, improves assessment accuracy and discriminative power, supports rapid iteration and interpretable assessment results, and reduces computational resource consumption.
Smart Images
Figure CN122489395A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence model evaluation technology, specifically to an efficient method for evaluating the capabilities of large models based on significant transition samples. This method is suitable for rapidly, lightweightly, and interpretably evaluating the relative capabilities of large-scale language models, visual models, and other artificial intelligence models. Background Technology
[0002] In recent years, with the rapid development of artificial intelligence technology, generative pre-trained models, represented by ChatGPT and DeepSeek, have demonstrated unprecedented capabilities in text understanding, question answering, logical reasoning, and generation tasks. These advancements have driven the widespread adoption of intelligent applications, gradually becoming core supporting technologies in various fields such as scientific research, education, healthcare, and industry. However, as model capabilities continue to improve, how to scientifically and effectively evaluate these capabilities has become a crucial issue of widespread concern in both academia and industry.
[0003] Existing large-scale model evaluation methods mostly rely on standardized public benchmark datasets, such as MMLU, GPQA, GSM8K, and MATH. While these benchmarks played a crucial role in driving early model development, their limitations have become increasingly apparent as model parameter sizes continue to grow. On the one hand, running these benchmarks in their entirety requires extremely high computational and time costs, consuming significant hardware resources and struggling to support frequent evaluations during rapid model iteration. On the other hand, research has found that some benchmark data may be memorized during the model's pre-training phase, leading to contaminated evaluation results and inflated scores. This "data leakage" problem makes it difficult for evaluation results to accurately reflect the model's reasoning and generalization capabilities.
[0004] To reduce computational overhead, existing research has attempted to use multiple random sampling methods, i.e., randomly selecting a portion of the samples from the benchmark for evaluation. While this method improves efficiency to some extent, the lack of sample representativeness often leads to large fluctuations in evaluation results, making it difficult to consistently differentiate models with similar capabilities, and it cannot avoid the negative impact of "data leakage." Some methods rely on experts manually selecting more challenging samples as a differentiated test set. Although this method is effective in specific scenarios, it is highly dependent on expert experience, subjective, and difficult to generalize for cross-domain and cross-architecture model evaluation. Furthermore, whether it's full evaluation or random sampling, existing methods mostly only provide macro-level indicators such as overall accuracy, lacking fine-grained characterization of the model's capability boundaries, and cannot answer at what capability level or parameter scale the model will stably solve tasks of a specific difficulty.
[0005] In summary, existing technologies for evaluating the capabilities of large models generally suffer from several shortcomings: full-scale benchmark evaluation is computationally expensive and inefficient, making it difficult to meet the needs of rapid iteration; random sampling methods yield unstable results, making it difficult to guarantee repeatability and discriminative power; manually constructed samples rely on experience and have poor generalization ability; existing evaluation metrics lack interpretability and cannot clearly characterize the leaps in model capabilities. These deficiencies severely limit the authenticity and practicality of the evaluation results, making it difficult to support the rapid application and optimization of large models in diverse scenarios. Summary of the Invention
[0006] To address the shortcomings of existing technologies, this invention provides an efficient method for evaluating the capabilities of large models based on significant transition samples. This method identifies stable transition points in the performance of models of different sizes, extracts highly discriminative significant transition samples, and constructs a structured evaluation sample pool based on these samples. Using this sample pool, a fast, lightweight, and interpretable capability evaluation of unknown models can be performed without running a complete benchmark test. This method reduces evaluation costs while accurately locating the model capability boundaries, thus effectively overcoming the shortcomings of existing technologies in terms of cost, efficiency, and interpretability.
[0007] The present invention achieves the above objectives through the following technical solutions: An efficient method for evaluating the capabilities of large models based on significant transition samples includes: Using a multi-scale inference module, inference operations are performed on the same benchmark sample on multiple reference models of different scales to generate an inference result vector; The transition index identification module performs in-depth analysis on the inference result vector to identify stable transition patterns in which the model's capabilities change from incorrect to correct answers, and assigns a corresponding transition index to each significant transition sample that satisfies the pattern. A significant transition sample pool construction module is adopted to store and organize all identified significant transition samples hierarchically according to the transition index, thereby constructing a significant transition sample pool; Using the structured evaluation module, a representative subset of samples is extracted from the significant transition sample pool according to a preset strategy to perform inference tests on the unknown model. Based on the test results, the accuracy distribution of the unknown model at different transition index levels is calculated to determine the capability boundary of the unknown model.
[0008] According to the efficient evaluation method for large model capabilities based on significant transition samples provided by the present invention, the multi-scale inference module employs the following steps when performing inference operations on the same benchmark sample across multiple reference models of different scales: When running in a distributed GPU cluster environment, assuming the benchmark sample is... The reference model series is The reference model size increases sequentially. Based on the resource availability of GPU nodes, the reference model series {M1, M2, ..., Mn} and the baseline sample Q are dynamically allocated. The inference result vector is defined by the following formula:
[0009] in, Representation Model The given answer is correct; Representation Model The given answer is incorrect, but the output is valid; Representation Model The output is invalid or illegal.
[0010] According to the present invention, a method for efficient evaluation of large model capabilities based on significant transition samples is provided. A task monitoring agent is deployed on each GPU node. The agent is bound to the inference process on the node, obtains the execution status information of the inference task in real time, and sends this status information to the central monitoring server at preset time intervals or in real time. The central monitoring server centrally manages and analyzes the status information from each GPU node, maintains a global task status table, and records the detailed status of each inference task. Once a GPU node is detected to have completed an inference task, the central monitoring server immediately sends a trigger command to the result collection program on the corresponding node. Upon receiving the trigger command, the result collection program reads the output inference result data from the GPU node that has completed inference, and performs integrity and accuracy verification on the data. After ensuring that the data is error-free, it accurately incorporates the inference result into the inference result vector according to the construction rules, and updates the global task status table to mark the inference task as completed.
[0011] According to the present invention, a method for efficient evaluation of the capabilities of large models based on significant transition samples is provided. The transition index identification module determines whether a sample has a unique stable capability transition point based on the inference result vector, and adopts the following steps: It receives the inference result vector as input information and performs a full sequence scan and analysis on the elements in the vector; Using pre-defined pattern recognition rules, detect whether there is a pattern of continuous error → continuous correct in the inference result vector, and verify that the pattern transition does not repeat itself after the transition; for significant transition samples that satisfy this pattern feature, i.e., have a unique stable capability transition point, determine the location where the transition occurs. Based on the determined transition positions, corresponding transition indices are assigned to the significant transition samples according to the transition index calculation rules, expressed as:
[0012] If the set is empty, then the samples with significant transitions do not meet the transition conditions; For significant transition samples that exhibit multiple repetitions, are always correct, are always incorrect, or contain invalid outputs, they are identified as not meeting the criteria for significant transition samples through logical judgment and are removed from the candidate set of significant transition samples.
[0013] According to the efficient evaluation method for large model capabilities based on significant transition samples provided by the present invention, the significant transition sample pool construction module performs hierarchical organization and resource management according to the following steps when constructing the significant transition sample pool: Based on the transition index value assigned to each significant transition sample by the transition index identification module, all identified significant transition samples are hierarchically classified, and samples with the same TI value are grouped into the same level, thereby constructing a significant transition sample pool with a clear hierarchical structure. The design of the significant transition sample pool follows the principle of balance. Through a pre-set sample quantity planning strategy, it is ensured that there are sufficient and relatively balanced samples at different TI levels. The established significant transition sample pool is used as the core data resource for subsequent evaluations and is managed and maintained in a unified manner. An efficient sample indexing and retrieval mechanism is established, which allows the sample pool to be reused in multiple evaluation tasks, avoiding the need to rerun the complete benchmark test for each evaluation, thereby significantly reducing resource consumption and improving evaluation efficiency.
[0014] The present invention provides an efficient method for evaluating the capabilities of large models based on significant transition samples, which is applicable to unknown models. During the evaluation, the structured evaluation module uses a stratified uniform sampling algorithm to extract samples from the significant transition sample pool to form a small subset. ; The extracted small subset S is used as input and imported into the unknown model. Initiating the inference process, unknown model It performs inference operations on each sample in a small subset S according to its own computational logic and outputs the corresponding inference results; Based on pre-set correct judgment criteria, for unknown models The correctness of the inference results at each transition index level is evaluated, and the accuracy rate at each transition index level is statistically calculated. ; Based on the accuracy of each transition index level calculated Define the unknown model according to the capacity boundary definition formula. The capability boundary is expressed by the following formula:
[0015] in, A pre-set threshold is used to ensure the stability of the evaluation results under different evaluation scenarios.
[0016] According to the present invention, a method for efficient evaluation of large model capabilities based on significant transition samples is provided. When setting pattern recognition rules, feature parameters of the continuous error → continuous correct pattern are pre-defined in the system, and a minimum length threshold for continuous errors is defined, including setting it to at least [X] consecutive error results; at the same time, a minimum length threshold for continuous correct results is defined, including at least [Y] consecutive correct results, which serve as the basic judgment basis for pattern recognition. The inference result vector is scanned element by element from the starting position. During the scanning process, the occurrence of consecutive errors and consecutive correct results is recorded in real time, and counters are used to count the number of consecutive errors and consecutive correct results respectively. Whenever an error result is encountered, the error counter is incremented by 1, and the correct counter is reset to 0; conversely, when a correct result is encountered, the correct counter is incremented by 1, and the error counter is reset to 0. During the scanning process, the values of the error counter and the correct counter are continuously checked to see if they meet the preset continuous error → continuous correct mode characteristic parameter requirements. If they are met, that is, the error counter reaches the minimum length threshold of continuous error and the correct counter subsequently reaches the minimum length threshold of continuous correct, then it is preliminarily determined that the mode may exist, and the mode start position and transition position at this time are recorded. After initially determining the existence of a continuous error → continuous correct pattern, continue scanning subsequent elements from the transition position; if an error result appears again during the subsequent scanning process, it indicates that the pattern has repeated after the transition, and this pattern determination is invalid; if no error result appears until the end of the inference result vector, it is confirmed that there is a continuous error → continuous correct pattern in the inference result vector and there is no repetition after the transition.
[0017] According to the present invention, a method for efficiently evaluating the capabilities of large models based on significant transition samples is provided. The sample size planning strategy includes the following execution steps: Before constructing the sample pool of significant transitions, the total number of all identified significant transition samples is first statistically analyzed to determine the total number of samples. N Simultaneously, based on the TI value assigned to the samples by the transition index identification module, the samples are divided into... k There are three different TI levels, denoted as follows: TI 1, TI 2,..., TI k ; Based on total sample size N and TI level number kPre-set the target number of samples that each TI level should include. n i ; Based on the target value of sample size n i For each TI level, a corresponding number of samples are drawn from all significant transition samples. During the sample extraction and tier filling process, the sample quantity distribution of each TI tier is monitored in real time, and the balance of the sample distribution is evaluated by calculating the deviation rate between the sample quantity of each tier and the target value. If the sample size deviation rate at a certain level is found to exceed the preset threshold, adjustment measures will be taken in a timely manner to ensure that the sample size at different TI levels in the final significant transition sample pool is sufficient and the distribution is relatively balanced.
[0018] This invention provides an efficient method for evaluating the capabilities of large models based on significant transition samples, in the case of unknown models. M new The input end is equipped with a standardized data interface for receiving and parsing small subsets. S The data structure of different types of samples; through data preprocessing operations, the input samples are transformed into those that conform to the unknown model. M new The input format required by the internal calculation logic; In unknown models M new An internal inference flow controller is set up to receive a pre-processed small subset. S After the input data is received, the inference process controller sends each sample in the subset into the model's computing core for inference operations in the order of inference. During the inference process, the inference process controller monitors the usage of computing resources in real time and dynamically adjusts the inference speed and task scheduling strategy according to the resource status. Among them, the unknown model M new The core computation relies on its pre-trained model parameters and algorithm logic to execute corresponding computational logic for each input sample to complete the inference of the sample; when the model is unknown... M new After completing the inference operation for a sample, the inference results are organized and packaged according to the set output format.
[0019] This invention provides an efficient method for evaluating the capabilities of large models based on significant transition samples, for unknown models. The correctness of the inference results at each transition index level sample is evaluated, including the following steps: Before the evaluation begins, clear and quantifiable criteria for correct judgment are pre-defined based on the specific type and requirements of the evaluation task. For classification tasks, if the true class labels of the samples are known, the unknown model will be... M new The predicted category is compared with the true category. If they match, the inference is considered correct. For regression tasks, an acceptable error range is set. If the error between the predicted value and the true value output by the unknown model is within this range, the inference is considered correct. For other types of tasks, a combination of manual evaluation and automatic indicators is used to set the correct judgment criteria. unknown model M new The reasoning result for each sample is matched with a pre-defined correct judgment criterion; for each sample, its correctness status is marked according to the matching result and recorded using a specified identifier; Based on the transition index level to which the sample belongs, the samples marked with correctness status are classified and statistically analyzed. A statistical counter is established for each transition index level to record the number of samples with correct reasoning and the total number of samples in that level. For each transition index level i Set the correct sample counter C correct i and total sample counter C total i During the process of traversing all samples, the corresponding counters are updated according to the transition exponent level and correctness label of the samples. After completing the correctness labeling and hierarchical statistics for all samples, based on each transition index level... i Correct sample counter C correct i and total sample counter C total i According to the accuracy calculation formula Acc i = C correct i / C total i Calculate unknown models M new Accuracy at this transition exponential level Acc i .
[0020] Therefore, compared with the prior art, the efficient evaluation method for large model capabilities based on significant transition samples proposed in this invention has the following advantages: 1. High discriminative capability assessment: By constructing a significant transition sample set, this invention can accurately capture the transition points in the capability performance of models of different sizes. Compared with full benchmark or random sampling methods, this scheme significantly improves the model capability discrimination and assessment accuracy.
[0021] 2. Low cost and high efficiency: This invention can complete capability positioning using a small number of highly discriminative samples without running a complete benchmark test, which greatly reduces the consumption of computing resources and time costs, and supports the rapid completion of large model evaluation in a limited computing power environment.
[0022] 3. High interpretability: The evaluation results of this invention not only provide the overall performance, but also clearly indicate the model's performance at different capability boundaries, helping researchers and developers to more intuitively understand the model's strengths and weaknesses, and improving the credibility and usability of the results.
[0023] 4. Adapting to rapid iteration needs: With the continuous updating and optimization of models, this invention can achieve lightweight and highly repeatable rapid evaluation, providing efficient support for scientific research experiments and industrial deployment.
[0024] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. Attached Figure Description
[0025] Figure 1 This is a flowchart of an embodiment of the present invention, which is a method for efficiently evaluating the capabilities of large models based on significant transition samples.
[0026] Figure 2 This is a schematic diagram of an embodiment of the present invention, which is a method for efficiently evaluating the capabilities of large models based on significant transition samples. Detailed Implementation
[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.
[0029] See Figure 1 and Figure 2 This embodiment provides an efficient method for evaluating the capabilities of large models based on significant transition samples, including: Step S1: Using the multi-scale inference module, inference operations are performed on the same benchmark sample on multiple reference models of different scales to generate an inference result vector. Step S2: Through the transition index identification module, the inference result vector is deeply analyzed to identify the stable transition pattern of the model capability from incorrect answer to correct answer, and a corresponding transition index is assigned to each significant transition sample that satisfies the pattern. Step S3: Using the significant transition sample pool construction module, all identified significant transition samples are hierarchically stored and organized according to the transition index to form a significant transition sample pool. Step S4: Using the structured evaluation module, a representative subset of samples is extracted from the significant transition sample pool according to a preset strategy to perform inference tests on the unknown model. Based on the test results, the accuracy distribution of the unknown model at different transition index levels is calculated to determine the capability boundary of the unknown model.
[0030] In step S1 above, the multi-scale inference module performs the following steps when performing inference operations on the same benchmark sample across multiple reference models of different scales: This development and implementation is based on inference frameworks with high-efficiency inference capabilities, such as vLLM. When running in a distributed GPU cluster environment, the baseline sample is assumed to be... The reference model series is The reference model size increases sequentially. Based on the computing performance, memory capacity, and other resource conditions of the GPU nodes, the reference model series {M1, M2, ..., Mn} and the benchmark sample Q are dynamically allocated. The inference result vector is defined by the following formula:
[0031] in, Representation Model The given answer is correct; Representation Model The given answer is incorrect, but the output is valid; Representation Model The output is invalid or illegal.
[0032] In this embodiment, a task monitoring agent is deployed on each GPU node. The agent is bound to the inference process on the node and obtains the execution status information of the inference task in real time, including but not limited to the inference progress percentage, the computing resources consumed, and the estimated remaining time. The agent sends this status information to the central monitoring server at preset time intervals or in real time. The central monitoring server centrally manages and analyzes the status information from each GPU node, maintains a global task status table, and records the detailed status of each inference task. Once a GPU node is detected to have completed an inference task, the central monitoring server immediately sends a trigger command to the result collection program on the corresponding node. Upon receiving the trigger command, the result collection program quickly reads the output inference result data from the GPU node that has completed inference, and performs integrity and accuracy verification on the data. After ensuring that the data is error-free, it accurately incorporates the inference result into the inference result vector IRV(Q) according to the construction rules, and updates the global task status table to mark the inference task as completed, so as to ensure that a complete and accurate inference result vector is obtained within a reasonable time.
[0033] In step S2 above, the transition index identification module uses the following steps when determining whether a sample has a unique stable capability transition point based on the inference result vector: The system receives the inference result vector IRV as input information and performs a full sequence scan and analysis on the elements in the vector. Using pre-defined pattern recognition rules, the system detects whether there is a pattern of "continuous errors → continuous correctness" in the inference result vector, and rigorously verifies that the pattern transition does not repeat itself. For significant transition samples that satisfy this pattern feature, i.e., have a unique stable capability transition point, the system determines the location where the transition occurs. Based on the determined transition positions, corresponding transition indices are assigned to the significant transition samples according to the transition index calculation rules, expressed as:
[0034] If the set is empty, then the samples with significant transitions do not meet the transition conditions; If a sample's result sequence exhibits a pattern of "consecutive errors → consecutive correct results," and this transition does not repeat itself, then the sample is considered a Significant Transition Sample (STS). For example, This is a typical significant transition pattern, with the transition occurring in the third model. In this case, the Transition Index (TI) is assigned to this sample as 3.
[0035] For results that are repeated multiple times (e.g., the result sequence is (0,0,1,0,1,1,0,1)), consistently correct (e.g., (1,1,1,1,1,1,1,1)), consistently incorrect (e.g., (0,0,0,0,0,0,0,0)), or contain invalid output (e.g., (0,0,0,1, Significant transition samples of 1,1,1,1) are identified as not meeting the conditions for significant transition samples through logical judgment and are removed from the candidate set of significant transition samples.
[0036] In step S3 above, the significant transition sample pool construction module performs hierarchical organization and resource management according to the following steps when constructing the significant transition sample pool (STS Pool): Based on the transition index (TI) value assigned to each significant transition sample (STS) by the transition index identification module, all identified significant transition samples are hierarchically classified, and samples with the same TI value are grouped into the same level, thereby constructing a significant transition sample pool (STS Pool) with a clear hierarchical structure. The design of the significant transition sample pool follows the principle of balance. Through a pre-set sample quantity planning strategy, it is ensured that there are sufficient and relatively balanced samples at different TI levels. The specific number is reasonably set according to the evaluation needs and the overall sample size to effectively avoid the situation of missing levels in the subsequent evaluation process. The completed Significant Transition Sample Pool (STS Pool) is used as the core data resource for subsequent evaluations and is managed and maintained in a unified manner. An efficient sample indexing and retrieval mechanism is established, which allows the sample pool to be reused in multiple evaluation tasks, avoiding the need to rerun the complete benchmark test for each evaluation, thereby significantly reducing resource consumption and improving evaluation efficiency.
[0037] In step S4 above, when dealing with the unknown model During the evaluation, the structured evaluation module uses a stratified uniform sampling algorithm to extract samples from the significant transition sample pool to form a small subset. The sampling algorithm ensures that samples are drawn at each transition index (TI) level according to a preset ratio, guaranteeing the representativeness of subset S to each level of STS Pool. The number of samples n+1 is reasonably set according to factors such as evaluation accuracy and computing resources. The extracted small subset S is used as input and imported into the unknown model. Initiating the inference process, unknown model It performs inference operations on each sample in a small subset S according to its own computational logic and outputs the corresponding inference results; Based on pre-set correct judgment criteria, for unknown models The correctness of the inference results at each transition index level is evaluated, and the accuracy rate at each transition index level is statistically calculated. ; Based on the accuracy of each transition index level calculated Define the unknown model according to the capacity boundary definition formula. The capability boundary is expressed by the following formula:
[0038] in, A pre-set threshold (e.g., 80%) is used to ensure the stability of evaluation results under different evaluation scenarios, through reasonable setting. The value is a model that can effectively distinguish different ability levels.
[0039] In this embodiment, when setting pattern recognition rules, feature parameters for the consecutive error → consecutive correct pattern are pre-defined in the system, and the minimum length threshold for consecutive errors is specified, including setting it to at least [X] consecutive error results (…). v i =0); Simultaneously define a minimum length threshold for consecutive correct results, such as at least [Y] consecutive correct results ( v i =1), which serves as the basic criterion for pattern recognition; The inference result vector is scanned element by element from the starting position. During the scanning process, the occurrence of consecutive errors and consecutive correct results is recorded in real time, and counters are used to count the number of consecutive errors and consecutive correct results respectively. Whenever an error result is encountered, the error counter is incremented by 1, and the correct counter is reset to 0. Conversely, when a correct result is encountered, the correct counter is incremented by 1, and the error counter is reset to 0.
[0040] During the scanning process, the values of the error counter and the correct counter are continuously checked to see if they meet the preset continuous error → continuous correct mode characteristic parameter requirements. If they are met, that is, the error counter reaches the minimum length threshold of continuous error and the correct counter subsequently reaches the minimum length threshold of continuous correct, then it is preliminarily determined that the mode may exist, and the mode start position and transition position at this time are recorded.
[0041] After initially determining that there is a continuous error → continuous correct pattern, continue scanning subsequent elements from the transition position; if an error result occurs again during the subsequent scanning process ( v iIf the result is 0, it indicates that there is a repetition after the mode transition, and the mode determination is invalid. If no error result appears until the end of the inference result vector, it is confirmed that there is a continuous error → continuous correct mode in the inference result vector and there is no repetition after the transition. It can be further used for subsequent operations such as transition index allocation.
[0042] In this embodiment, the sample size planning strategy includes the following execution steps: Before constructing the sample pool of significant transitions, the total number of all identified significant transition samples is first statistically analyzed to determine the total number of samples. N Simultaneously, based on the TI value assigned to the samples by the transition index identification module, the samples are divided into... k There are three different TI levels, denoted as follows: TI 1, TI 2,..., TI k ; Based on total sample size N and TI level number k Pre-set the target number of samples that each TI level should include. n i ( i =1,2,..., k The target value ni The settings comprehensively consider factors such as the required assessment accuracy, the overall distribution of the sample, and the importance of each level. For example, a proportional allocation strategy can be adopted, i.e. n i = N / k Alternatively, based on the criticality of each TI level in the model capability assessment, a weighted allocation strategy can be adopted to allocate more sample numbers to the critical levels.
[0043] According to the pre-set target sample size n i From all significant transition samples, random sampling or other suitable sampling methods are used to draw a corresponding number of samples for each TI level. During the sampling process, the randomness and representativeness of the samples are ensured to avoid human bias. If the sample size for a certain TI level is less than the target value... n i If the remaining samples are insufficient to completely fill all the target values of all levels, the target values of each level will be adjusted appropriately according to the actual situation to ensure that the number of samples in each TI level in the final significant transition sample pool is relatively balanced.
[0044] During sample extraction and tier filling, the sample size distribution at each TI tier is monitored in real time. The balance of the sample distribution is assessed by calculating the deviation rate between the sample size at each tier and the target value. If the deviation rate of the sample size at a certain tier exceeds a preset threshold (e.g., ±10%), adjustment measures are taken promptly, such as resampling or allocating samples from other tiers, to ensure that the final significant transition sample pool has a sufficient number of samples at different TI tiers and a relatively balanced distribution, meeting the needs of subsequent evaluation tasks.
[0045] In this embodiment, in the unknown model M new The input end is equipped with a standardized data interface that is compatible with multiple data formats and can receive and parse small subsets. S The data structure includes different types of samples (such as text, images, audio, etc., determined according to the type of evaluation task). Through data preprocessing, the input samples are transformed to conform to the unknown model. M new The internal computational logic requires specific input formats, such as word vector encoding for text samples and size normalization and pixel value standardization for image samples.
[0046] In unknown models M new An internal inference flow controller is set up to receive a pre-processed small subset. S After receiving the input data, the inference process controller sequentially sends each sample in the subset to the model's computational core for inference operations according to a preset inference order (such as sequential inference, parallel inference, etc., selected based on the model architecture and computing resources). During inference, the inference process controller monitors the usage of computing resources in real time, such as GPU memory usage and computational core utilization, and dynamically adjusts the inference speed and task scheduling strategy based on resource conditions to ensure efficient and stable inference.
[0047] Unknown model M new The core computation of a deep neural network relies on its pre-trained model parameters and specific algorithmic logic to perform feature extraction, transformation, and prediction operations on each input sample. For example, in a deep neural network model, the input sample is gradually mapped to the output space through the weight calculation of multiple layers of neurons and the action of activation functions, resulting in the corresponding inference result. Different model types (such as convolutional neural networks, recurrent neural networks, and Transformers) execute corresponding computational logic to complete the inference on the sample based on their own structural and functional characteristics.
[0048] When the unknown model M newAfter completing the inference operation for a sample, the inference results are organized and packaged according to the set output format. The output format can be customized according to the evaluation requirements. For example, for classification tasks, the output results can be the probability distribution of the sample belonging to each category; for regression tasks, the output results can be continuous predicted values. The organized inference results are transmitted to an external storage device or the result collection module of the evaluation system for storage via the output interface, so as to perform accuracy statistics and capability boundary determination on the inference results of unknown models at different TI levels later.
[0049] In this embodiment, for the unknown model The correctness of the inference results at each transition index level sample is evaluated, including the following steps: Before the evaluation begins, clear and quantifiable criteria for correct judgment should be pre-defined based on the specific type and requirements of the evaluation task. For classification tasks, if the true class labels of the samples are known, the unknown model will be... M new The predicted category is compared with the true category. If they match, the inference is considered correct. For regression tasks, an acceptable error range is set. If the error between the predicted value and the true value output by the unknown model is within this range, the inference is considered correct. For other types of tasks, such as generation tasks, a combination of manual evaluation and automatic indicators (such as BLEU, ROUGE, etc.) can be used to set the correct judgment criteria.
[0050] unknown model M new The reasoning result for each sample is matched against pre-defined correctness criteria. For each sample, its correctness status is marked according to the matching result, using specific identifiers (e.g., "1" for correct and "0" for incorrect). During the matching process, it is ensured that the judgment is strictly made according to the set criteria to avoid subjective factors and misjudgments.
[0051] Based on the transition index (TI) level to which the samples belong, the samples labeled with correctness states are classified and statistically analyzed. A statistical counter is established for each TI level to record the number of correctly inferred samples and the total number of samples in that level. Specifically, for each TI level... i Set the correct sample counter C correct i and total sample counter C total i During the process of traversing all samples, the corresponding counters are updated according to the TI level and correctness label of the sample.
[0052] After completing the correctness labeling and hierarchical statistics for all samples, based on each TI level... i Correct sample counter C correct i and total sample counter C total i According to the accuracy calculation formula Acc i = C correct i / C total i Calculate unknown models M new Accuracy at this TI level Acc i .
[0053] In practical applications, the method of this invention runs in a distributed inference environment that deploys the Qwen3 series models. The Qwen3 series models cover eight models with different parameter scales: Qwen3-0.6B, Qwen3-1.7B, Qwen3-4B, Qwen3-8B, Qwen3-14B, Qwen3-30B-A3B, Qwen3-32B, and Qwen3-235B-A22B, which can provide multi-scale references for screening significant transition samples.
[0054] The multi-scale inference module receives samples from various benchmark datasets, including MMLU, MMLU-Pro, GPQA, SuperGPQA, GSM8K, and MATH, covering different tasks such as language understanding, scientific reasoning, and mathematical computation. All samples are input into the Qwen3 series models to obtain the inference result vector IRV. For example, if a sample answers incorrectly on Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B, but answers correctly on a model with a parameter size greater than 4B, then IRV(Q) = (0,0,0,1,1,1,1,1).
[0055] The transition index identification module analyzes the IRV. For the above sample, it changes from error to correct at the fourth model (Qwen3-8B), and remains correct at larger-scale models. Therefore, it is identified as a significant transition sample, and the transition index TI=4 is assigned. If the IRV repeatedly shows 0,0,1,0,1,1,0,1, consistently correct 1,1,1,1,1,1,1,1,1, consistently incorrect 0,0,0,0,0,0,0,0, or contains invalid output 0,0,0,1, it is considered a significant transition sample. Samples with a value of 1,1,1,1 are excluded from the STS (Simplified Chinese Study).
[0056] The significant transition sample pool construction module stores the identified STSs hierarchically according to their TI values, forming a reusable evaluation significant transition sample pool (STS Pool). This hierarchical organization method ensures that targeted sampling and testing can be carried out for different capability levels when evaluating unknown models in the future.
[0057] When evaluating an unknown model, the structured evaluation module first extracts a structured sample subset from the STS Pool. For example, 20 samples are randomly selected from each level from TI=1 to TI=8, forming an evaluation set of 160 samples. This evaluation set is then input into the model to be evaluated for inference to calculate the model's accuracy at each TI level.
[0058] In a complete implementation, the four modules interact as follows: the multi-scale inference module generates IRVs; the transition detection and index labeling module filters STSs and labels TIs; the significant transition sample pool construction module organizes samples and maintains hierarchical balance; and the structured evaluation module uses this sample pool to test subsets to obtain the capability boundary of the model to be evaluated. This method eliminates the need to run large-scale benchmark tests on unknown models, allowing for accurate determination of their capability levels in a relatively short time.
[0059] To verify the effectiveness of the method in this embodiment, experiments were conducted on the LLaMA3-8B and GLM4-9B models, respectively. The results show that the LLaMA3-8B model achieves an accuracy of 91% on samples with TI=2, 85% on samples with TI=3, and drops to 57% on samples with TI=4. Based on a threshold θ=80%, its capability boundary is determined to be TI=3, meaning this model's capability is comparable to Qwen3-4B. Correspondingly, when using GLM4-9B as the model to be evaluated, its accuracy remains at 83% on samples with TI=4, but drops to 61% on samples with TI=5. Therefore, its capability boundary is determined to be TI=4, corresponding to the capability level of the Qwen3-8B model. The experimental results are shown in Table 1. These results are completely consistent with the model ranking obtained using the full benchmark test. Furthermore, the difference in overall accuracy between LLaMA3-8B and GLM4-9B is small, but the significant jump sample method clearly reveals their discriminatory ability across different difficulty levels. Specifically, the GLM4-9B significantly outperforms the LLaMA3-8B on complex tasks at the high TI level, a difference that is not significant in traditional overall accuracy metrics.
[0060] Table 1: Experimental results of LLaMA3-8B and GLM4-9B models Model Name TI = 2 accuracy TI = 3 accuracy TI = 4 accuracy TI = 5 accuracy Capability Boundaries (TI Value) Corresponding to Qwen3 model capability level LLaMA3 - 8B 91% 85% 57% - 3 Qwen3 - 4B GLM4 - 9B - - 83% 61% 4 Qwen3 - 8B The experimental results show that the LLaMA3-8B model achieves high accuracy (approaching or exceeding 80%) on samples with TI = 2 and TI = 3, but its accuracy drops significantly to 57% on samples with TI = 4. Based on a threshold θ = 80%, its capability boundary is determined to be TI = 3, indicating that this model's capability is comparable to Qwen3-4B. The GLM4-9B model achieves 83% accuracy on samples with TI = 4, but its accuracy drops to 61% on samples with TI = 5. Therefore, its capability boundary is determined to be TI = 4, corresponding to the capability level of the Qwen3-8B model.
[0061] The experimental results above demonstrate that the efficient large-model capability evaluation method based on significant transition samples proposed in this invention can accurately identify the capability boundaries of models in different benchmark tasks and distinguish the relative levels of models of different sizes. This method not only improves the efficiency and discriminative power of the evaluation but also provides stronger interpretability.
[0062] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0063] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. Any non-substantial changes and substitutions made by those skilled in the art based on the present invention shall fall within the scope of protection claimed by the present invention.
Claims
1. A method for efficiently evaluating the capabilities of large models based on significant transition samples, characterized in that, include: Using a multi-scale inference module, inference operations are performed on the same benchmark sample on multiple reference models of different scales to generate an inference result vector. The transition index identification module performs in-depth analysis on the inference result vector to identify stable transition patterns in which the model's capabilities change from incorrect to correct answers, and assigns a corresponding transition index to each significant transition sample that satisfies the pattern. A significant transition sample pool construction module is adopted to store and organize all identified significant transition samples in a hierarchical manner according to the transition index, thereby constructing a significant transition sample pool; Using the structured evaluation module, a representative subset of samples is extracted from the significant transition sample pool according to a preset strategy to perform inference tests on the unknown model. Based on the test results, the accuracy distribution of the unknown model at different transition index levels is calculated to determine the capability boundary of the unknown model.
2. The method of claim 1, wherein, When performing inference operations on the same benchmark sample across multiple reference models of different scales, the multi-scale inference module employs the following steps: When running in a distributed GPU cluster environment, assuming the benchmark sample is... The reference model series is The reference model size increases sequentially. Based on the resource availability of GPU nodes, the reference model series {M1, M2, ..., Mn} and the baseline sample Q are dynamically allocated. The inference result vector is defined by the following formula: wherein, representation model The given answer is correct; representation model The given answer is incorrect, but the output is legal; representation model The output is invalid or illegal.
3. The method according to claim 2, characterized in that: A task monitoring agent is deployed on each GPU node. This agent is bound to the inference process on the node, obtains the execution status information of the inference task in real time, and sends this status information to the central monitoring server at preset time intervals or in real time. The central monitoring server centrally manages and analyzes the status information from each GPU node, maintains a global task status table, and records the detailed status of each inference task. Once a GPU node is detected to have completed an inference task, the central monitoring server immediately sends a trigger command to the result collection program on the corresponding node. After receiving the trigger command, the results collection program reads the output inference result data from the GPU node that has completed inference, and performs integrity verification and accuracy verification on the data. After ensuring the data is correct, the inference result is accurately incorporated into the vector according to the construction rules of the inference result vector. At the same time, the global task status table is updated to mark the inference task as completed.
4. The method of claim 2, wherein, The transition index identification module determines whether a sample has a unique stable capability transition point based on the inference result vector, using the following steps: It receives the inference result vector as input information and performs a full sequence scan and analysis on the elements in the vector; Using pre-defined pattern recognition rules, detect whether there is a pattern of continuous error → continuous correct in the inference result vector, and verify that the pattern transition does not repeat itself after the transition; for significant transition samples that satisfy this pattern feature, i.e., have a unique stable capability transition point, determine the location where the transition occurs. Based on the determined transition positions, corresponding transition indices are assigned to the significant transition samples according to the transition index calculation rules, expressed as: If the set is empty, then the samples with significant transitions do not meet the transition conditions; For significant transition samples that exhibit multiple repetitions, are always correct, are always incorrect, or contain invalid outputs, they are identified as not meeting the criteria for significant transition samples through logical judgment and are removed from the candidate set of significant transition samples.
5. The method of claim 2, wherein, When constructing the significant transition sample pool, the module performs hierarchical organization and resource management according to the following steps: Based on the transition index value assigned to each significant transition sample by the transition index identification module, all identified significant transition samples are hierarchically classified, and samples with the same TI value are grouped into the same level, thereby constructing a significant transition sample pool with a clear hierarchical structure. The design of the significant transition sample pool follows the principle of balance. Through a pre-set sample quantity planning strategy, it is ensured that there are sufficient and relatively balanced samples at different TI levels. The established significant transition sample pool is used as the core data resource for subsequent evaluations and is managed and maintained in a unified manner. An efficient sample indexing and retrieval mechanism is established, which allows the sample pool to be reused in multiple evaluation tasks, avoiding the need to rerun the complete benchmark test for each evaluation, thereby significantly reducing resource consumption and improving evaluation efficiency.
6. The method according to claim 2, characterized in that: In the case of unknown models During the evaluation, the structured evaluation module uses a stratified uniform sampling algorithm to extract samples from the significant transition sample pool to form a small subset. ; The extracted small subset S is used as input and imported into the unknown model. Initiating the inference process, unknown model It performs inference operations on each sample in a small subset S according to its own computational logic and outputs the corresponding inference results; Based on pre-set correct judgment criteria, for unknown models The correctness of the inference results at each transition index level is evaluated, and the accuracy rate at each transition index level is statistically calculated. ; Based on the accuracy of each transition index level calculated Define the unknown model according to the capacity boundary definition formula. The capability boundary is expressed by the following formula: wherein is a pre-set threshold value for ensuring the stability of the evaluation result under different evaluation scenarios.
7. The method according to claim 4, characterized in that: When setting pattern recognition rules, feature parameters for the consecutive error → consecutive correct pattern are pre-defined in the system, and a minimum length threshold for consecutive errors is defined, including setting it to at least [X] consecutive error results; at the same time, a minimum length threshold for consecutive correct results is defined, including at least [Y] consecutive correct results, which serve as the basic judgment criteria for pattern recognition. The inference result vector is scanned element by element starting from the starting position. During the scanning process, the occurrence of consecutive errors and consecutive correct results is recorded in real time, and counters are used to count the number of consecutive errors and consecutive correct results respectively. Whenever an error result is encountered, the error counter is incremented by 1, and the correct counter is reset to 0; conversely, when a correct result is encountered, the correct counter is incremented by 1, and the error counter is reset to 0. During the scanning process, the values of the error counter and the correct counter are continuously checked to see if they meet the preset continuous error → continuous correct mode characteristic parameter requirements. If they are met, that is, the error counter reaches the minimum length threshold of continuous error and the correct counter subsequently reaches the minimum length threshold of continuous correct, then it is preliminarily determined that the mode may exist, and the mode start position and transition position at this time are recorded. After initially determining that there is a continuous error → continuous correct pattern, continue scanning subsequent elements from the transition position; if an error result occurs again during the subsequent scanning process, it indicates that the pattern transition has repeated, and this pattern determination is invalid. If no further erroneous results appear until the end of the inference result vector, then it is confirmed that there is a continuous error → continuous correct pattern in the inference result vector that meets the requirements and there is no repetition after the transition.
8. The method of claim 5, wherein, The sample size planning strategy includes the following steps: Before constructing the sample pool of significant transitions, the total number of all identified significant transition samples is first statistically analyzed to determine the total number of samples. N Simultaneously, based on the TI value assigned to the samples by the transition index identification module, the samples are divided into... k There are three different TI levels, denoted as follows: TI 1, TI 2,..., TI k ; based on the total number of samples N and the number of TI levels k a target value of the number of samples to be included in each TI level is set in advance n i ; Based on the target value of sample size n i For each TI level, a corresponding number of samples are drawn from all significant transition samples. During the sample extraction and tier filling process, the sample quantity distribution of each TI tier is monitored in real time, and the balance of the sample distribution is evaluated by calculating the deviation rate between the sample quantity of each tier and the target value. If the sample size deviation rate at a certain level is found to exceed the preset threshold, adjustment measures will be taken in a timely manner to ensure that the sample size at different TI levels in the final significant transition sample pool is sufficient and the distribution is relatively balanced.
9. The method according to claim 6, characterized in that: In unknown models M new The input end is equipped with a standardized data interface for receiving and parsing small subsets. S The data structure of different types of samples; through data preprocessing operations, the input samples are transformed into those that conform to the unknown model. M new The input format required by the internal computational logic; In unknown models M new An internal inference flow controller is set up to receive a pre-processed small subset. S After the input data is received, the inference process controller sends each sample in the subset into the model's computational core for inference operations in the order of inference. During the inference process, the inference process controller monitors the usage of computing resources in real time and dynamically adjusts the inference speed and task scheduling strategy according to the resource status. Among them, the unknown model M new The core computation relies on its pre-trained model parameters and algorithm logic to execute corresponding computational logic for each input sample to complete the inference of the sample; when the model is unknown... M new After completing the inference operation for a sample, the inference results are organized and packaged according to the set output format.
10. The method according to claim 6, characterized in that, For unknown models The correctness of the inference results at each transition index level sample is evaluated, including the following steps: Before the evaluation begins, clear and quantifiable criteria for correct judgment are pre-defined based on the specific type and requirements of the evaluation task. For classification tasks, if the true class labels of the samples are known, the unknown model will be... M new The predicted category is compared with the true category. If they match, the inference is considered correct. For regression tasks, an acceptable error range is set. If the error between the predicted value and the true value output by the unknown model is within this range, the inference is considered correct. For other types of tasks, a combination of manual evaluation and automatic indicators is used to set the correct judgment criteria. unknown model M new The reasoning result for each sample is matched with a pre-defined correct judgment criterion; for each sample, its correctness status is marked according to the matching result and recorded using a specified identifier; Based on the transition index level to which the sample belongs, the samples marked with correctness status are classified and statistically analyzed. A statistical counter is established for each transition index level to record the number of samples with correct reasoning and the total number of samples in that level. For each transition index level i Set the correct sample counter C correct i and total sample counter C total i During the process of traversing all samples, the corresponding counters are updated according to the transition exponent level and correctness label of the samples. After completing the correctness labeling and hierarchical statistics for all samples, based on each transition index level... i Correct sample counter C correct i and total sample counter C total i According to the accuracy calculation formula Acc i = C correct i / C total i Calculate unknown models M new Accuracy at this transition exponential level Acc i .