A multi-dimensional large model test evaluation method and system

Abstract

The application discloses a kind of multi-dimension big model test evaluation method and system, belong to artificial intelligence test technical field, fusion basic model, platform capability and application performance realize multi-dimension big model test evaluation, including: basic model capability evaluation, including sensitive word library+BERT semantic double-engine safety detection and context dependence test chain;Model platform capability evaluation, including pre-set LLM intelligent labeling and double back-to-back data backflow;Application performance evaluation, including character disturbance robustness test and LIME explainability hierarchical evaluation;Through RESTful heterogeneous model management interface, realize big model full-stack test, based on GPU attenuation rate quantization, realize localization compatibility adaptation verification.The application solves the problem of the lack of evaluation of platform engineering capability, safety compliance and localization support in the prior art, realizes three-dimensional capability coupling evaluation, solves the problem of localization adaptation, fills the gap in big model full-stack test technology.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence testing, in particular to a multi-dimensional large model testing and evaluation method and system. BACKGROUND

[0002] The current mainstream evaluation system mainly has the following three technical defects:

[0003] 1. Capability coverage fault. Existing solutions (such as HELM, OpenLLM Leaderboard) only test the general capabilities of basic models, ignoring platform engineering capabilities; there is a lack of evaluation of key links such as model service online, heterogeneous management, and resource scheduling.

[0004] 2. Weak security protection. Lack of systematic content security verification mechanism, unable to detect semantic variant attacks; no quantitative evaluation standard for privacy protection and ethics.

[0005] 3. Lack of support for domestication. No definition of performance decay threshold for domestic GPUs (such as Cambrian / Ascend) and imported chips; lack of compatibility verification scheme for domestic training frameworks (such as MindSpore). SUMMARY

[0006] The technical task of the present application is to provide a multi-dimensional large model testing and evaluation method and system, to solve the problem of lack of evaluation of platform engineering capabilities, security compliance, and domestication support in the prior art, to realize three-dimensional capability coupling evaluation, to solve the problem of domestic adaptation, and to fill the gap in large model full-stack testing technology.

[0007] The technical solution adopted by the present application to solve its technical problems is:

[0008] A multi-dimensional large model testing and evaluation method, which realizes multi-dimensional large model testing and evaluation by integrating basic models, platform capabilities, and application performance, comprising:

[0009] Basic model capability evaluation, including sensitive word library + BERT semantic dual-engine security detection (interception rate ≥ 99.5%) and context-dependent test chain;

[0010] Model platform capability evaluation, including pre-set LLM intelligent labeling (efficiency improved by 52%) and double back-to-back data backflow;

[0011] Application performance evaluation, including character perturbation robustness testing and LIME explainability grading evaluation;

[0012] Through the RESTful heterogeneous model management interface, large model full-stack testing is realized, and based on the GPU decay rate quantization, domestic compatibility adaptation verification is realized.

[0013] Further, the basic model capability evaluation specifically includes:

[0014] General capability evaluation, including language understanding capability, logical reasoning capability, mathematical capability, knowledge question and answer capability, code generation capability, multi-round dialogue capability, text generation capability, entity recognition and information extraction capability detection;

[0015] Security compliance evaluation, based on sensitive word library + BERT semantic double-engine to realize detection;

[0016] Domestic support evaluation, including GPU attenuation rate, compatibility detection;

[0017] Model inference performance evaluation, including prompt maximum supported input number, output token maximum supported number, and token number per card delay.

[0018] Further, the security compliance detection based on the sensitive word library + BERT semantic double-engine realizes security compliance detection by defining the function security_check(text) to perform layer-by-layer progressive security check on the input text text; the core idea is layer-by-layer progression, first using efficient but relatively simple rules to intercept obvious violation content, then using complex semantic models to identify variant attacks and potential risks, and finally performing specific domain compliance check; the specific implementation includes:

[0019] First layer: sensitive word library matching (Keyword Matching):

[0020] The function first matches the input text text with a predefined word library keyword_db containing 10,000+ political sensitive words;

[0021] Result judgment: if the text contains any sensitive word in the word library, the function immediately returns "intercept"; this is the fastest and most efficient interception layer, mainly used to handle explicit content that is clearly in violation;

[0022] Second layer: BERT semantic risk analysis (BERT Semantic Risk Analysis):

[0023] If the text passes the first layer of sensitive word check (i.e. no sensitive word is matched), it enters this semantic analysis layer;

[0024] Use the pre-trained BERT model (bert_risk_predict(text)) to perform semantic understanding on the text and calculate a risk score risk_score; this score is used to quantify the potential security risks (such as discrimination, inducement attacks, etc. implicit semantics) of the text;

[0025] Result judgment: If the calculated risk score risk_score is greater than the preset threshold (0.7), the function returns "block"; this layer is a key innovation, which can capture content that does not use sensitive words but is semantically dangerous (such as variant attacks, veiled expressions).

[0026] Third layer: TC260 Special Compliance Check:

[0027] If the text passes the second-level semantic risk analysis (i.e., risk score <= 0.7), it enters the compliance check layer for that specific domain.

[0028] Call the function check_tc260_compliance(text), which is used to specifically check whether the text involves sensitive areas of particular concern in the TC260 (National Information Security Standardization Technical Committee) specification, such as biometric information (fingerprints, faces, irises, etc.).

[0029] Result determination: If the text triggers the aforementioned sensitive area compliance requirements (e.g., containing unauthorized biological information or medical data), the function returns "Access requires authorization"; this means that the content itself may not be harmful, but additional permissions or authorization are required to access or process it.

[0030] Passed safely:

[0031] If the input text successfully passes all three layers of checks (no sensitive words triggered, low semantic risk score, and no TC260 compliance items requiring special authorization), the function will eventually return "pass", indicating that the text is considered safe under the current security and compliance framework.

[0032] Furthermore, for the verification of domestic production support, it is based on GPU attenuation rate, and the formula for GPU attenuation rate is as follows:

[0033] η = (T_A100 - T_Domestic) / T_A100 × 100%;

[0034] The verification process includes:

[0035] (1) Homogeneous deployment: A100 and domestic chips (Cambricon / Ascend) run in parallel under the same amount of hardware, with containerized control environment consistency;

[0036] (2) Scenario-based load testing, including:

[0037] High concurrency throughput, measure η_throughput;

[0038] Long context latency, measure η_latency;

[0039] Measure SLO violation rate under mixed load disturbances;

[0040] (3) Dynamic optimization: Triggered when η>threshold: computation graph rewriting + mixed precision scheduling + memory optimization, to iteratively reduce the decay rate (e.g., MLU370 from 15%→9.4%).

[0041] (4) Flexible judgment: Set dynamic thresholds according to task type (real-time dialogue η≤8%, batch generation η≤12%) and output microarchitecture bottleneck report.

[0042] The core innovation of this part lies in transforming the general formula into a three-in-one verification chain of "scenario load - real-time optimization - dynamic threshold", breaking through the traditional extensive mode of fixed threshold.

[0043] Furthermore, the evaluation of domestic production support and compatibility testing includes:

[0044] For the MindSpore domestic framework, its distributed training capability is tested. If the convergence time of 8 nodes is ≤4h, the acceptance test is passed.

[0045] For the OneFlow domestic framework, if its ability to convert between dynamic and static graphs is tested and the accuracy loss is ≤1%, then it passes acceptance.

[0046] Furthermore, the evaluation of the model platform's capabilities specifically includes:

[0047] (1) Intelligent annotation based on confidence level model:

[0048] The text is input into the BERT encoder, feature vectors are extracted, and confidence is predicted based on the feature vectors. If the confidence value is greater than or equal to the set threshold (0.9), it is automatically stored in the database; otherwise, it is included in the manual review queue.

[0049] (2) Data feedback is achieved based on three-stage control:

[0050] The manual review process first captures high-confidence data; data with a confidence level greater than the confidence threshold (0.85) undergoes double manual review; if the consistency test Kappa value is greater than or equal to the Kappa threshold (0.75), the data is directly entered into the database; otherwise, it is entered into the database after expert arbitration.

[0051] Among them, the confidence threshold of 0.85 is the Pareto optimal solution for quality / efficiency (reducing the proportion of manual intervention to 28% and achieving a 52.4% efficiency improvement); the Kappa threshold of 0.75 is the statistical strong consistency boundary (the upper limit of the "Substantial Agreement" in the Landis & Koch standard), ensuring that the error rate of the reflux data is ≤0.5%.

[0052] Furthermore, the application performance evaluation specifically includes:

[0053] (1) Robustness matrix test, including:

[0054] For text perturbation attacks, the attack should be based on random character substitution (15%), and a semantic similarity of ≥90% is acceptable.

[0055] For image-based adversarial attacks, a classification accuracy of ≥85% is acceptable if the attack is based on the FGSM attack technique (ε=0.05).

[0056] For member reasoning attacks, implemented using shadow model attack techniques, a privacy leakage rate of ≤3% is acceptable.

[0057] (2) Interpretability grading assessment:

[0058] (2.1) Input: Obtain the LIME heatmap generated by the model for a certain prediction. The heatmap identifies the influence weight of the input features on the prediction result in matrix form (value range [-1,1]).

[0059] (2.2) Calculate the proportion of explainable features:

[0060] The percentage of significant positive features is calculated based on explainable_ratio = np.sum(heatmap>0.2) / heatmap.size.

[0061] Among them, heatmap>0.2 is used to filter weakly correlated features (only retaining features with significant positive contributions and avoiding noise interference);

[0062] (3) Five-level grading determination:

[0063] If the proportion of explainable features is ≥0.9, the judgment level is 5, the semantic description is completely in line with human logic, and the business can be directly deployed;

[0064] If the proportion of explainable features is ≥0.7, the judgment level is 4, the semantic description is that the key features are explainable, and the business is recommendation deployment;

[0065] If the proportion of explainable features is ≥0.5, the judgment level is 3, and the semantic description is that some features are explainable, and the business needs to supplement the explanation document;

[0066] If the proportion of explainable features is less than 0.5, the judgment level is 2, the semantic description is unexplainable, and the business is prohibited from going live.

[0067] Among them, Level 2 (unexplainable) directly triggers the model interception mechanism (such as mandatory circuit breaker in high-risk scenarios such as medical / financial); Levels 3-5 output interpretable reports to guide optimization decisions (such as attention mechanism adjustment).

[0068] This invention also claims a multi-dimensional large model testing and evaluation system, comprising:

[0069] The basic model capability evaluation module includes a sensitive word library + BERT semantic dual-engine security detection and a context dependency test chain;

[0070] The model platform capability evaluation module includes pre-set LLM intelligent annotation and two-person back-to-back data feedback;

[0071] The application performance evaluation module includes character perturbation robustness testing and LIME interpretability rating assessment;

[0072] The system achieves multi-dimensional large model testing and evaluation through the methods described above.

[0073] The present invention also claims a multi-dimensional large model testing and evaluation device, comprising: at least one memory and at least one processor;

[0074] The at least one memory is used to store a machine-readable program;

[0075] The at least one processor is used to call the machine-readable program to implement the above method.

[0076] The present invention also claims a computer-readable medium storing computer instructions that, when executed by a processor, enable the implementation of the above-described method.

[0077] Compared with existing technologies, the multi-dimensional large model testing and evaluation method and system of the present invention have the following advantages:

[0078] Achieving full-stack capability coverage: Pioneering a three-dimensional evaluation model encompassing basic capabilities, platform functions, and scenario performance, filling a gap in the industry. The basic capability layer adds a dual-engine security and compliance system (sensitive word database + semantic analysis), increasing the security interception rate to ≥99.5% (traditional solutions ≤95%). The platform function layer implements a three-stage quality control system for data feedback, improving training data quality by 35% (reducing the manual review error rate from 8% to 0.5%). The scenario performance layer establishes a robust attack matrix, improving model stability by 50% in noisy environments (accuracy fluctuation ≤10%).

[0079] Achieve deep support for domestic production: Define the standard for domestic GPU performance degradation rate (η=(T0-T) / T0): The degradation rate of Cambricon MLU370 inference task is optimized from 15% to 9.4%; compatible with domestic frameworks such as MindSpore / OneFlow, and the training convergence speed is improved by 40%.

[0080] Achieving a leap in engineering efficiency: intelligent annotation efficiency increased by 52.4%; model service deployment speed increased by 600%; and security detection latency decreased by 52.5%. Attached Figure Description

[0081] Figure 1 This is a diagram illustrating the architecture of the multi-dimensional large model testing and evaluation method provided in this embodiment of the invention;

[0082] Figure 2 This is a flowchart illustrating the process of achieving domestic production verification based on GPU attenuation rate, provided in an embodiment of the present invention.

[0083] Figure 3 This is a flowchart illustrating intelligent annotation based on a confidence level model, provided in an embodiment of the present invention.

[0084] Figure 4 This is a diagram illustrating the three-stage quality control process for data feedback provided in an embodiment of the present invention. Detailed Implementation

[0085] The present invention will be further described below with reference to specific embodiments.

[0086] This invention provides a multi-dimensional large-scale model testing and evaluation method that integrates the basic model, platform capabilities, and application performance to achieve multi-dimensional large-scale model testing and evaluation, including:

[0087] Evaluation of basic model capabilities, including security detection using a dual-engine approach of sensitive word library + BERT semantics (interception rate ≥ 99.5%) and context dependency test chain;

[0088] The evaluation of the model platform's capabilities includes pre-built LLM intelligent annotation (52% efficiency improvement) and two-person back-to-back data feedback;

[0089] Application performance evaluation includes character perturbation robustness testing and LIME interpretability rating assessment.

[0090] Large-scale full-stack testing is achieved through a RESTful heterogeneous model management interface, and domestic compatibility and adaptation verification is realized based on GPU decay rate quantization.

[0091] This method addresses the lack of evaluation of platform engineering capabilities, security compliance, and domestic support in existing technologies by establishing an evaluation system covering the entire chain of large-scale model development, deployment, and application.

[0092] The specific framework of the three-dimensional evaluation system is as follows:

[0093] I. Basic model capability assessment, specifically including:

[0094] The general ability assessment includes eight categories of general ability tests: language comprehension, logical reasoning, mathematical ability, knowledge question answering, code generation, multi-turn dialogue, text generation, entity recognition, and information extraction.

[0095] Security and compliance assessment is achieved through a dual-engine approach combining a sensitive word database and BERT semantics.

[0096] Evaluation of domestic production support, including GPU attenuation rate and compatibility testing.

[0097] Model inference performance evaluation includes the maximum number of inputs supported by the prompt, the maximum number of output tokens supported, and the latency per token.

[0098] II. Evaluation of model platform capabilities, including:

[0099] Intelligent labeling (confidence level classification);

[0100] Data feedback (three-stage control);

[0101] Heterogeneous management (RESTful API).

[0102] III. Performance evaluation of application scenarios, including:

[0103] Robustness (tested against 6 types of attacks);

[0104] Explainability (Level 5 standard).

[0105] Achieve a closed-loop evaluation across the entire chain, from model development to platform deployment to scenario application.

[0106] The basic model capability evaluation technology includes the following:

[0107] 1. Dual-engine approach to safety and compliance;

[0108] The process defines a `security_check(text)` function to perform a three-level security check on the input text `text`. The core idea is a progressive approach: first, use efficient but relatively simple rules to block obviously violating content; then, use a complex semantic model to identify variant attacks and potential risks; and finally, perform domain-specific compliance checks. The process is as follows:

[0109] First layer: Keyword Matching.

[0110] The function first matches the input text (text) against a predefined thesaurus (keyword_db) containing 10,000+ politically sensitive words.

[0111] Result judgment:

[0112] If the text contains any sensitive words from the dictionary, the function immediately returns "Blocked". This is the fastest and most efficient interception layer, primarily used to handle explicit content that clearly violates the rules.

[0113] The second layer: BERT Semantic Risk Analysis.

[0114] If the text passes the first layer of sensitive word check (i.e. no sensitive words are matched), it enters a deeper semantic analysis layer.

[0115] A pre-trained BERT model (bert_risk_predict(text)) is used to perform semantic understanding of the text and calculate a risk score (risk_score). This score quantifies the potential security risks of the text (such as implicit semantics like discrimination and misleading attacks).

[0116] Result judgment:

[0117] If the calculated risk score (risk_score) is greater than the preset threshold of 0.7, the function returns "block". This layer is a key innovation; it can capture content that does not use sensitive words but is semantically dangerous (such as variant attacks and cryptic expressions).

[0118] The third layer: TC260 Special Compliance Check.

[0119] If the text also passes the second-level semantic risk analysis (i.e., risk score <= 0.7), it enters the domain-specific compliance check layer.

[0120] Call the function check_tc260_compliance(text), which is specifically designed to check whether the text involves sensitive areas that are of particular concern in the TC260 (National Information Security Standardization Technical Committee) specification, such as biometric information (fingerprints, faces, irises, etc.).

[0121] Result judgment:

[0122] If the text triggers these specific compliance requirements (such as containing unauthorized biometric information or medical data), the function returns "Access Required." This means that the content itself is not necessarily harmful, but additional permissions or authorization are required to access or process it.

[0123] Passed safely:

[0124] If the input text successfully passes all three layers of checks (no sensitive words triggered, low semantic risk score, and no TC260 compliance items requiring special authorization), the function will ultimately return "pass". This indicates that the text is considered safe under the current security and compliance framework.

[0125] The process code is as follows:

[0126] defsecurity_check(text):

[0127] # Sensitive word matching (10,000+ politically sensitive words)

[0128] ifmatch_keyword(text, keyword_db):

[0129] return "intercept"

[0130] # BERT Semantic Risk Analysis

[0131] risk_score = bert_risk_predict(text)

[0132] if risk_score > 0.7: # Risk threshold

[0133] return "intercept"

[0134] # TC260 Special Testing

[0135] if check_tc260_compliance(text): # Bioinformatics / Medical Data Project

[0136] Return "Access requires authorization"

[0137] return "Passed".

[0138] 2. Domestic Production Verification:

[0139] This innovative verification process is based on the attenuation rate formula: η=(T_A100-T_Domestic) / T_A100×100%.

[0140] (1) Homogeneous deployment: A100 and domestic chips (Cambricon / Ascend) run in parallel on equal hardware, with consistent containerized control environment.

[0141] (2) Scenario-based load testing:

[0142] High concurrency throughput (measured by η_throughput);

[0143] Long context latency (measured η_latency);

[0144] Mixed load disturbances (measuring SLO violation rate).

[0145] (3) Dynamic optimization: When η>threshold, the computation graph is rewritten + mixed precision scheduling + memory optimization is triggered to iteratively reduce the decay rate (e.g., MLU370 from 15%→9.4%).

[0146] (4) Flexible judgment: Set dynamic thresholds according to task type (real-time dialogue η≤8%, batch generation η≤12%) and output microarchitecture bottleneck report.

[0147] The core innovation lies in transforming the general formula into a three-in-one verification chain of "scenario load - real-time optimization - dynamic threshold", breaking through the traditional extensive mode of fixed threshold.

[0148] like Figure 2 The figure shows a process example for verifying domestic production based on the attenuation rate formula.

[0149] The compatibility requirements are shown in Table 1 below:

[0150] Table 1 Compatibility Requirements

[0151] .

[0152] The technical implementation of the model platform capabilities includes the following:

[0153] 1. Intelligent annotation based on confidence level model:

[0154] The text is input into the BERT encoder, feature vectors are extracted, and confidence levels are predicted based on these feature vectors. If the confidence level is greater than or equal to a set threshold (0.9), it is automatically stored in the database; otherwise, it is placed in a manual review queue. Figure 3 As shown in Table 2 below, the efficiency comparison is as follows:

[0155] Table 2 Comparison of Smart Labeling Efficiency

[0156] .

[0157] 2. Data feedback based on three-stage control:

[0158] like Figure 4 The manual review process involves first capturing high-confidence data; then conducting double manual review on data with a confidence level greater than 0.85; if the consistency test Kappa value is greater than or equal to 0.75, the data is directly entered into the database; otherwise, it is entered into the database after expert arbitration.

[0159] Among them, the confidence threshold of 0.85 is the Pareto optimal solution for quality / efficiency (reducing the proportion of manual intervention to 28% and achieving a 52.4% efficiency improvement); the Kappa threshold of 0.75 is the statistical strong consistency boundary (the upper limit of the "Substantial Agreement" in the Landis & Koch standard), ensuring that the error rate of the reflux data is ≤0.5%.

[0160] The application performance (scenario capability) evaluation, technically implemented, includes the following:

[0161] 1. Robustness matrix testing, including:

[0162] For text perturbation attacks, the attack should be based on random character substitution (15%), and a semantic similarity of ≥90% is acceptable.

[0163] For image-based adversarial attacks, a classification accuracy of ≥85% is acceptable if the attack is based on the FGSM attack technique (ε=0.05).

[0164] For member reasoning attacks, implemented using shadow model attack techniques, a privacy leakage rate of ≤3% is acceptable.

[0165] 2. Interpretability grading assessment:

[0166] (1) Input: Obtain the LIME heatmap generated by the model for a certain prediction. The heatmap identifies the influence weight of the input features on the prediction result in matrix form (value range [-1,1]).

[0167] (2) Calculate the proportion of explainable features:

[0168] The percentage of significant positive features is calculated based on explainable_ratio = np.sum(heatmap>0.2) / heatmap.size.

[0169] Among them, heatmap>0.2 is used to filter weakly correlated features (only retaining features with significant positive contributions and avoiding noise interference).

[0170] 3. The five-level grading system is shown in Table 3 below:

[0171] Table 3 Five-level classification judgment

[0172]

[0173] Among them, Level 2 (unexplainable) directly triggers the model interception mechanism (such as mandatory circuit breaker in high-risk scenarios such as medical / financial); Levels 3-5 output interpretable reports to guide optimization decisions (such as attention mechanism adjustment).

[0174] The process code is as follows:

[0175] defexplainability_score(heatmap):

[0176] "A 5-level scoring system based on LiIME heatmaps"

[0177] # Calculate the proportion of explainable features

[0178] explainable_ratio = np.sum(heatmap>0.2) / heatmap.size

[0179] ifexplainable_ratio>= 0.9:

[0180] return 5 # Completely in line with human logic

[0181] elifexplainable_ratio>= 0.7:

[0182] return 4 # Key features are explainable

[0183] elifexplainable_ratio>= 0.5:

[0184] return 3 # Some features can be explained

[0185] else:

[0186] return 2 # Unexplainable (Do Not Go Online)

[0187] This method achieves full-stack capability coverage: it pioneers a three-dimensional evaluation model encompassing basic capabilities, platform functions, and scenario performance, filling a gap in the industry. The basic capability layer adds a dual-engine security and compliance system (sensitive word database + semantic analysis), increasing the security interception rate to ≥99.5% (compared to ≤95% for traditional solutions); the platform function layer implements a three-stage quality control system for data feedback, improving training data quality by 35% (reducing the manual review error rate from 8% to 0.5%); and the scenario performance layer establishes a robust attack matrix, improving model stability by 50% in noisy environments (accuracy fluctuation ≤10%).

[0188] Achieve deep support for domestic production: Define the standard for domestic GPU performance degradation rate (η=(T0-T) / T0): The degradation rate of Cambricon MLU370 inference task is optimized from 15% to 9.4%; compatible with domestic frameworks such as MindSpore / OneFlow, and the training convergence speed is improved by 40%.

[0189] The results demonstrate a significant leap in engineering efficiency, as shown in Table 4 below:

[0190] Table 4 Leap in Engineering Efficiency

[0191] .

[0192] The technical indicators are compared in Table 5 below:

[0193] Table 5 Comparison of Technical Indicators

[0194] .

[0195] The implementation of this method has enhanced the following application value:

[0196] A standardized evaluation system outputs machine-readable evaluation reports (JSON format) and supports multi-dimensional comparisons.

[0197] (1) Data Acquisition: Real-time acquisition of quantitative results of the three-dimensional evaluation system (basic capabilities / platform capabilities / scenario performance), for example:

[0198] Basic competency level: Language comprehension score 92.3%, security compliance interception rate 99.7%;

[0199] Platform capability layer: Data feedback quality inspection score 96 / 100;

[0200] Scenario performance layer: robustness attack pass rate, interpretability level, etc.

[0201] (2) Structured packaging:

[0202] The report is automatically generated using the following machine-readable JSON schema:

[0203] {

[0204] "Basic Abilities": {

[0205] Language Comprehension: 92.3, / / Percentage score

[0206] "Security and Compliance": {

[0207] "Interception Rate": 99.7%, / / Actual measured value of dual-engine security system

[0208] "TC260 Compliance Items": ["Bioinformatics", "Medical Authorization"] / / Specific Test Results

[0209] }

[0210] },

[0211] Platform capabilities: {

[0212] "Data feedback quality inspection score": 96 / / 100-point scale (out of 100)

[0213] },

[0214] / / Other dimensions of data...

[0215] }

[0216] (3) Dynamic field expansion:

[0217] Fields are automatically added or removed based on the test project (e.g., if "code generation" is not tested, this field will not exist) to avoid redundant data.

[0218] This invention also provides a multi-dimensional large model testing and evaluation system, including:

[0219] The basic model capability evaluation module includes a sensitive word library + BERT semantic dual-engine security detection and a context dependency test chain;

[0220] The model platform capability evaluation module includes pre-set LLM intelligent annotation and two-person back-to-back data feedback;

[0221] The application performance evaluation module includes character perturbation robustness testing and LIME interpretability rating assessment;

[0222] This system achieves multi-dimensional large model testing and evaluation through the multi-dimensional large model testing and evaluation method described in the above embodiments.

[0223] I. The basic model capability evaluation module specifically includes:

[0224] The general ability assessment includes eight categories of general ability tests: language comprehension, logical reasoning, mathematical ability, knowledge question answering, code generation, multi-turn dialogue, text generation, entity recognition, and information extraction.

[0225] Security and compliance assessment is achieved through a dual-engine approach combining a sensitive word database and BERT semantics.

[0226] Evaluation of domestic production support, including GPU attenuation rate and compatibility testing.

[0227] Model inference performance evaluation includes the maximum number of inputs supported by the prompt, the maximum number of output tokens supported, and the latency per token.

[0228] Its technical implementation includes the following:

[0229] 1. Dual-engine approach to safety and compliance;

[0230] The process defines a `security_check(text)` function to perform a three-level security check on the input text `text`. The core idea is a progressive approach: first, use efficient but relatively simple rules to block obviously violating content; then, use a complex semantic model to identify variant attacks and potential risks; and finally, perform domain-specific compliance checks. The process is as follows:

[0231] First layer: Keyword Matching.

[0232] The function first matches the input text (text) against a predefined thesaurus (keyword_db) containing 10,000+ politically sensitive words.

[0233] Result judgment:

[0234] If the text contains any sensitive words from the dictionary, the function immediately returns "Blocked". This is the fastest and most efficient interception layer, primarily used to handle explicit content that clearly violates the rules.

[0235] The second layer: BERT Semantic Risk Analysis.

[0236] If the text passes the first layer of sensitive word check (i.e. no sensitive words are matched), it enters a deeper semantic analysis layer.

[0237] A pre-trained BERT model (bert_risk_predict(text)) is used to perform semantic understanding of the text and calculate a risk score (risk_score). This score quantifies the potential security risks of the text (such as implicit semantics like discrimination and misleading attacks).

[0238] Result judgment:

[0239] If the calculated risk score (risk_score) is greater than the preset threshold of 0.7, the function returns "block". This layer is a key innovation; it can capture content that does not use sensitive words but is semantically dangerous (such as variant attacks and cryptic expressions).

[0240] The third layer: TC260 Special Compliance Check.

[0241] If the text also passes the second-level semantic risk analysis (i.e., risk score <= 0.7), it enters the domain-specific compliance check layer.

[0242] Call the function check_tc260_compliance(text), which is specifically designed to check whether the text involves sensitive areas that are of particular concern in the TC260 (National Information Security Standardization Technical Committee) specification, such as biometric information (fingerprints, faces, irises, etc.).

[0243] Result judgment:

[0244] If the text triggers these specific compliance requirements (such as containing unauthorized biometric information or medical data), the function returns "Access Required." This means that the content itself is not necessarily harmful, but additional permissions or authorization are required to access or process it.

[0245] Passed safely:

[0246] If the input text successfully passes all three layers of checks (no sensitive words triggered, low semantic risk score, and no TC260 compliance items requiring special authorization), the function will ultimately return "pass". This indicates that the text is considered safe under the current security and compliance framework.

[0247] 2. Domestic Production Verification:

[0248] This innovative verification process is based on the attenuation rate formula: η=(T_A100-T_Domestic) / T_A100×100%.

[0249] (1) Homogeneous deployment: A100 and domestic chips (Cambricon / Ascend) run in parallel on equal hardware, with consistent containerized control environment.

[0250] (2) Scenario-based load testing:

[0251] High concurrency throughput (measured by η_throughput);

[0252] Long context latency (measured η_latency);

[0253] Mixed load disturbances (measuring SLO violation rate).

[0254] (3) Dynamic optimization: When η>threshold, the computation graph is rewritten + mixed precision scheduling + memory optimization is triggered to iteratively reduce the decay rate (e.g., MLU370 from 15%→9.4%).

[0255] (4) Flexible judgment: Set dynamic thresholds according to task type (real-time dialogue η≤8%, batch generation η≤12%) and output microarchitecture bottleneck report.

[0256] The core innovation lies in transforming the general formula into a three-in-one verification chain of "scenario load - real-time optimization - dynamic threshold", breaking through the traditional extensive mode of fixed threshold.

[0257] Compatibility requirements include:

[0258] For the MindSpore domestic framework, its distributed training capability is tested. If the convergence time of 8 nodes is ≤4h, the acceptance test is passed.

[0259] For the OneFlow domestic framework, if its ability to convert between dynamic and static graphs is tested and the accuracy loss is ≤1%, then it passes acceptance.

[0260] II. The model platform capability evaluation module includes:

[0261] Intelligent labeling (confidence level classification);

[0262] Data feedback (three-stage control);

[0263] Heterogeneous management (RESTful API).

[0264] The technical implementation of the model platform capabilities includes the following:

[0265] 1. Intelligent annotation based on confidence level model:

[0266] The text is input into the BERT encoder, feature vectors are extracted, and confidence levels are predicted based on these feature vectors. If the confidence level is greater than or equal to a set threshold (0.9), it is automatically stored in the database; otherwise, it is placed in a manual review queue. Figure 3 As shown.

[0267] 2. Data feedback based on three-stage control:

[0268] like Figure 4 The manual review process involves first capturing high-confidence data; then conducting double manual review on data with a confidence level greater than 0.85; if the consistency test Kappa value is greater than or equal to 0.75, the data is directly entered into the database; otherwise, it is entered into the database after expert arbitration.

[0269] Among them, the confidence threshold of 0.85 is the Pareto optimal solution for quality / efficiency (reducing the proportion of manual intervention to 28% and achieving a 52.4% efficiency improvement); the Kappa threshold of 0.75 is the statistical strong consistency boundary (the upper limit of the "Substantial Agreement" in the Landis & Koch standard), ensuring that the error rate of the reflux data is ≤0.5%.

[0270] III. Application performance evaluation module, including:

[0271] Robustness, tested against 6 types of attacks;

[0272] Explainability, Level 5 standard.

[0273] Achieve a closed-loop evaluation across the entire chain, from model development to platform deployment to scenario application.

[0274] Application performance (scenario capability) evaluation, the technical implementation includes the following:

[0275] 1. Robustness matrix testing, including:

[0276] For text perturbation attacks, the attack should be based on random character substitution (15%), and a semantic similarity of ≥90% is acceptable.

[0277] For image-based adversarial attacks, a classification accuracy of ≥85% is acceptable if the attack is based on the FGSM attack technique (ε=0.05).

[0278] For member reasoning attacks, implemented using shadow model attack techniques, a privacy leakage rate of ≤3% is acceptable.

[0279] 2. Interpretability grading assessment:

[0280] (1) Input: Obtain the LIME heatmap generated by the model for a certain prediction. The heatmap identifies the influence weight of the input features on the prediction result in matrix form (value range [-1,1]).

[0281] (2) Calculate the proportion of explainable features:

[0282] The percentage of significant positive features is calculated based on explainable_ratio = np.sum(heatmap>0.2) / heatmap.size.

[0283] Among them, heatmap>0.2 is used to filter weakly correlated features (only retaining features with significant positive contributions and avoiding noise interference).

[0284] 3. Five-level grading determination:

[0285] If the proportion of explainable features is ≥0.9, the judgment level is 5, the semantic description is completely in line with human logic, and the business can be directly deployed;

[0286] If the proportion of explainable features is ≥0.7, the judgment level is 4, the semantic description is that the key features are explainable, and the business is recommendation deployment;

[0287] If the proportion of explainable features is ≥0.5, the judgment level is 3, and the semantic description is that some features are explainable, and the business needs to supplement the explanation document;

[0288] If the proportion of explainable features is less than 0.5, the judgment level is 2, the semantic description is unexplainable, and the business is prohibited from going live.

[0289] Among them, Level 2 (unexplainable) directly triggers the model interception mechanism (such as mandatory circuit breaker in high-risk scenarios such as medical / financial); Levels 3-5 output interpretable reports to guide optimization decisions (such as attention mechanism adjustment).

[0290] This invention also provides a multi-dimensional large model testing and evaluation device, comprising: at least one memory and at least one processor;

[0291] The at least one memory is used to store a machine-readable program;

[0292] The at least one processor is used to call the machine-readable program to implement the multi-dimensional large model testing and evaluation method described in the above embodiments.

[0293] This invention also provides a computer-readable medium storing computer instructions. When executed by a processor, the computer instructions cause the processor to perform the multi-dimensional large model testing and evaluation method described in the above embodiments. Specifically, a system or apparatus equipped with a storage medium storing software program code that implements the functions of any of the above embodiments can be provided, and the computer (or CPU or MPU) of the system or apparatus can read and execute the program code stored in the storage medium.

[0294] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0295] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0296] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0297] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0298] The present invention has been shown and described in detail above with reference to the accompanying drawings and preferred embodiments. However, the present invention is not limited to these disclosed embodiments. Based on the above embodiments, those skilled in the art will know that more embodiments of the present invention can be obtained by combining the code review methods in the different embodiments. These embodiments are also within the protection scope of the present invention.

Claims

1. A multi-dimensional large model testing and evaluation method, characterized in that, Integrating basic models, platform capabilities, and application performance to achieve multi-dimensional large-scale model testing and evaluation, including: Evaluation of basic model capabilities, including sensitive word library + BERT semantic dual-engine security detection and context dependency test chain; Model platform capability evaluation, including pre-built LLM intelligent annotation and two-person back-to-back data feedback; Application performance evaluation includes character perturbation robustness testing and LIME interpretability rating assessment; Large-scale full-stack testing of models is achieved through a RESTful heterogeneous model management interface, and domestic compatibility and adaptation verification is achieved based on GPU decay rate quantization. The security compliance detection based on a sensitive word database and BERT semantic dual engine is implemented by defining a function `security_check(text)` to perform progressive security checks on the input text `text`. The specific implementation includes: First layer: Sensitive word database matching: The function first matches the input text against a predefined thesaurus containing 10,000+ politically sensitive words; Result judgment: If the text contains any sensitive words from the dictionary, the function immediately returns "intercept"; Second layer: BERT semantic risk analysis: If the text passes the first layer of sensitive word detection, it enters the semantic analysis layer; The text is semantically understood using a pre-trained BERT model, and a risk score is calculated; this score is used to quantify the potential security risks of the text. Result judgment: If the calculated risk score is greater than the preset threshold, the function returns "block". Third layer: TC260 specific compliance testing: If the text passes the second-level semantic risk analysis, it enters the compliance check layer for the specific domain. Call the function check_tc260_compliance(text), which is used to specifically check whether the text involves sensitive areas of particular concern in the TC260 specification; Result determination: If the text triggers the compliance requirements of the aforementioned sensitive area, the function returns "Access requires authorization"; Passed safely: If the input text successfully passes all three layers of checks, the function will eventually return "pass", indicating that the text is considered safe under the current security and compliance framework. The evaluation of the model platform's capabilities specifically includes: (1) Intelligent annotation based on confidence level model: The text is input into the BERT encoder, feature vectors are extracted, and confidence is predicted based on the feature vectors. If the confidence value is greater than or equal to a set threshold, it is automatically stored in the database; otherwise, it is included in the manual review queue. (2) Data feedback is achieved based on three-stage control: The manual review process first captures high-confidence data; data with a confidence level greater than the confidence threshold undergoes double manual review; if the consistency test Kappa value is greater than or equal to the Kappa threshold, the data is directly entered into the database; otherwise, it is entered into the database after expert arbitration. The application performance evaluation specifically includes: (1) Robustness matrix test, including: For text perturbation attacks, those based on random character substitution are considered acceptable if the semantic similarity is ≥90%. For image-based adversarial attacks, those based on FGSM attack techniques are considered acceptable if the classification accuracy is ≥85%. For member reasoning attacks, implemented using shadow model attack techniques, a privacy leakage rate of ≤3% is acceptable. (2) Interpretability grading assessment: (2.1) Input: Obtain the LIME heatmap generated by the model for a certain prediction. The heatmap identifies the weights of the input features on the prediction results in matrix form. (2.2) Calculate the proportion of explainable features: The formula `explainable_ratio=np.sum(heatmap > 0.2) / heatmap.size` is used to calculate the proportion of significant positive features. Among them, heatmap > 0.2 is used to filter weakly correlated features; (3) Five-level grading determination: If the proportion of explainable features is ≥0.9, the judgment level is 5, the semantic description is completely in line with human logic, and the business can be directly deployed; If the proportion of explainable features is ≥0.7, the judgment level is 4, the semantic description is that the key features are explainable, and the business is recommendation deployment; If the proportion of explainable features is ≥0.5, the judgment level is 3, and the semantic description is that some features are explainable, and the business needs to supplement the explanation document; If the proportion of explainable features is less than 0.5, the judgment level is 2, the semantic description is unexplainable, and the business is prohibited from going live. Level 2 directly triggers the model interception mechanism; Levels 3-5 output interpretable reports to guide optimization decisions.

2. The multi-dimensional large model testing and evaluation method according to claim 1, characterized in that, The evaluation of the basic model's capabilities specifically includes: General ability assessment includes tests of language comprehension, logical reasoning, mathematical ability, knowledge-based question answering, code generation, multi-turn dialogue, text generation, entity recognition, and information extraction. Security compliance assessment is achieved through a dual-engine approach combining a sensitive word database and BERT semantic analysis. Evaluation of domestic production support, including GPU attenuation rate and compatibility testing; Model inference performance evaluation includes the maximum number of inputs supported by the prompt, the maximum number of output tokens supported, and the latency per token.

3. A multi-dimensional large model testing and evaluation method according to claim 1 or 2, characterized in that, For verification of domestic production support, it is based on GPU attenuation rate, and the formula for GPU attenuation rate is as follows: η = (T_A100 - T_Domestic) / T_A100 × 100%; T_A100: refers to the time taken to run a specific task on an NVIDIA A100 GPU. T_Domestic: refers to the time spent running the same task on a domestically produced GPU; The verification process includes: (1) Homogeneous deployment: A100 and domestic chips run in parallel under the same amount of hardware, and the containerized control environment is consistent; (2) Scenario-based load testing, including: High concurrency throughput, measure η_throughput; Long context latency, measure η_latency; Measure SLO violation rate under mixed load disturbances; (3) Dynamic optimization: Triggered when η>threshold: computation graph rewriting + mixed precision scheduling + memory optimization, to iteratively reduce the decay rate; (4) Flexible judgment: Set dynamic thresholds according to task type and output microarchitecture bottleneck report.

4. The multi-dimensional large model testing and evaluation method according to claim 2, characterized in that, The evaluation of domestic production support and compatibility testing includes: For the MindSpore domestic framework, its distributed training capability is tested. If the convergence time of 8 nodes is ≤4h, the acceptance test is passed. For the OneFlow domestic framework, if its ability to convert between dynamic and static graphs is tested and the accuracy loss is ≤1%, then it passes acceptance.

5. A multi-dimensional large-scale model testing and evaluation system, characterized in that, include: The basic model capability evaluation module includes a sensitive word library + BERT semantic dual-engine security detection and a context dependency test chain; The model platform capability evaluation module includes pre-set LLM intelligent annotation and two-person back-to-back data feedback; The application performance evaluation module includes character perturbation robustness testing and LIME interpretability rating assessment; The system achieves multi-dimensional large model testing and evaluation through the method described in any one of claims 1 to 4.

6. A multi-dimensional large-scale model testing and evaluation device, characterized in that, include: At least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to invoke the machine-readable program to implement the method according to any one of claims 1 to 4.

7. A computer-readable medium, characterized in that, The computer-readable medium stores computer instructions that, when executed by a processor, enable the implementation of the method described in any one of claims 1 to 4.

Images