Large Model Automated Type Recognition and Full Index Evaluation System

Abstract

This invention discloses a large-scale automated type identification and full-index evaluation system, belonging to the field of artificial intelligence model evaluation technology. The technical problem this invention aims to solve is how to achieve automated model type identification, full-index evaluation in a single step, and automated output of unified results. The technical solution includes a user interaction layer and a core functional layer. The user interaction layer is used to access the model under test, fill in preset information, and display reports. The core functional layer is a layered architecture composed of an interface adaptation layer, a type identification engine, an index engine cluster, a result aggregation, and a report generator. The interface adaptation layer acts as a bridge between the model under test and the user interaction layer, supporting multiple access methods and completing format conversion through a framework adaptation plugin. The type identification engine is used to automatically determine the type of the model under test. The index engine cluster is used to schedule evaluation tasks in the corresponding domain based on the category label of the model under test determined by the type identification engine.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model evaluation technology, specifically a large-scale model automated type identification and full-index evaluation system. Background Technology

[0002] Current large-scale model evaluation mainly relies on frameworks such as HELM, OpenCompass, and Eleuther Harness. Although these frameworks provide basic testing tasks, they have the following key shortcomings:

[0003] 1) High cost of manual intervention: Existing frameworks require manual specification of model type (such as "language model" or "image model") and manual configuration of test scripts (such as selecting MMLU for language model). Taking multimodal model evaluation as an example, language test scripts and image test scripts need to be called separately. The switching process takes 2-3 hours and is prone to result deviation due to differences in parameter settings.

[0004] 2) Incomplete indicator coverage: Most current large-scale model evaluation tools focus on a single capability dimension (e.g., HELM focuses on language understanding, and OpenCompass focuses on general capabilities), making it difficult to simultaneously cover engineering indicators such as safety ethics (e.g., bias detection) and system efficiency (e.g., memory usage). For example, when evaluating a large dialogue model, it is necessary to call third-party tools (e.g., Hugging Face Safety Checker) to detect security risks, resulting in a fragmented evaluation process.

[0005] 3) Inconsistent result formats: The output formats of different indicators vary greatly (e.g., accuracy is numerical, security detection is text labels, and efficiency indicators are time series), which requires manual processing and seriously affects the evaluation efficiency.

[0006] Therefore, how to achieve automated identification of model types, one-time evaluation of all indicators, and automated evaluation of outputting unified results is a technical problem that urgently needs to be solved. Summary of the Invention

[0007] The technical objective of this invention is to provide a large-scale automated type identification and full-index evaluation system to address the problem of how to achieve automated identification of model types, one-time evaluation of all indicators, and automated evaluation of outputting unified results.

[0008] The technical task of this invention is achieved in the following way: a large model automated type recognition and full index evaluation system, which includes a user interaction layer and a core function layer. The user interaction layer is used to access the model to be tested, fill in preset information and display reports. The core function layer is a layered architecture composed of an interface adaptation layer, a type recognition engine, an index engine cluster, a result aggregation and report generator.

[0009] Among them, the interface adaptation layer serves as a bridge connecting the model under test and the user interaction layer, supporting multiple access methods, and completing format conversion through the framework adaptation plugin. It also supports offline evaluation.

[0010] The type recognition engine is used to automatically determine the type of the model under test;

[0011] The indicator engine cluster is used to schedule evaluation tasks in the corresponding domain based on the category label of the model under test determined by the type recognition engine.

[0012] The results aggregation and report generator is used to uniformly process and output the test results of the category recognition engine and the indicator engine cluster.

[0013] Preferred access methods include HTTP access, gRPC access, and local file access;

[0014] HTTP access refers to communicating with cloud-based model services (such as the OpenAI API) via a RESTful API;

[0015] gRPC access uses a binary protocol to interact with models within the local area network or LAN.

[0016] Local file access refers to directly parsing locally stored model weight files (formats such as .pt, .ckpt, .bin, etc.).

[0017] As a preferred option, the framework adapter plugin uses built-in PyTorch and TensorFlow converters.

[0018] As a preferred embodiment, the type recognition engine includes a structural fingerprint extractor, a behavioral feature capturer, and a type classifier;

[0019] Among them, the structural fingerprint extractor is used to deeply parse the configuration file (such as config.json) and layer type (Transformer layer, CNN layer, RNN layer) of the model under test, and generate a 128-dimensional structural fingerprint;

[0020] The behavior feature capturer is used to input preset probe data into the model under test and capture the features of the output results. The preset probe data includes text data, code data, mathematical formula data and image data. A 64-dimensional behavior feature vector is constructed using the word vector distribution output from the text data, the syntax tree structure output from the code data and the feature mapping matrix output from the image data.

[0021] The type classifier is used to classify the 192-dimensional joint features of "128-dimensional structural fingerprint + 64-dimensional behavioral features" in a pre-trained multi-label classification network with ResNet-18 architecture, and outputs deep learning model category labels; among them, the model category labels include language model labels, multimodal model labels, and code model labels.

[0022] As a preferred option, the indicator engine cluster includes a language understanding engine, an inference engine, a multimodal engine, a security and ethics engine, and a system efficiency engine;

[0023] Among them, the language understanding engine covers general language understanding based on the MMLU model, Chinese language ability based on the C-EVAL model, and advanced semantic understanding based on the SuperGLUE model.

[0024] The inference engine covers data inference based on the GSM8K problem dataset and code generation based on the MBPP programming benchmark, and supports automatic code execution and result verification.

[0025] The multimodal engine covers text-image interaction based on VQA visual question answering evaluation, description consistency based on CIDEr and image generation quality based on FID similarity evaluation with real images, cross-modal matching based on CLIPScore text-image matching evaluation, and supports image input with resolutions from 256×256 to 1024×1024.

[0026] The security ethics engine covers bias detection, content security, and adversarial robustness;

[0027] The system efficiency engine covers end-to-end latency, first packet latency, effective throughput, batch processing efficiency, memory fragmentation rate, power consumption-performance ratio, and cold start time.

[0028] More specifically, end-to-end latency refers to the total time from when a user request is sent to when the last token is returned; first packet latency refers to the network + scheduling + loading latency before the first token; effective throughput refers to the actual number of tokens output per GPU per hour after deducting retries and failed requests; batch processing efficiency refers to the used sequence length / maximum sequence length, used to measure fill waste; memory fragmentation rate refers to the proportion of idle but unallocatable memory blocks; power consumption-performance ratio refers to the effective throughput per watt of power consumption; cold start time refers to the time from pulling the image to when the first request that the model under test can respond to is available.

[0029] As a preferred embodiment, the result aggregator and report generator includes a data processing module, a report output module, and an interface extension module;

[0030] The data processing module is used for alignment, unification, and ranking.

[0031] The report output module supports output in JSON, EXCEL and PDF formats, and outputs content including basic information of the model under test, scores of each dimension, analysis of strengths and weaknesses and optimization suggestions;

[0032] The interface extension module provides a RESTful API that allows external systems, including model management platforms and CI / CD pipelines, to directly access evaluation results.

[0033] More ideally, alignment refers to unifying the timestamps of different metrics and the metadata of the model version under test;

[0034] Standardization refers to mapping all indicators to a score of 0-100 through min-max standardization;

[0035] Ranking processing refers to comparing data with historical data of similar models under test to generate percentile rankings.

[0036] Ideally, JSON contains both the original data and the processed scores;

[0037] Excel includes pivot charts and supports manual analysis;

[0038] The PDF includes radar charts and bar charts for visualization.

[0039] As a preferred option, the working process of this system is as follows:

[0040] (1) User interaction page: It supports users to access the cloud model under test through three methods: HTTP interface, gRPC connection parameters or local file path, select the result output method, such as preset report output format, report push method, etc., and automatically record the name and version of the model under test.

[0041] (2) Interface adaptation: Call the parser of the corresponding access method to complete the identification, format conversion and loading of the model framework under test: If the conversion fails, the interface adaptation layer directly feeds back to the user interaction page and outputs the failure information; if the conversion is successful, the type recognition engine starter is called.

[0042] (3) Type Recognition: The type recognition engine starts and automatically completes structural fingerprint extraction, behavioral feature capture and classification, and outputs category labels. After the type recognition engine starter starts, it simultaneously calls the structural fingerprint extractor and the behavioral feature captureor. In the structural fingerprint extractor, the configuration file and layer type information of the model under test are parsed. If the parsing fails, the user interface failure information is returned. If the extraction is successful, 128-dimensional structural fingerprint data is generated and the data is passed to the type classifier. In the behavioral feature captureor, behavioral feature vectors are captured by inputting preset probe data. If the capture fails, the user interface failure information is returned. If the capture is successful, 64-dimensional behavioral feature vectors are generated and the data is passed to the type classifier. After receiving the data from the structural fingerprint extractor and the behavioral feature captureor, the type classifier starts to perform type classification. If the classification fails, the user interface failure information is returned. If the classification is completed, the category label of the model under test is returned and the index engine starter is called.

[0043] (4) Metric Engine Cluster: The metric engine cluster activates the corresponding task based on the category label (such as activating the language understanding, multimodal, security ethics, etc. engines in the multimodal model): if the result generation fails, the failure information of the user interaction interface is fed back; if the result generation is successful, the result data is fed back to the result aggregator and report generator.

[0044] (5) Results aggregation and report generation: The results aggregation and report generator receives real-time data from each engine, completes alignment, normalization and ranking, generates a comprehensive score, and provides feedback according to the preset report output format and push method.

[0045] The large-model automated type identification and full-index evaluation system of the present invention has the following advantages:

[0046] (I) This invention adopts a layered architecture design to achieve full-process automation from model access to report output. Specifically, it is compatible with three access methods: HTTP, gRPC, and local file through an interface adaptation layer, and supports large model loading for most mainstream architectures (PyTorch, TensorFlow, MindSpore, etc.); the type recognition engine is based on the dual dimensions of "structural fingerprint + behavioral features" to automatically determine the model category in a short time; for different types of models, the indicator engine cluster is dynamically called to cover multiple dimensions and indicators such as language understanding, mathematical reasoning, multimodal, and security ethics system efficiency at one time. After the evaluation, a unified format report is generated through result aggregation and report generator, realizing automated evaluation of model type, full indicator evaluation at one time, and output of unified results.

[0047] (II) This invention automates the entire process from model access to report output, completing multi-dimensional and multi-indicator evaluations such as language understanding, mathematical reasoning, multimodal processing, security ethics, and system efficiency in one go. Through versioned management of test case sets, test scripts, and hardware configurations, it ensures the reusability of the execution process and outputs reports in a unified format. At the same time, it supports adding indicators through a plugin mechanism without modifying the core code, adapting to the needs of rapid iteration of large model technology.

[0048] (III) This invention requires no manual intervention from the input of the model under test to the output of the report, which solves the operational costs of manually selecting indicators, editing scripts and selecting test case sets, while avoiding the result deviation caused by human operation, thus achieving full automation and reducing labor costs.

[0049] (iv) This invention covers multiple dimensions of indicators such as language, reasoning, multimodality, security, and efficiency in one go, solves the problem of indicator fragmentation, achieves full indicator coverage, and improves the completeness of evaluation;

[0050] (v) This invention ensures that the results are 100% reproducible by logging the test script version, random seed, and hardware configuration; the unified format report can be directly imported into the model management system without secondary processing, achieving reliable and reusable results and supporting engineering implementation.

[0051] (vi) The present invention adopts a plug-in design. Adding new indicators only requires developing a plug-in and registering it to the indicator engine cluster. No core code needs to be modified. It adapts to the rapid evolution of large model technology, achieves high scalability, and adapts to technological development. Attached Figure Description

[0052] The invention will be further described below with reference to the accompanying drawings.

[0053] Appendix Figure 1 A schematic diagram of the structure of a large-scale automated type identification and full-index evaluation system;

[0054] Appendix Figure 2 A flowchart illustrating the working process of a large-scale automated type identification and full-index evaluation system. Detailed Implementation

[0055] The large-model automated type identification and full-index evaluation system of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0056] Example 1:

[0057] As attached Figure 1As shown, this embodiment provides a large model automated type recognition and full index evaluation system. The system includes a user interaction layer and a core function layer. The user interaction layer is used to access the model to be tested, fill in preset information and display reports. The core function layer is a layered architecture composed of an interface adaptation layer, a type recognition engine, an index engine cluster, a result aggregation and report generator.

[0058] The interface adaptation layer serves as a bridge connecting the model under test and the user interaction layer. It supports multiple access methods and performs format conversion through the framework adaptation plugin. It also supports offline evaluation. The access methods include HTTP access, gRPC access, and local file access. HTTP access refers to communicating with cloud model services (such as OpenAI API) through a RESTful API. gRPC access uses a binary protocol to interact with models on the local machine or within a local area network. Local file access refers to directly parsing locally stored model weight files (.pt, .ckpt, .bin, etc.). The framework adaptation plugin uses built-in PyTorch and TensorFlow converters.

[0059] The type recognition engine is used to automatically determine the type of the model under test;

[0060] The indicator engine cluster is used to schedule evaluation tasks in the corresponding domain based on the category label of the model under test determined by the type recognition engine.

[0061] The results aggregation and report generator is used to uniformly process and output the test results of the category recognition engine and the indicator engine cluster.

[0062] The type recognition engine in this embodiment includes a structural fingerprint extractor, a behavioral feature capturer, and a type classifier;

[0063] Among them, the structural fingerprint extractor is used to deeply parse the configuration file (such as config.json) and layer type (Transformer layer, CNN layer, RNN layer) of the model under test, and generate a 128-dimensional structural fingerprint;

[0064] The behavior feature capturer is used to input preset probe data into the model under test and capture the features of the output results. The preset probe data includes text data, code data, mathematical formula data and image data. A 64-dimensional behavior feature vector is constructed using the word vector distribution output from the text data, the syntax tree structure output from the code data and the feature mapping matrix output from the image data.

[0065] The type classifier is used to classify the 192-dimensional joint features of "128-dimensional structural fingerprint + 64-dimensional behavioral features" in a pre-trained multi-label classification network with ResNet-18 architecture, and outputs deep learning model category labels; among them, the model category labels include language model labels, multimodal model labels, and code model labels.

[0066] The indicator engine cluster in this embodiment includes a language understanding engine, an inference engine, a multimodal engine, a security and ethics engine, and a system efficiency engine;

[0067] Among them, the language understanding engine covers general language understanding based on the MMLU model, Chinese language ability based on the C-EVAL model, and advanced semantic understanding based on the SuperGLUE model.

[0068] The inference engine covers data inference based on the GSM8K problem dataset and code generation based on the MBPP programming benchmark, and supports automatic code execution and result verification.

[0069] The multimodal engine covers text-image interaction based on VQA visual question answering evaluation, description consistency based on CIDEr and image generation quality based on FID similarity evaluation with real images, cross-modal matching based on CLIPScore text-image matching evaluation, and supports image input with resolutions from 256×256 to 1024×1024.

[0070] The security ethics engine covers bias detection, content security, and adversarial robustness;

[0071] The system efficiency engine covers end-to-end latency, first-packet latency, effective throughput, batch processing efficiency, memory fragmentation rate, power consumption-performance ratio, and cold start time. End-to-end latency refers to the total time from when a user request is sent to when the last token is returned. First-packet latency refers to the network, scheduling, and loading latency before the first token. Effective throughput refers to the actual number of tokens output per GPU per hour after deducting retries and failed requests. Batch processing efficiency is the ratio of used sequence length to maximum sequence length, used to measure fill waste. Memory fragmentation rate refers to the percentage of idle but unallocated memory blocks. Power consumption-performance ratio refers to the effective throughput per watt of power consumption. Cold start time refers to the time from pulling the image to when the model under test can respond to the first request.

[0072] The result aggregation and report generator in this embodiment includes a data processing module, a report output module, and an interface extension module;

[0073] The data processing module is used for alignment, unification, and ranking. Alignment involves unifying the timestamps of different indicators and the metadata of the model version under test. Unification involves mapping all indicators to a score of 0-100 through min-max standardization. Ranking involves comparing the data with historical data of similar models under test to generate percentile rankings.

[0074] The report output module supports output in JSON, EXCEL, and PDF formats, and outputs content including basic information of the model under test, scores of each dimension, analysis of strengths and weaknesses, and optimization suggestions; among them, JSON contains raw data and processed scores; EXCEL contains pivot charts and supports manual analysis; PDF contains radar charts and bar charts for visualization.

[0075] The interface extension module provides a RESTful API that allows external systems, including model management platforms and CI / CD pipelines, to directly access evaluation results.

[0076] As attached Figure 2 As shown, the specific working process of this system is as follows:

[0077] (1) User interaction page: It supports users to access the cloud model under test through three methods: HTTP interface, gRPC connection parameters or local file path, select the result output method, such as preset report output format, report push method, etc., and automatically record the name and version of the model under test.

[0078] (2) Interface adaptation: Call the parser of the corresponding access method to complete the identification, format conversion and loading of the model framework under test: If the conversion fails, the interface adaptation layer directly feeds back to the user interaction page and outputs the failure information; if the conversion is successful, the type recognition engine starter is called.

[0079] (3) Type Recognition: The type recognition engine starts and automatically completes structural fingerprint extraction, behavioral feature capture and classification, and outputs category labels. After the type recognition engine starter starts, it simultaneously calls the structural fingerprint extractor and the behavioral feature captureor. In the structural fingerprint extractor, the configuration file and layer type information of the model under test are parsed. If the parsing fails, the user interface failure information is returned. If the extraction is successful, 128-dimensional structural fingerprint data is generated and the data is passed to the type classifier. In the behavioral feature captureor, behavioral feature vectors are captured by inputting preset probe data. If the capture fails, the user interface failure information is returned. If the capture is successful, 64-dimensional behavioral feature vectors are generated and the data is passed to the type classifier. After receiving the data from the structural fingerprint extractor and the behavioral feature captureor, the type classifier starts to perform type classification. If the classification fails, the user interface failure information is returned. If the classification is completed, the category label of the model under test is returned and the index engine starter is called.

[0080] (4) Metric Engine Cluster: The metric engine cluster activates the corresponding task based on the category label (such as activating the language understanding, multimodal, security ethics, etc. engines in the multimodal model): if the result generation fails, the failure information of the user interaction interface is fed back; if the result generation is successful, the result data is fed back to the result aggregator and report generator.

[0081] (5) Results aggregation and report generation: The results aggregation and report generator receives real-time data from each engine, completes alignment, normalization and ranking, generates a comprehensive score, and provides feedback according to the preset report output format and push method.

[0082] Example 2:

[0083] To obtain an evaluation report covering multiple dimensions such as language, reasoning, multimodality, security, and efficiency for a multimodal dialogue model with 120 billion parameters within 2 hours, the following is a detailed description:

[0084] ① Connect the vendor's large model interface to the large model evaluation tool, select the report output format and report push method information, without configuring the evaluation script and test case set, and click submit;

[0085] ② After submission, the evaluation tool will automatically identify the type of large model access and automatically evaluate all indicators. The evaluation tool has built-in evaluation models and test case sets for language understanding, mathematical reasoning, multimodal processing, security ethics, system efficiency, etc., and the evaluation standards are executed automatically.

[0086] ③ After the evaluation is completed, an evaluation report will be output, which includes basic model information, scores for each dimension, analysis of strengths / weaknesses, and optimization suggestions;

[0087] ④ After the user makes the rectification, the evaluation can be performed again, and the evaluation version number can be automatically generated, supporting data comparison.

[0088] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large-scale model automated type recognition and full-index evaluation system, characterized in that, The system includes a user interaction layer and a core function layer. The user interaction layer is used to access the model under test, fill in preset information, and display reports. The core function layer is a layered architecture consisting of an interface adaptation layer, a type recognition engine, an indicator engine cluster, a result aggregation and report generator. Among them, the interface adaptation layer serves as a bridge connecting the model under test and the user interaction layer, supporting HTTP access, gRPC access and local file access, and completing format conversion through the framework adaptation plugin. It also supports offline evaluation. The type recognition engine is used to automatically determine the type of the model under test; The metrics engine cluster is used to activate corresponding tasks based on category labels: if the result generation fails, it will send a failure message to the user interaction layer; if the result generation succeeds, it will send the result data back to the result aggregator and report generator. The results aggregation and report generator is used to uniformly process and output the test results of the category recognition engine and the indicator engine cluster; The type recognition engine includes a structural fingerprint extractor, a behavioral feature capturer, and a type classifier. The structural fingerprint extractor is used to deeply analyze the configuration file and layer type of the model under test, and generate a 128-dimensional structural fingerprint; the layer types include Transformer layers, CNN layers, and RNN layers. The behavior feature capturer is used to input preset probe data into the model under test and capture the features of the output results. The preset probe data includes text data, code data, mathematical formula data and image data. A 64-dimensional behavior feature vector is constructed using the word vector distribution output from the text data, the syntax tree structure output from the code data and the feature mapping matrix output from the image data. The type classifier is used to classify the 192-dimensional joint features of "128-dimensional structural fingerprint + 64-dimensional behavioral features" in a pre-trained multi-label classification network with ResNet-18 architecture, and outputs deep learning model category labels; among them, the model category labels include language model labels, multimodal model labels, and code model labels. The metrics engine cluster includes a language understanding engine, an inference engine, a multimodal engine, a security and ethics engine, and a system efficiency engine; Among them, the language understanding engine covers general language understanding based on the MMLU model, Chinese language ability based on the C-EVAL model, and advanced semantic understanding based on the SuperGLUE model. The inference engine covers data inference based on the GSM8K problem dataset and code generation based on the MBPP programming benchmark, and supports automatic code execution and result verification. The multimodal engine covers text-image interaction based on VQA visual question answering evaluation, description consistency based on CIDEr and image generation quality based on FID similarity evaluation with real images, cross-modal matching based on CLIPScore text-image matching evaluation, and supports image input with resolutions from 256×256 to 1024×1024. The security ethics engine covers bias detection, content security, and adversarial robustness; The system efficiency engine covers end-to-end latency, first packet latency, effective throughput, batch processing efficiency, memory fragmentation rate, power consumption-performance ratio, and cold start time.

2. The large-scale model automated type identification and full-index evaluation system according to claim 1, characterized in that, HTTP access refers to communicating with cloud-based model services via a RESTful API; gRPC access uses a binary protocol to interact with models within the local area network or LAN. Local file access refers to directly parsing the model weight files stored locally.

3. The large-model automated type identification and full-index evaluation system according to claim 1, characterized in that, The framework adaptation plugin uses built-in PyTorch and TensorFlow converters.

4. The large-scale model automated type identification and full-index evaluation system according to claim 1, characterized in that, End-to-end latency refers to the total time from when a user request is sent to when the last token is returned; first packet latency refers to the network, scheduling, and loading latency before the first token; effective throughput refers to the actual number of tokens output per GPU per hour after deducting retries and failed requests; batch processing efficiency refers to the ratio of used sequence length to maximum sequence length, used to measure fill waste; memory fragmentation rate refers to the percentage of idle but unallocatable memory blocks; power consumption-performance ratio refers to the effective throughput per watt of power consumption; cold start time refers to the time from pulling the image to when the model under test can respond to the first request.

5. The large-model automated type identification and full-index evaluation system according to claim 1, characterized in that, The results aggregation and report generator includes a data processing module, a report output module, and an interface extension module; The data processing module is used for alignment, unification, and ranking. The report output module supports output in JSON, EXCEL and PDF formats, and outputs content including basic information of the model under test, scores of each dimension, analysis of strengths and weaknesses and optimization suggestions; The interface extension module provides a RESTful API that allows external systems, including model management platforms and CI / CD pipelines, to directly access evaluation results.

6. The large-model automated type identification and full-index evaluation system according to claim 5, characterized in that, Alignment processing refers to unifying the timestamps of different metrics and the metadata of the model version under test; Standardization refers to mapping all indicators to a score of 0-100 through min-max standardization; Ranking processing refers to comparing data with historical data of similar models under test to generate percentile rankings.

7. The large-model automated type identification and full-index evaluation system according to claim 5, characterized in that, The JSON contains the original data and the processed scores; Excel includes pivot charts and supports manual analysis; The PDF includes radar charts and bar charts for visualization.

8. The large-model automated type identification and full-index evaluation system according to claim 1, characterized in that, The working process of this system is as follows: (1) User interaction interface: It supports users to access the cloud-based model under test via HTTP interface, gRPC connection parameters or local file path, select the result output method, and automatically record the name and version of the model under test; (2) Interface adaptation: Call the parser of the corresponding access method to complete the identification, format conversion and loading of the model framework under test: If the conversion fails, the interface adaptation layer directly feeds back to the user interface and outputs the failure information; If the conversion is successful, the type recognition engine launcher will be invoked; (3) Type recognition: The type recognition engine starts and automatically completes structural fingerprint extraction, behavioral feature capture and classification, and outputs category labels; After the type recognition engine launcher starts, the structural fingerprint extractor and behavioral feature capturer are invoked simultaneously. In the structural fingerprint extractor, the configuration file and layer type information of the model under test are parsed. If parsing fails, a user interface failure message is returned. If extraction is successful, 128-dimensional structural fingerprint data is generated and passed to the type classifier. In the behavioral feature capturer, behavioral feature vectors are captured by inputting preset probe data. If capture fails, a user interface failure message is returned. If capture is successful, a 64-dimensional behavioral feature vector is generated and passed to the type classifier. After receiving data from both the structural fingerprint extractor and the behavioral feature capturer, the type classifier begins type classification. If classification fails, a user interface failure message is returned. If classification is successful, the category label of the model under test is returned, and the indicator engine launcher is invoked. (4) Metric Engine Cluster: The metric engine cluster activates the corresponding task based on the category label: if the result generation fails, the user interaction interface will be fed back with a failure message. If the result is successfully generated, the result data will be fed back to the result aggregator and report generator. (5) Results aggregation and report generation: The results aggregation and report generator receives real-time data from each engine, completes alignment, normalization and ranking, and generates a comprehensive score; It will provide feedback according to the preset report output format and push method.

Images