A deep learning test system based on a pipeline design

Abstract

The application relates to a deep learning test system based on a pipeline design, which comprises a test environment configuration module, a test project management module, a multi-framework / format model configuration module, a test pipeline management module and a test result visualization and report generation module; environment configuration and initialization are performed through the test environment configuration module, then project creation, version creation and case creation are performed through the test project management module, model configuration and instantiation are performed through the multi-framework / format model configuration module in the process, environment resources are scheduled through the test pipeline management module when the case is executed, each test component is sequentially driven to execute, and finally the test result is output to the test result visualization and report generation module for display. The application solves the problems of process fragmentation, low automation degree, complex environment configuration, single evaluation dimension, non-automatic result visualization and report generation and the like in the existing deep learning model test process.

Description

Technical Field

[0001] This invention relates to a deep learning testing system based on pipeline design, belonging to the field of software testing technology. Background Technology

[0002] Deep learning technology has been widely applied in fields such as computer vision, natural language processing, and speech recognition, and is gradually being deployed in safety-critical scenarios such as autonomous driving, medical diagnosis, and financial risk control. The reliability, robustness, fairness, and efficiency of the models are crucial. However, testing deep learning models faces significant challenges: (1) Complex and tedious process: A complete model test requires multiple steps such as data cleaning, model inference, adversarial attack, performance analysis (such as accuracy, latency, power consumption), and result visualization. Manual operation is inefficient and prone to errors.

[0003] (2) Strong environmental dependence: Testing may need to be carried out in different hardware (CPU, GPU, NPU), frameworks (PyTorch, TensorFlow, ONNX Runtime) and deployment environments (cloud, edge), with complex environment configuration and poor reusability.

[0004] (3) Multiple evaluation dimensions: Accuracy alone is not enough to guarantee model quality. It is also necessary to evaluate the robustness of the model against adversarial examples (such as FGSM, PGD attacks), the uncertainty of the model, etc. There is a lack of a unified evaluation framework.

[0005] In existing technologies, there are some independent model testing tools or frameworks, such as ART for model robustness, NNCF for model compression, and MLPerf for performance benchmarking. However, these are usually piecemeal solutions and lack integration. Therefore, there is an urgent need in this field for a solution that can standardize and automate the various stages of deep learning testing, and provide a unified testing system and method that is configurable, scalable, and reusable. Summary of the Invention

[0006] The technical problem solved by this invention is to overcome the shortcomings of the prior art and propose a deep learning testing system based on pipeline design, which solves the problems of process fragmentation, low degree of automation, complex environment configuration, single evaluation dimension, and non-automated result visualization and report generation in the existing deep learning model testing process.

[0007] The technical solution of this invention is: a deep learning testing system based on pipeline design, comprising: a test environment configuration module, a test project management module, a multi-framework / format model configuration module, a test pipeline management module, and a test result visualization and report generation module; wherein, The test environment configuration module is used to manage computing resources in a unified manner in a distributed test environment. This module provides an underlying resource pool for test task execution, and the resources it manages can be directly called by the test pipeline. The test project management module is used to create, delete, edit, and archive test projects, versions, and test cases. It provides a unified management interface for uploading and configuring model configuration files and datasets. As the core entry point for users to organize test activities, the created projects will act as containers, associating and scheduling the test environment configuration module, multi-framework / format model configuration module, test pipeline management module, and test result visualization and report generation module, to achieve centralized management of test assets and version traceability. The multi-framework / format model configuration module is used to automatically identify, load, and adapt models generated by different deep learning frameworks. This module serves as a bridge between specific AI models and the upper-level testing process. It receives model files configured by the project management module, loads and instantiates them, and then provides them to the test pipeline management module for scheduling and testing. The test pipeline management module, as the core controller of the system, is used to design and define the complete test workflow. Based on the tasks defined by the test project management module, this module organizes the models under test provided by the multi-framework / format model configuration module, calls the resources of the test environment configuration module, and drives the various test components to execute in sequence. Finally, it outputs the raw results to the test result visualization and report generation module. The test result visualization and report generation module displays the configuration, execution progress, and test results of each test stage in real time, presenting the evaluation data intuitively through line charts, bar charts, and scatter plots. This module directly connects to the pipeline management module to acquire and parse test status and result data in real time. The generated reports can be archived, forming a complete closed loop from planning, execution to analysis.

[0008] The test environment configuration module maintains an execution machine resource pool, which can interface with physical servers, virtual machines, and containers. Its functions include: registration of execution machines, collecting and maintaining the specification information of each execution machine; real-time and continuous status monitoring of all execution machines in the resource pool; environment initialization, providing environment templates and automated configuration capabilities; and task allocation and scheduling, acting as a resource scheduling hub, receiving task requests from the test pipeline, and automatically distributing test tasks to the corresponding computing nodes for execution based on task requirements and scheduling strategies.

[0009] The test project management module serves as the core entry point and central control hub for users to organize test activities, enabling centralized and versioned management of test assets and processes. This module allows users to create structured test projects and versions, which serve as a unified container for managing test cases, configuration model files, and datasets, and provides asset uploading and version tagging functions. The test project management module, as the central hub driving the test process, dynamically associates the models and data configured within the project with other core modules, assigns computing resources in the test environment configuration module to the task, passes the model to the multi-framework model configuration module for loading, drives the test pipeline management module to execute tests according to the preset process, and automatically archives all results output by the test result visualization and report generation module to the corresponding project, thereby forming an end-to-end closed loop from planning, configuration, execution to backtracking, ensuring that the entire test activity is traceable and assets are reusable.

[0010] The multi-framework / format model configuration module has built-in parsers for mainstream frameworks. After a user uploads a model file, the module can automatically identify its format, configure the corresponding runtime environment for it, and encapsulate the model into a standard test interface for subsequent testing, thereby achieving transparent testing of heterogeneous models.

[0011] The test pipeline management module is responsible for the lifecycle management of the entire pipeline, including creation, startup, pause, stop, and tracking of pipeline execution status. Through a graphical interface or configuration file, the entire test workflow is divided into multiple sequential execution stages, specifically including: data review, data augmentation, model structure review, correctness testing, robustness testing, uncertainty testing, and coverage testing. Each environment can independently select and configure indicators.

[0012] The data review includes a comprehensive assessment of data quality based on indicators such as data accuracy, data integrity, data consistency, dataset size, data rationality, data duplication, data category diversity, and data volatility.

[0013] The data augmentation includes providing geometric transformations, color transformations, noise transformations, blur transformations, and weather transformations, which can be flexibly selected to improve the diversity and coverage of the test data.

[0014] The model structure review includes: supporting the complete display of the model architecture, allowing users to interactively view network layers, module connection relationships and detailed parameter configurations, and assisting in the analysis of structural rationality.

[0015] The correctness test includes: verifying the model's functionality and performance under the expected environment; being able to customize the pass rate according to the model category and task type, and verifying the model's correctness through F1 score, pixel accuracy, and recall.

[0016] The robustness test includes two parts: adversarial testing and anti-interference testing. The adversarial test provides gradient sign method, projective gradient descent method, and Carlini & Wagner attack method, which can adjust the attack strength by setting perturbation parameters. The anti-interference test provides interference indicators including histogram equalization, color channel bit depth transformation, and illumination change.

[0017] 12. The uncertainty test includes: quantifying the uncertainty of the tested image classification, object detection, and semantic segmentation through Bayesian network modeling, which is used to guide the reliability of the prediction.

[0018] 13. The coverage test includes five indicators: neuron coverage, K-segment neuron coverage, Top-K neuron coverage, strong activation coverage, and boundary coverage. These indicators evaluate the degree to which test cases cover the internal structure of the model, thereby guiding the generation of new test cases to improve the comprehensiveness of the test.

[0019] The advantages of this invention compared to the prior art are: This invention provides a highly integrated and flexibly configurable deep learning testing pipeline system, effectively solving problems such as fragmented testing processes, insufficient automation, complex environment adaptation, limited evaluation dimensions, and low report generation efficiency in the current testing landscape. By integrating scattered testing steps into a standardized pipeline, it achieves fully automated execution and unified management, significantly improving testing efficiency and coverage. The system is compatible with mainstream deep learning frameworks and multiple operating systems, possessing good applicability and meeting the model testing needs of various practical scenarios, while providing comprehensive and objective data support for model quality evaluation. Specifically, this invention has the following technical advantages: 1. This invention utilizes a "test pipeline management module" and a graphical / configurable pipeline customization function to integrate previously fragmented and independent deep learning testing processes (data review, model testing, etc.) into an automated process that can be customized on demand and executed sequentially. This completely changes the traditional testing model of piling up point-by-point tools and manually connecting them, achieving standardization, visualization, and full-process automation of the testing process. It significantly reduces the complexity of test configuration and improves the efficiency and consistency of test execution.

[0020] 2. This invention utilizes an execution machine resource pool and dynamic scheduling mechanism built by the "Test Environment Configuration Module." It uniformly manages heterogeneous computing resources such as physical machines, virtual machines, and containers, monitors their status and load in real time, and intelligently distributes tasks to appropriate computing nodes according to test task requirements. This solves the configuration complexity and resource silo problems of cross-platform and cross-environment testing. It achieves elastic scheduling and efficient utilization of computing resources, ensuring stable execution of test tasks in a high-performance, highly available environment, making it particularly suitable for large-scale, distributed testing scenarios.

[0021] 3. This invention utilizes a "multi-framework / format model configuration module" and its built-in multi-framework parsers, which can automatically identify and load model files from different frameworks such as PyTorch, TensorFlow, and ONNX, encapsulating them into standardized test interfaces. This achieves seamless and transparent access to heterogeneous models. Users do not need to worry about underlying framework differences and cumbersome environment configurations; they can perform unified testing on models from different sources on the same platform, greatly improving the convenience and coverage of testing.

[0022] 4. This invention employs an integrated testing scheme encompassing seven key stages: data review, correctness testing, robustness testing, uncertainty testing, and coverage testing. It provides a multi-dimensional and systematic quality assessment framework, from the data source to the model's internal behavior, replacing traditional testing methods that focus solely on accuracy or a single attribute. This enables a more comprehensive and in-depth evaluation of deep learning model quality. It not only identifies functional errors but also recognizes data defects, quantifies adversarial risks, assesses prediction reliability and test adequacy, providing a solid and multi-dimensional quality basis for model deployment in high-reliability scenarios.

[0023] 5. This invention utilizes a "Test Result Visualization and Report Generation Module" and its built-in GJB 438C standardized report template. It provides real-time visualization of the progress and results of each stage, automatically aggregating all test configurations, process data, and result charts, and generating a structured WORD-format test report with a single click. This achieves a high degree of transparency and traceability in the testing process and results. Intuitive charts facilitate rapid analysis, while automated and standardized report generation greatly reduces the burden of manual document writing and organization, ensuring the standardization of deliverables and directly meeting project audit and compliance requirements. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the testing process of the present invention; Figure 2 This is a schematic diagram of a deep learning testing system architecture based on pipeline design according to the present invention. Detailed Implementation

[0025] like Figure 1 , 2The diagram shows the deep learning testing process and system composition designed in this invention. A complete testing process is formed through modularization. First, the environment is configured and initialized through the test environment configuration module. Then, the project, version, and test cases are created through the test project management module. During the process, the model is configured and instantiated through the multi-framework / format model configuration module. When executing test cases, the test pipeline management module schedules environment resources and drives each test component to execute in sequence. Finally, the test results are output to the test result visualization and report generation module for display, and a test report can be generated with one click.

[0026] (1) Test Environment Configuration Module: This module is used for unified management of computing resources in a distributed test environment. It maintains a pool of execution machine resources and can interface with physical servers, virtual machines, and containers (such as Docker). Its functions include: execution machine registration; collecting and maintaining the specifications of each execution machine (such as CPU architecture, operating system, IP address, etc.); status monitoring; real-time and continuous status monitoring of all execution machines in the resource pool (such as GPU utilization and memory usage); environment initialization; providing environment templates and automated configuration capabilities; automatically initializing the corresponding runtime environment on the target execution machine based on test task requirements (such as the need for PyTorch 2.0 + CUDA 11.8), including the installation and configuration of dependency libraries, deep learning frameworks, and test toolchains, ensuring the consistency and reproducibility of the test environment; and task allocation and scheduling. As a resource scheduling hub, it receives task requests from the test pipeline and automatically distributes test tasks to appropriate computing nodes for execution based on task requirements and scheduling strategies. Through the above capabilities, this module translates the abstract concept of "unified resource management" into concrete and automated operation and maintenance, forming a flexible infrastructure for the platform's efficient and stable operation. It transforms the originally scattered and heterogeneous computer resources into a unified and flexible intelligent resource pool, and through real-time monitoring and intelligent scheduling, it achieves global optimization and efficient utilization of computing resources, reducing hardware idle costs.

[0027] (2) Test Project Management Module: Supports the creation, deletion, editing, and archiving of test projects, versions, and test cases, and provides a unified management interface for uploading and configuring model configuration files and datasets. This module serves as the central control hub and core entry point for users to organize and manage intelligent testing activities, achieving structured and versioned centralized control of test assets and processes. Its core principle lies in designing each test project as an intelligent container that can dynamically associate and schedule all platform services. When a user initiates a test task, this module acts as a central coordinator, automatically configuring computing resources in the task-associated test environment module, transferring model assets to the multi-framework model configuration module for parsing and loading, and driving the test pipeline management module to execute tests according to a preset, programmable process. After the test is completed, all process data and final results are automatically returned and archived to the original project, thus forming a complete, closed-loop, and traceable engineering process from requirement definition, asset configuration, task execution to result backtracking. The advantages of this design are that it achieves the engineering and standardization of testing activities, transforming discrete testing tasks into reusable and auditable assets; it greatly improves the efficiency of team collaboration and knowledge accumulation, with all configurations, data, and results having a unique version source, avoiding information silos; and it ensures a high degree of automation and reproducibility of the testing process. Through automatic linkage between modules, it reduces the links and errors of manual intervention, enabling each test to be accurately reproduced, providing a solid foundation for continuous integration and quality evolution.

[0028] (3) Multi-framework / format model configuration module: This module enables automatic identification, unified loading, and seamless adaptation of models generated from different deep learning frameworks. Its core principle lies in its built-in dedicated parser and dynamic environment construction mechanism covering mainstream frameworks such as TensorFlow, PyTorch, Keras, and ONNX. When a user uploads a model file, the module automatically identifies its format type by parsing the file structure characteristics and intelligently matches the corresponding runtime environment—including automatically creating or selecting containers or virtual environments with the corresponding Python version, framework dependencies, and hardware driver support. This allows various heterogeneous models to be uniformly encapsulated into test interfaces that conform to platform standards. This process not only completely shields the complexity of underlying framework differences and environment configuration but also enables all subsequent testing stages to call and execute models from different sources in a consistent manner, achieving true "upload and test instantly." Its core benefit lies in significantly reducing the technical threshold and time cost of testing multi-framework models. Users can complete large-scale and automated testing of diverse models on the same platform without manually adapting the environment or rewriting code, which greatly improves the coverage, execution efficiency and system scalability of the testing process, laying a technical foundation for building an open and compatible AI evaluation ecosystem.

[0029] (4) Test Pipeline Management Module: As the core controller of the system, this module designs and defines the complete test workflow, including creation, startup, pause, stop, and tracking of pipeline execution status. Users can divide the entire testing process into multiple sequential or parallel stages through a graphical interface or configuration file, specifically including: data review, data augmentation, model structure review, correctness testing, robustness testing, uncertainty testing, and coverage testing. Each environment can independently select and configure metrics. The design principle of this module is based on the concept of orchestratable workflow and declarative configuration management. This module abstracts the complete deep learning test into a series of standardized, pluggable test stages (such as data review, model testing, etc.) and manages the dependencies between stages through the underlying scheduling engine. The core benefit of this design is that it transforms the complex, multi-step model testing process into a standardized, automated, and highly controllable engineering pipeline. Users can achieve "one-click" full-process testing without manually connecting scattered tools and scripts, greatly reducing operational complexity and the risk of human error. At the same time, the pipeline supports flexible orchestration, significantly improving testing efficiency and resource utilization. All testing activities are structured and monitored, providing a solid and reliable automated foundation for continuous integration, model iteration, and quality backtracking.

[0030] a. The data review in the testing pipeline comprehensively evaluates data quality based on 26 indicators across 8 dimensions: data accuracy (image corruption rate, category error rate, etc.), data integrity (incomplete annotation, category integrity), data consistency (image standard mismatch rate), dataset size (total number of images, total number of categories, etc.), data rationality (proportion of unreasonable annotations), data duplication (image duplication rate, annotation duplication rate), data category diversity (image size distribution, category diversity, etc.), and data volatility (image brightness volatility, image contrast volatility). Users can customize review items by checking boxes.

[0031] b. Data augmentation in the test pipeline offers 33 enhancement methods across 5 categories, including geometric transformations (image scaling, image translation, etc.), color transformations (image brightness, image contrast), noise transformations (Gaussian noise, random shadows, etc.), blur transformations (Gaussian blur, mean blur, etc.), and weather transformations (fog, rain, etc.). Users can flexibly choose to improve the diversity and coverage of test data.

[0032] c. The model structure review in the test pipeline supports a complete model architecture display, allowing users to interactively view network layers, module connection relationships, and detailed parameter configurations, assisting in structural rationality analysis.

[0033] d. Correctness testing in the testing pipeline verifies the model's functionality and performance under expected conditions. It supports customizable metrics such as pass rate based on model category and task type, and verifies model correctness using 29 metrics including F1 score, pixel accuracy, and recall.

[0034] e. Robustness testing in the testing pipeline is divided into two parts: adversarial testing and interference resistance testing. Adversarial testing provides 16 attack methods, including the fast gradient sign method, projective gradient descent, and Carlini & Wagner attack, and the attack strength can be adjusted by setting parameters such as perturbation magnitude. Interference resistance testing provides 12 interference metrics, such as histogram equalization, color channel bit depth transformation, and illumination variation. Both parts of the test allow users to freely select one or more metrics to comprehensively evaluate the reliability of the model's predictions.

[0035] f. Uncertainty Testing in the Testing Pipeline: Uncertainty is quantified in the prediction results for tasks such as image classification, object detection, and semantic segmentation using Bayesian network modeling to guide the reliability of the predictions. This system evaluates the reliability of the model's predictions using five metrics: activation value distribution, parameter perturbation, gradient method, regradient method, and prediction entropy method.

[0036] g. Coverage testing in the test pipeline introduces five metrics: neuron coverage, K-segment neuron coverage, Top-K neuron coverage, strong activation coverage, and boundary coverage, to evaluate the degree to which test cases cover the internal structure of the model, thereby guiding the generation of new test cases to improve the comprehensiveness of testing.

[0037] (5) Test Result Visualization and Report Generation Module: Provides a graphical interface to display the configuration, execution progress, and test results of each test stage in real time, and presents the evaluation data intuitively through various forms such as line charts, bar charts, and scatter plots. It has a built-in test report template that conforms to the GJB 438C standard, automatically aggregates test configurations and results summaries, and generates a complete WORD format test report with one click. The core principle of this module lies in data aggregation mapping and automatic template generation. This module collects and structures configuration, progress, and result data from each test stage in real time, and dynamically maps multi-dimensional evaluation indicators into interactive charts such as line charts, bar charts, and scatter plots through the built-in rendering engine, realizing panoramic visualization and monitoring of the test process and results; at the same time, the system automatically integrates all test metadata, process records, and result charts based on preset standardized templates, and generates a well-structured and richly illustrated WORD format test report with one click through the document automation engine. This mechanism not only achieves transparency and real-time traceability of the entire testing process, enabling users to intuitively and quickly locate problems and evaluate model performance, but also completely eliminates the tedious and error-prone steps of traditional manual document preparation and writing through a highly automated report generation process. This ensures the standardization and consistency of outputs, directly meeting the strict compliance and archiving requirements of the high-reliability field for the testing process, and significantly improving the closed-loop efficiency and professional delivery capabilities of testing activities.

[0038] based on Figure 1 The pipeline method shown forms as follows Figure 2 The flexibly configurable deep learning testing system shown is divided into three parts: a functional component layer, a service layer, and a user interface layer.

[0039] The functional component layer is further divided into basic components and application components. Basic components include a general environment management component for virtual environment management, basic environment adaptive management, and custom environment management; a general intelligent framework adaptation component for heterogeneous model content abstraction, model instantiation logic analysis, and a unified interface for model information; an intelligent model format standardization component for dynamic loading of heterogeneous models, model format reconstruction, and model output error verification; and a general data management component for metadata calibration, automatic metadata annotation, and data format standardization. The application components of the functional component layer include seven components: data review, data augmentation, model visualization review, correctness testing, robustness testing, coverage testing, and uncertainty testing. Figure 1 The seven testing stages in the test pipeline management module.

[0040] The service layer includes a distributed scheduling service responsible for intelligently allocating and coordinating test tasks to multiple computing nodes to achieve parallel execution and load balancing; a data access service providing a unified interface for secure and efficient access to the models, datasets, and intermediate results required for testing; a log monitoring and result analysis service that collects system operation logs and test results in real time, performs visualization analysis and anomaly warnings; a data model scoring service that comprehensively and quantitatively evaluates and scores model performance based on multi-dimensional test metrics; and a test report generation service that automatically aggregates test configurations, process data, and scoring results to generate structured and compliant test report documents with one click.

[0041] The system consists of three parts: a user interface client, a browser-based task collaboration module, and a continuous integration plugin, making it easy to view information such as test progress, results, and logs.

[0042] This implementation example uses Project P, an image classification task within the PyTorch framework, to illustrate the entire implementation process, from system installation and deployment, project creation, model data configuration, to test case execution and output. This example uses Docker for installation instructions.

[0043] (1) Test Environment Configuration: First, deploy the test environment on a distributed computer. On the execution machine, import the Docker image using the command `sudo docker load -i aitest_engines.tar`, and then start the container using the command `sudo docker run`, mounting the GPU resources and working directory. On the server side, start the web service container in a similar manner to complete the basic environment setup of the test platform.

[0044] (2) Test Project Management: Access the system's web interface via a browser (e.g., http: / / [server IP]:8086 / sw / ), create a test project named "Image Classification Project P", and create a corresponding test version. Upload model files and test datasets through the unified configuration interface to complete centralized management of project resources.

[0045] (3) Multi-framework / format model configuration: The system automatically recognizes that the user-uploaded model_def.py file is in PyTorch format, calls the built-in PyTorch parser to load the model structure and weights, and automatically configures the corresponding Python 3.8 + PyTorch 1.12 + CUDA 11.6 runtime environment. The model is encapsulated as a standard test interface for subsequent calls, realizing transparent access to cross-framework models.

[0046] (4) Test pipeline customization: Design the test pipeline in the graphical interface and configure the following steps and their parameters in sequence: a. Data review: Select the test dataset as the review object, check all 26 review indicators in 8 dimensions, and set the threshold for "Category size diversity" to 100. The system will automatically perform the review and generate data quality results. b. Data augmentation: Based on the original test set, the augmentation is performed twice. Five augmentation methods are selected: scaling (parameter: shrink), brightness adjustment (parameter: high), Gaussian noise (parameter: low), Gaussian blur (parameter: high), and weather simulation (parameter: light rain). The system automatically generates the augmented dataset. c. Model Structure Review: The system analyzes and visualizes the PyTorch model structure, allowing users to interactively view the parameter configurations and connection relationships of each convolutional layer and fully connected layer, assisting in the analysis of structural rationality.

[0047] d. Correctness test: Set the passing standard for precision and recall to 80%, and the passing standard for F1 score to 85%. The system performs inference on the test set and outputs the pass / fail status. e. Robustness testing: a) Adversarial testing: Select three methods: FGSM (perturbation size 0.03), C&W attack (perturbation size 0.1), and Newton attack (perturbation size 0.1). The system will generate adversarial examples and evaluate the robustness of the model.

[0048] b) Anti-interference test: Select interference methods such as histogram equalization, color channel bit depth transformation, and illumination variation to test the stability of the model under image quality changes.

[0049] f. Uncertainty Test: Select the parameter perturbation method (perturbation number 10, intensity 10), gradient method and regradient method (perturbation number 5, intensity 0.001). The system quantifies the uncertainty predicted by the model through a Bayesian network and outputs the confidence level. g. Coverage testing: Set the neuron coverage rate (number of segments 20) and the Top-K neuron coverage rate (K value 20). The system analyzes the activation of neurons within the model by the test cases, evaluates the sufficiency of the test, and guides the generation of supplementary test cases.

[0050] (5) Visualization of Test Results and Report Generation: The system updates the progress of each stage in real time during execution and dynamically displays indicators such as accuracy curves, robust attack success rates, and coverage changes through a dashboard. After the test is completed, users can view all detailed results online, including data augmentation images, adversarial examples, uncertainty analysis charts, and newly generated test cases, which can be downloaded separately. Finally, users can trigger the report generation function with one click, and the system automatically integrates all test configurations, result charts, and analysis conclusions to output a complete test report in standard WORD format.

[0051] Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications to the technical solutions of the present invention based on the above-disclosed technical content without departing from the spirit and scope of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention shall fall within the protection scope of the technical solutions of the present invention.

Claims

1. A deep learning testing system based on pipeline design, characterized in that, include: The system includes modules for test environment configuration, test project management, multi-framework / format model configuration, test pipeline management, and test result visualization and report generation. The test environment configuration module is used to manage computing resources in a unified manner in a distributed test environment. This module provides an underlying resource pool for test task execution, and the resources it manages can be directly called by the test pipeline. The test project management module is used to create, delete, edit, and archive test projects, versions, and test cases. It provides a unified management interface for uploading and configuring model configuration files and datasets. As the core entry point for users to organize test activities, the created projects will act as containers, associating and scheduling the test environment configuration module, multi-framework / format model configuration module, test pipeline management module, and test result visualization and report generation module, to achieve centralized management of test assets and version traceability. The multi-framework / format model configuration module is used to automatically identify, load, and adapt models generated by different deep learning frameworks. This module serves as a bridge between specific AI models and the upper-level testing process. It receives model files configured by the project management module, loads and instantiates them, and then provides them to the test pipeline management module for scheduling and testing. The test pipeline management module, as the core controller of the system, is used to design and define the complete test workflow. Based on the tasks defined by the test project management module, this module organizes the models under test provided by the multi-framework / format model configuration module, calls the resources of the test environment configuration module, and drives the various test components to execute in sequence. Finally, it outputs the raw results to the test result visualization and report generation module. The test result visualization and report generation module displays the configuration, execution progress, and test results of each test stage in real time, presenting the evaluation data intuitively through line charts, bar charts, and scatter plots. This module directly connects to the pipeline management module to acquire and parse test status and result data in real time. The generated reports can be archived, forming a complete closed loop from planning, execution to analysis.

2. The deep learning testing system based on pipeline design according to claim 1, characterized in that, The test environment configuration module maintains an execution machine resource pool, which can interface with physical servers, virtual machines, and containers. Its functions include: registration of execution machines, collecting and maintaining the specification information of each execution machine; real-time and continuous status monitoring of all execution machines in the resource pool; environment initialization, providing environment templates and automated configuration capabilities; and task allocation and scheduling, acting as a resource scheduling hub, receiving task requests from the test pipeline, and automatically distributing test tasks to the corresponding computing nodes for execution based on task requirements and scheduling strategies.

3. The deep learning testing system based on pipeline design according to claim 1, characterized in that, The test project management module serves as the core entry point and central control hub for users to organize test activities, enabling centralized and versioned management of test assets and processes. This module allows users to create structured test projects and versions, which act as a unified container for managing test cases, configuration model files, and datasets, and provides asset uploading and version tagging functions.

4. The deep learning testing system based on pipeline design according to claim 1, characterized in that, The test project management module, as the central hub driving the test process, dynamically associates the models and data configured within the project with other core modules, assigns computing resources in the test environment configuration module to the task, passes the model to the multi-framework model configuration module for loading, drives the test pipeline management module to execute tests according to the preset process, and automatically archives all results output by the test result visualization and report generation module to the corresponding project, thereby forming an end-to-end closed loop from planning, configuration, execution to backtracking, ensuring that the entire test activity is traceable and assets are reusable.

5. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The multi-framework / format model configuration module has built-in parsers for mainstream frameworks. After a user uploads a model file, the module can automatically identify its format, configure the corresponding runtime environment for it, and encapsulate the model into a standard test interface for subsequent testing, thereby achieving transparent testing of heterogeneous models.

6. The deep learning testing system based on pipeline design according to claim 1, characterized in that, The test pipeline management module is responsible for the entire pipeline lifecycle management, including creation, startup, pause, stop, and pipeline execution status tracking; The entire testing workflow is divided into multiple sequential execution phases through a graphical interface or configuration file, including: data review, data augmentation, model structure review, correctness testing, robustness testing, uncertainty testing, and coverage testing. Each environment can independently select and configure metrics.

7. A deep learning testing system based on pipeline design according to claim 6, characterized in that, The data review includes a comprehensive assessment of data quality based on indicators such as data accuracy, data integrity, data consistency, dataset size, data rationality, data duplication, data category diversity, and data volatility.

8. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The data augmentation includes providing geometric transformations, color transformations, noise transformations, blur transformations, and weather transformations, which can be flexibly selected to improve the diversity and coverage of the test data.

9. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The model structure review includes: supporting the complete display of the model architecture, allowing users to interactively view network layers, module connection relationships and detailed parameter configurations, and assisting in the analysis of structural rationality.

10. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The correctness test includes: verifying the model's functionality and performance under the expected environment; being able to customize the pass rate according to the model category and task type, and verifying the model's correctness through F1 score, pixel accuracy, and recall.

11. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The robustness test includes two parts: adversarial testing and anti-interference testing. The adversarial test provides gradient sign method, projective gradient descent method, and Carlini & Wagner attack method, which can adjust the attack strength by setting perturbation parameters. The anti-interference test provides interference indicators including histogram equalization, color channel bit depth transformation, and illumination change.

12. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The uncertainty test includes: quantifying the uncertainty of the tested image classification, object detection, and semantic segmentation through Bayesian network modeling, which is used to guide the reliability of the prediction.

13. A deep learning testing system based on pipeline design according to claim 1, characterized in that, The coverage test includes five metrics: neuron coverage, K-segment neuron coverage, Top-K neuron coverage, strong activation coverage, and boundary coverage. These metrics evaluate the extent to which test cases cover the internal structure of the model, thereby guiding the generation of new test cases to improve the comprehensiveness of the test.

Images