Intelligent system intelligent capability testing device based on test bed

Abstract

The application discloses a kind of intelligent capability test devices based on test bed, comprising: test support server, for providing the computing resource and storage resource of intelligent system model operation and application operation, provide the basic operation environment required;To be tested application container cloud module, for the unified management scheduling of server resource, resource can be distributed to model and application operation use;While providing the operation sandbox of model and application, provide the resource isolation when testing model and application;Test integrated framework module, for the integration of functional components on test framework, responsible for the interface docking between functional components and platform functional components, the input and output of data, including the input of model inference result data, application operation result data and the output of evaluation result.The application realizes the standardization of intelligent capability quantitative evaluation by intelligent capability evaluation resource integrated framework design.

Description

Technical Field

[0001] This invention relates to ship system testing technology, and more particularly to an intelligent capability testing device based on a testbed. Background Technology

[0002] With the widespread application of machine learning, deep learning, and other technologies in ship systems for perception and decision-making, traditional function-based testing devices and methods are increasingly unable to effectively cover the new characteristics of intelligent systems and technologies. The difficulty of testing intelligent application modules in these systems is becoming increasingly apparent. Whether the actual performance of new intelligent systems improves compared to existing systems remains to be evaluated and verified. Currently, most testing methods for artificial intelligence involve manually running models or applications, analyzing the results, and calculating metrics to assess the intelligence capabilities of the models or applications. This leads to inconsistent testing environments, low testing efficiency, and the presence of human interference, resulting in inconsistent and unreliable test results. Therefore, testing for artificial intelligence currently lacks a unified testing environment, effective testing devices or platforms, and automated testing methods. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an intelligent capability testing device based on a test bed, addressing the deficiencies in the prior art.

[0004] The technical solution adopted by this invention to solve its technical problem is: an intelligent capability testing device based on a test bed, comprising:

[0005] The test support server provides computing and storage resources for running intelligent system models and applications, and provides the necessary basic operating environment.

[0006] The application container cloud module under test is used for unified management and scheduling of server resources, and can allocate resources to models and applications for operation.

[0007] The container cloud module for the application under test adopts a microservice architecture and combines Docker container virtualization with Kubernetes container management technology. It provides containerized test applications with complete functions including deployment, service discovery, and dynamic scaling, improving the convenience of large-scale container cluster management. It has comprehensive cluster management capabilities, multi-layered security protection and access control mechanisms, multi-tenant application support capabilities, transparent service registration and discovery mechanisms, built-in intelligent load balancer, powerful fault detection and self-healing capabilities, rolling service upgrades and online expansion capabilities, scalable automatic resource scheduling mechanisms, and multi-granularity resource quota management capabilities. At the same time, it provides a "sandbox" (a separate running tool isolated from external applications) for model and application operation, for resource isolation during model and application operation.

[0008] The test integration framework module is used to integrate functional components on the framework. It is responsible for completing the interface connection between functional components and platform functional components, as well as the input and output of data, including the input of inference result data and application running result data, and the output of evaluation results.

[0009] The functional components include a test data management module, an application management module, an application test management module, a simulation management module, a model management module, a model package repository, and a model test management module; the platform functional components include an evaluation service module, a test task management module, a test result management module, and a test visualization service module.

[0010] The functional components include a test data management module, an application management module, an application test management module, a simulation management module, a model management module, a model package repository, and a model test management module.

[0011] The test data management module is used to manage the test dataset and ground truth data in the model test package, and provides test dataset details, adversarial test data generation, and test data annotation.

[0012] The test data management module manages the test datasets and ground truth data in the model test package, providing functions such as viewing test dataset details, generating adversarial test data, and annotating test data. Test dataset information includes dataset name, import time, description, and version. Test data management provides adversarial attack test data generation, which can be used for model security testing and to evaluate model reliability. Test data annotation performs online annotation on the generated adversarial attack test data, forming corresponding ground truth data.

[0013] The application management module is used for registering, modifying, and deleting intelligent application information. It provides an application information interface to import application information. The intelligent application information mainly includes application name, version, description information, and DDS theme.

[0014] The application test management module connects with the infrastructure platform to enable application virtual machines and application startup during the full lifecycle automated testing of intelligent applications. It also subscribes to DDS topics with the application to collect application runtime results.

[0015] The simulation management module is used to drive the simulator service to send simulation test data to intelligent applications through the DDS interface, providing intelligent applications with different types of test data such as images, videos, audio, radio frequency, and situational awareness, promoting the full-lifecycle automated testing and verification of applications, and providing a simulation management visualization interface;

[0016] The model management module provides a model test package format specification and enables the import of model test packages.

[0017] The model package repository is used to provide model package import, model package generation, model management, and test data management.

[0018] The model test management module is used to start the model container and connect with the test code service within the model container during intelligent model evaluation and testing to collect model inference results.

[0019] The evaluation service module is used to store the model inference results, application operation results, and solution results through data collection. Then, based on the evaluation indicator system and evaluation method, it will complete the evaluation indicator calculation service. The evaluation indicator calculation service will be used to analyze and quantify the above result data, derive indicator data, and generate a task summary report to support the review and inspection of the effectiveness of model / application task execution.

[0020] Based on the evaluation system and methodology, the evaluation computing service adopts a microservice organization to support the calculation of each evaluation indicator. In terms of algorithms, it supports the evaluation of the algorithmic capabilities of deep learning models, focusing on the performance indicators (accuracy, precision, recall, etc.) of the algorithm models used by the participating applications. In terms of applications, it supports the performance evaluation of intelligent applications, such as video images and situational simulations. Combining specific use cases and personalized characteristics, the evaluation service also supports the calculation of assessment indicators for integrated intelligent applications.

[0021] The test task management module is used to create full-cycle verification test tasks, supports task status query, task stop and termination operations, provides a task repository to support task management, including task detail query and deletion operations, and supports task report generation, task report download and online preview of task reports.

[0022] Tasks in the task repository are displayed in a list format, providing management options for created tasks. Users can delete completed or stopped tasks, view results, and other related functions. Detailed task information includes task name, task description, task status, creation time, and available operations such as deletion, viewing results, and report download.

[0023] Task creation involves selecting a suitable template from the template repository for verification and creation. Task control is supported, including tasks such as termination and deletion. Task status queries allow real-time viewing of the task's running status and current stage, as well as viewing output logs for each stage. Task status control provides management of running tasks, supporting operations such as stopping and terminating. Stopping a task pauses it; terminating a task terminates it.

[0024] The test result management module is used to provide a unified data storage and query function for model / application execution results for intelligent performance testing;

[0025] The test results management module provides unified data storage and querying functions for model / application runtime results in intelligent performance testing. It supports data storage for model inference results and runtime results, and provides download capabilities for inference results, runtime results, and solution data. It also supports data storage for model evaluation results, application evaluation results, and simulation evaluation results, and provides download capabilities for various evaluation results. Through visualization technology, the test results module enables unified management and display of data such as inference results, runtime results, solution data, model evaluation results, application evaluation results, and parallel simulation evaluation results from the algorithm layer, application layer, and link layer evaluation processes.

[0026] The test visualization service module is used to provide visualization of the intelligent model / application intelligent capability assessment process and assessment results.

[0027] The test visualization service module provides visualization capabilities for intelligent system testing, particularly the visualization of the evaluation process and results for intelligent models / applications. Different visualization formats for evaluation metrics are presented based on the variations in the format of the evaluation output data and the characteristics of the models / applications. Evaluation process visualization includes the visualization of the current running stage of the evaluation process, the results of completed stages, and log information from completed stages, providing users with the ability to monitor the evaluation process's status in real time. Evaluation result visualization includes visualization of model / application evaluation result analysis and result comparison; the visualization design will employ appropriate visualization methods for different categories.

[0028] The beneficial effects of this invention are:

[0029] 1) This invention proposes an intelligent testing device that meets the environmental requirements for intelligent model and intelligent application operation, and has the open and integrated capabilities of a "testbed", making it more suitable for artificial intelligence testing than existing testing devices.

[0030] 2) This invention, through the design of an intelligent capability assessment resource integration framework, clarifies the quantitative assessment process of the three-layer assessment indicators of "algorithm-application-link", thereby achieving the standardization of intelligent capability quantitative assessment. Attached Figure Description

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0032] Figure 1 This is a structural schematic diagram of an embodiment of the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0034] like Figure 1 As shown, an intelligent capability testing device based on a testbed includes:

[0035] The test support server provides computing and storage resources for running intelligent system models and applications, and provides the necessary basic operating environment.

[0036] The application container cloud module under test is used for unified management and scheduling of server resources, and can allocate resources to models and applications for operation.

[0037] The container cloud module for the application under test adopts a microservice architecture and combines Docker container virtualization with Kubernetes container management technology. It provides containerized test applications with complete functions including deployment, service discovery, and dynamic scaling, improving the convenience of large-scale container cluster management. It has comprehensive cluster management capabilities, multi-layered security protection and access control mechanisms, multi-tenant application support capabilities, transparent service registration and discovery mechanisms, built-in intelligent load balancer, powerful fault detection and self-healing capabilities, rolling service upgrades and online expansion capabilities, scalable automatic resource scheduling mechanisms, and multi-granularity resource quota management capabilities. At the same time, it provides a "sandbox" (a separate running tool isolated from external applications) for model and application operation, for resource isolation during model and application operation.

[0038] The test integration framework module is used to integrate functional components on the framework. It is responsible for completing the interface connection between functional components and platform functional components, as well as the input and output of data, including the input of inference result data and application running result data, and the output of evaluation results.

[0039] The functional components include a test data management module, an application management module, an application test management module, a simulation management module, a model management module, a model package repository, and a model test management module; the platform functional components include an evaluation service module, a test task management module, a test result management module, and a test visualization service module.

[0040] The functional components include a test data management module, an application management module, an application test management module, a simulation management module, a model management module, a model package repository, and a model test management module.

[0041] The test data management module is used to manage the test dataset and ground truth data in the model test package, and provides test dataset details, adversarial test data generation, and test data annotation.

[0042] The test data management module manages the test datasets and ground truth data in the model test package, providing functions such as viewing test dataset details, generating adversarial test data, and annotating test data. Test dataset information includes dataset name, import time, description, and version. Test data management provides adversarial attack test data generation, which can be used for model security testing and to improve model reliability. Test data annotation performs online annotation on the generated adversarial attack test data, forming corresponding ground truth data.

[0043] The application management module is used for registering, modifying, and deleting intelligent application information. It provides an application information interface to import application information. The intelligent application information mainly includes application name, version, description information, and DDS theme.

[0044] The application test management module connects with the infrastructure platform to enable application virtual machines and application startup during the full lifecycle automated testing of intelligent applications. It also subscribes to DDS topics with the application to collect application runtime results.

[0045] The simulation management module is used to drive the simulator service to send simulation test data to intelligent applications through the DDS interface, providing intelligent applications with different types of test data such as images, videos, audio, radio frequency, and situational awareness, promoting the full-lifecycle automated testing and verification of applications, and providing a simulation management visualization interface;

[0046] The model management module provides a model test package format specification and enables the import of model test packages.

[0047] The model package repository is used to provide model package import, model package generation, model management, and test data management.

[0048] The model test management module is used to start the model container and connect with the test code service within the model container during intelligent model evaluation and testing to collect model inference results.

[0049] The evaluation service module is used to store the model inference results, application operation results, and solution results through data collection. Then, based on the evaluation indicator system and evaluation method, it will complete the evaluation indicator calculation service. The evaluation indicator calculation service will be used to analyze and quantify the above result data, derive indicator data, and generate a task summary report to support the review and inspection of the effectiveness of model / application task execution.

[0050] Based on the evaluation system and methodology, the evaluation computing service adopts a microservice organization to support the calculation of each evaluation indicator. In terms of algorithms, it supports the evaluation of the algorithmic capabilities of deep learning models, focusing on the performance indicators (accuracy, precision, recall, etc.) of the algorithm models used by the participating applications. In terms of applications, it supports the performance evaluation of intelligent applications, such as video images and situational simulations. Combining specific use cases and personalized characteristics, the evaluation service also supports the calculation of assessment indicators for integrated intelligent applications.

[0051] The test task management module is used to create full-cycle verification test tasks, supports task status query, task stop and termination operations, provides a task repository to support task management, including task detail query and deletion operations, and supports task report generation, task report download and online preview of task reports.

[0052] Tasks in the task repository are displayed in a list format, providing management options for created tasks. Users can delete completed or stopped tasks, view results, and other related functions. Detailed task information includes task name, task description, task status, creation time, and available operations such as deletion, viewing results, and report download.

[0053] Task creation involves selecting a suitable template from the template repository for verification and creation. Task control is supported, including tasks such as termination and deletion. Task status queries allow real-time viewing of the task's running status and current stage, as well as viewing output logs for each stage. Task status control provides management of running tasks, supporting operations such as stopping and terminating. Stopping a task pauses it; terminating a task terminates it.

[0054] The test result management module is used to provide a unified data storage and query function for model / application execution results for intelligent performance testing;

[0055] The test result data management module provides unified data storage and query functions for model / application runtime results in intelligent performance testing. It supports data storage for model inference results and runtime results, and provides download capabilities for inference results, runtime results, and solution data. It also supports data storage for model evaluation results, application evaluation results, and simulation evaluation results, and provides download capabilities for various evaluation results. Through visualization technology, the data management module enables unified management and display of data such as inference results, runtime results, solution data, model evaluation results, application evaluation results, and parallel simulation evaluation results from the algorithm layer, application layer, and link layer evaluation processes.

[0056] The test visualization service module is used to provide visualization of the intelligent model / application intelligent capability assessment process and assessment results.

[0057] The test visualization service module provides visualization capabilities for intelligent system testing, particularly the visualization of the evaluation process and results for intelligent models / applications. Different visualization formats for evaluation metrics are presented based on the variations in the format of the evaluation output data and the characteristics of the models / applications. Evaluation process visualization includes the visualization of the current running stage of the evaluation process, the results of completed stages, and log information from completed stages, providing users with the ability to monitor the evaluation process's status in real time. Evaluation result visualization includes visualization of model / application evaluation result analysis and result comparison; the visualization design will employ appropriate visualization methods for different categories.

[0058] Based on the above device, automated testing of intelligent models can be carried out. The method includes five steps: test sample generation, model inference task execution, inference result data collection, evaluation index calculation, and evaluation result display.

[0059] It can also be used for automated testing of intelligent application software. The method includes five steps: application startup, simulator triggering, application result collection, evaluation index calculation, and evaluation result display.

[0060] (1) Application startup: The application is started within the test device, and the application running results are subscribed to through the interface;

[0061] (2) Simulator triggering: completes the triggering of the test simulator, enabling the simulator to provide simulation test data to the application inference, prompting the application to start executing tasks;

[0062] (3) Save results: Receive and save the running result data from the application through the interface;

[0063] (4) Evaluation index calculation: Complete the calculation of evaluation indexes for smart applications;

[0064] (5) Evaluation results display: Complete the visualization display of the application evaluation results.

[0065] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. An intelligent capability testing device based on a testbed, characterized in that, include: The test support server provides computing and storage resources for running intelligent system models and applications, and provides the necessary basic operating environment. The application container cloud module under test is used for unified management and scheduling of server resources, and can allocate resources to models and applications for operation. It also provides a sandbox for running models and applications, and provides resource isolation when testing models and applications; The test integration framework module is used to integrate functional components on the test framework. It is responsible for completing the interface connection between functional components and platform functional components, as well as the input and output of data, including the input of model inference result data and application running result data, and the output of evaluation results. The functional components include a test data management module, an application management module, an application test management module, a simulation management module, a model management module, a model package repository, and a model test management module. The platform's functional components include: an evaluation service module, a test task management module, a test result management module, and a test visualization service module; The functional components include a test data management module, an application management module, an application test management module, a simulation management module, a model management module, a model package repository, and a model test management module. The test data management module is used to manage the test dataset and ground truth data in the model test package, and provides test dataset details, adversarial test data generation, and test data annotation. The test data management module manages the test datasets and ground truth data in the model test package, providing functions for viewing test dataset details, generating adversarial test data, and annotating test data. Test dataset information includes dataset name, dataset import time, dataset description, and dataset version. Test data management provides adversarial attack test data generation, which can be used for model security testing and reliability assessment. Test data annotation performs online annotation on the generated adversarial attack test data to form corresponding ground truth data. The application management module is used for registering, modifying, and deleting intelligent application information. It provides an application information interface to import application information. The intelligent application information mainly includes application name, version, description information, and DDS theme. The application test management module connects with the infrastructure platform to enable application virtual machines and application startup during the full lifecycle automated testing of intelligent applications, and subscribes to DDS topics with the applications to collect application running results. The simulation management module is used to drive the simulator service to send simulation test data to intelligent applications through the DDS interface. It provides intelligent applications with different types of test data, including images, videos, audio, radio frequency, and situational awareness, to promote the full-lifecycle automated testing and verification of applications, and provides a simulation management visualization interface. The model management module provides a model test package format specification and enables the import of model test packages. The model package repository provides functions for importing model packages, generating model packages, managing models, and managing test data. The model test management module is used to start the model container and connect with the test code service within the model container during intelligent model evaluation and testing to collect model inference results. The evaluation service module is used to store the model inference results, application operation results and scheme results through data collection. Then, based on the evaluation indicator system and evaluation method, it will complete the evaluation indicator calculation service, and use the evaluation indicator calculation service to analyze and quantify the above result data to obtain indicator data. The test task management module is used to create full-cycle verification test tasks and supports task status query, task stop, and termination operations. The test result management module is used to provide a unified data storage and query function for model or application execution results of intelligent performance testing. The test result data management module provides a unified data storage and query function for model or application execution results for intelligent performance testing. It supports data storage for model inference results and execution results, and provides the ability to download inference results, execution results, and solution data. It also supports data storage for model evaluation results, application evaluation results, and deduction evaluation results, and provides the ability to download various evaluation results. The test visualization service module is used to provide visualization of the intelligent capability assessment process and results of intelligent models / applications.

2. The intelligent capability testing device based on a testbed according to claim 1, characterized in that, The evaluation service module adopts a microservice organization form and supports the calculation of each evaluation indicator according to the evaluation system and evaluation method. The evaluation service module supports the algorithm capability evaluation of deep learning models and calculates and evaluates the performance indicators of the algorithm models used by the participating applications. It also supports the performance evaluation of intelligent applications, including video images and situational simulations. In combination with specific application fields and personalized characteristics, the evaluation service module supports the calculation service of assessment indicators for integrated intelligent applications.

Images