Methods and systems for testing artificial intelligence systems
By testing the infrastructure, core logic, and performance of an AI system in stages within a CI/CD pipeline and generating a comprehensive report, this approach addresses the issues of resource waste and low testing efficiency in existing technologies, achieving efficient and comprehensive test coverage and rapid problem discovery.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MOXIN ARTIFICIAL INTELLIGENCE TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-03-16
- Publication Date
- 2026-07-03
Smart Images
Figure CN121880210B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence (AI) technology, and more specifically to methods and systems for testing artificial intelligence systems. Background Technology
[0002] With the rapid development of artificial intelligence technology, especially deep learning and large models, the complexity of AI systems is increasing daily. A complete AI system typically involves a complex software stack (such as specific versions of deep learning frameworks, compilers, and drivers), dedicated hardware (such as accelerator cards), diverse models, and upstream and downstream business logic. Therefore, there is an urgent need to provide testing methods for AI systems. Summary of the Invention
[0003] In one aspect, embodiments of this application provide a method for testing an artificial intelligence (AI) system, comprising: triggering a continuous integration / continuous deployment (CI / CD) pipeline at a test system, driving the AI system to perform a first-phase test on whether the infrastructure of the AI system is ready through a test framework; if the first-phase test passes, driving the AI system to perform a second-phase test on whether the core logic of the AI system functions normally through the test framework; after the second-phase test passes, driving the AI system to deploy an AI model and perform a third-phase test on whether the entire chain operation of the AI model is smooth; after the third-phase test passes, driving the AI system to perform a fourth-phase test on whether the performance of the AI system under stress meets the standards; and after the fourth-phase test passes, collecting full-process logs from the first to the fourth phase tests in the test system, and analyzing the collected logs to generate a comprehensive test report.
[0004] In another aspect, embodiments of this application provide a system for testing an artificial intelligence (AI) system, comprising: a triggering module configured to trigger a continuous integration / continuous deployment (CI / CD) pipeline at the testing system; an infrastructure testing module configured to drive the AI system through a testing framework to perform a first-phase test on whether the infrastructure of the AI system is ready; a core logic testing module configured to, if the first-phase test is passed, drive the AI system through a testing framework to perform a second-phase test on whether the core logic of the AI system functions normally; an integration testing module configured to, after the second-phase test is passed, drive the AI system to deploy an AI model and perform a third-phase test on whether the entire chain operation of the AI model is smooth; a performance testing module configured to, after the third-phase test is passed, drive the AI system to perform a fourth-phase test on whether the performance of the AI system under stress meets the standards; and a log analysis module configured to, after the fourth-phase test is passed, collect the entire process logs from the first to the fourth phase of testing in the testing system and analyze the collected logs to generate a comprehensive test report.
[0005] In another aspect, embodiments of this application provide a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for testing an artificial intelligence system according to embodiments of this application.
[0006] In another aspect, embodiments of this application provide a computer program product storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method for testing an artificial intelligence system according to embodiments of this application.
[0007] The method and system for testing artificial intelligence systems according to embodiments of this application provide a progressive testing scheme combined with CI / CD pipelines. This scheme enables a systematic interconnected framework for testing the infrastructure, core logic, integration operation, performance stress testing, and log reporting of artificial intelligence systems. As a result, it can be deeply integrated into the CI / CD pipeline, achieving efficient resource utilization, comprehensive coverage of the quality dimensions of artificial intelligence systems, and improved testing efficiency. Attached Figure Description
[0008] When read in conjunction with the accompanying drawings, various aspects of this disclosure are best understood through the following detailed description. It should be understood that the drawings are for illustrative and descriptive purposes only and not for limiting purposes. In the drawings:
[0009] Figure 1 This is a schematic diagram of the test architecture according to an embodiment of this application.
[0010] Figure 2 This is a flowchart of a method for testing an artificial intelligence system according to an embodiment of this application.
[0011] Figure 3 This is an application diagram illustrating a method for testing an artificial intelligence system according to an embodiment of this application.
[0012] Figure 4 This is a block diagram of a system for testing an artificial intelligence system according to an embodiment of this application.
[0013] Figure 5 This is a schematic diagram of a computing device that can implement a system for testing an artificial intelligence system according to embodiments of this application. Detailed Implementation
[0014] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this disclosure. Of course, these are merely examples and not limiting.
[0015] Traditional testing processes for AI systems typically employ a complete linear workflow, which can lead to unnecessary waste of time and resources when infrastructure is not yet ready or has issues (such as missing drivers or unavailable accelerator cards). Traditional testing processes also tend to have limited testing dimensions, often executing functional testing, integration testing, and performance testing in isolation. Furthermore, traditional testing processes cannot be embedded in continuous integration / continuous deployment (CI / CD) pipelines, making it difficult to automate "submission-as-test" processes. Finally, traditional testing processes lack standardized verification methods for the core logic of AI systems.
[0016] In view of the above, this application proposes a method and system for testing artificial intelligence systems, which can be deeply integrated into the CI / CD pipeline to achieve efficient resource utilization, comprehensively cover the quality dimensions of artificial intelligence systems, and improve testing efficiency.
[0017] Figure 1 This is a schematic diagram of a test architecture according to an embodiment of this application. For example... Figure 1As shown, the test architecture 100 according to an embodiment of this application includes a test management platform 101 and an artificial intelligence system (also called an artificial intelligence data center) 102 that communicates with the test management platform 101. The communication between the test management platform 101 and the artificial intelligence system 102 can be implemented using direct communication methods based on network protocols (e.g., through specific application programming interfaces (APIs) or container orchestration engines), cloud-native and service mesh-based communication methods (e.g., through the control plane of a service network or a cloud-native API gateway), communication methods based on automated operation and maintenance tools (e.g., through a CI / CD pipeline), and dedicated protocol and hardware-level communication methods (e.g., through the Intelligent Platform Management Interface (IPMI)).
[0018] The artificial intelligence system 102 includes a test environment 1021. The test environment 1021 can also be referred to as a "hardware environment" because it is a combined hardware-software instance, such as consisting of servers, operating systems, and accelerator cards. Artificial intelligence models can run on various servers. It should be understood that the artificial intelligence system 102 may also include any other components known in the art (e.g., supporting storage, network infrastructure, etc.), the description of which is omitted to avoid unnecessarily obscuring this application. Testing of the artificial intelligence system 102 can be performed in an isolated portion of the test environment 1021 (such as an isolated virtualization environment) to avoid interference with the development and production environments. Furthermore, it should be understood that although the above description uses a homogeneous architecture artificial intelligence system, the test architecture according to the embodiments of this application is not limited to this, but can be similarly applied to heterogeneous architecture artificial intelligence systems (i.e., the artificial intelligence system includes test environments 1021 with different configurations). In a heterogeneous architecture context, more than one test environment 1021 can be adapted to the test process according to the embodiments of this application.
[0019] As the "command and dispatch center" for testing, Test Management Platform 101 is the core of the entire testing system. It can perform various tests on different testing environments. Test Management Platform 101 is responsible for the automated orchestration and dynamic scheduling of test tasks, automatically adjusting the number of stress nodes based on the resource load of the artificial intelligence system to avoid wasting test resources. During test execution, Test Management Platform 101 collects real-time end-to-end data from the artificial intelligence system 102, including hardware metrics, system status, and business performance data. Through its built-in intelligent analysis engine, it performs bottleneck identification, root cause analysis, and performance evaluation, ultimately generating visualized test reports or providing error feedback to support the development, maintenance, and optimization of the artificial intelligence system.
[0020] The AI System 102 serves as the "test environment carrier," completely replicating the production-grade configuration. It fully supports various hardware and software resources and can simulate the load characteristics of real business scenarios. The AI System 102 supports both single-test-environment testing and collaborative testing across multiple test environments.
[0021] The following will combine Figures 2-5 A system and method for testing an artificial intelligence system according to embodiments of this application are described in more detail.
[0022] Figure 2 This is a flowchart of a method for testing an artificial intelligence system according to an embodiment of this application. Figure 2 As shown, the method 200 for testing an artificial intelligence system according to an embodiment of this application includes steps S201-S205.
[0023] In step S201, after triggering the continuous integration / continuous deployment (CI / CD) pipeline at the test system, the AI system is driven to perform the first phase of testing on whether the AI system's infrastructure is ready through the test framework.
[0024] The first phase of testing primarily eliminates infrastructure-related issues. A CI / CD pipeline could be, for example, a GitLab CI / CD pipeline. In some implementations, a CI / CD pipeline is triggered in one of the following ways: when testers submit test code to the CI / CD pipeline, automatically at a predetermined time, or manually by the testers.
[0025] In the first phase of testing, after the CI / CD pipeline is triggered, the installation of the software stack required by the artificial intelligence system (including deep learning frameworks, compilers, drivers, etc.) is completed automatically, for example, by following the prepare.sh script.
[0026] In some implementations, the shellspec testing framework can be used in the first phase of testing. In some implementations, the first phase of testing includes pre-checking the hardware, software, system resources, and dependencies of the AI system. Hardware checks include the existence, model, and health status of the accelerator card; software checks include the version and loading status of the accelerator card driver; system resource checks include available disk space, memory capacity, and network connectivity; and dependency checks include the operating system version, kernel header files, compiler, and the installation status of dependent packages. Compilers may include, for example, the gcc compiler, and dependent packages may include, for example, Python dependencies. In the first phase of testing, if dependent packages are not yet installed, they can be installed automatically.
[0027] In step S202, if the first phase test is passed, the AI system is driven to conduct a second phase test on whether the core logic of the AI system functions normally through the test framework.
[0028] The second phase of testing primarily takes place in a stable infrastructure environment, testing the normal functionality of various core logic functions of the AI system. In some implementations, the testing framework used in the second phase may include pytest and shellspec. In other implementations, the second phase of testing includes: accelerator card bandwidth testing, model code unit testing, utility function testing, and critical business logic testing. Specifically, accelerator card bandwidth testing verifies whether the actual computation and memory access bandwidth of the accelerator card meets expectations; model code unit testing verifies the correct execution of model construction functions, weight loading functions, and forward propagation functions; utility function testing verifies the functional correctness of the data preprocessing module, data postprocessing module, and performance indicator calculation module; and critical business logic testing verifies the correct execution of the multimedia decoding logic.
[0029] In step S203, after the second phase test is passed, the artificial intelligence system is driven to deploy the artificial intelligence model, and the third phase test is conducted to check whether the entire chain operation of the artificial intelligence model is smooth.
[0030] The third phase of testing primarily verifies the integration effectiveness of the AI system under the premise that the infrastructure is ready and the core logic functions correctly. For example, it involves applying the AI model to perform inference pipeline testing in an isolated environment. By using the pytest and shellspec testing framework to initiate end-to-end calls, test data is automatically read, the deployed model service interface is called, and inference results are obtained to verify whether the complete data flow from loading, preprocessing, model inference to postprocessing is smooth.
[0031] In step S204, after the third stage test is passed, the artificial intelligence system is driven to conduct a fourth stage test to determine whether the performance of the artificial intelligence system under pressure meets the standard.
[0032] The fourth phase of testing mainly involves high-load stress testing of the deployed artificial intelligence model (e.g., simulating high-concurrency request scenarios based on the shellspec or pytest testing framework), collecting performance indicator data in real time, and evaluating whether the performance indicator data of the artificial intelligence system meets the preset standards by automatically comparing the collected performance indicator data with thresholds.
[0033] In some implementations, the fourth phase of testing includes: collecting performance metrics data from the AI system and comparing the collected data with predefined quality gate thresholds to verify whether the performance metrics meet the threshold requirements. In some implementations, the collected performance metrics data include: throughput (e.g., QPS / TPS / FPS), average response latency, P99 latency, GPU / accelerator card utilization, video memory usage, accelerator card frequency, accelerator card voltage, accelerator card power consumption, accelerator card temperature, and accelerator card fan speed.
[0034] In the fourth phase of testing, if the performance indicators meet the standards, the test is automatically judged as "passed"; if the performance indicators fail to meet the standards, the test is automatically judged as "failed." The fourth phase of testing can also analyze performance issues arising from failure to meet performance indicators and provide feedback.
[0035] In step S205, after the fourth phase test is passed, the entire process log from the first phase test to the fourth phase test is collected in the test system, and the collected logs are analyzed to generate a comprehensive test report.
[0036] Log collection and analysis can be considered the fifth phase of this methodology. It primarily involves collecting all log data from the entire testing process, performing systematic analysis, and generating a comprehensive test report. This comprehensive test report includes the status of each phase, a summary of performance metrics, and links to archived logs, providing data support for issue backtracking and subsequent analysis. The comprehensive test report can also be sent to relevant testing personnel.
[0037] In some implementations, logs include system logs and application logs. For example, logging tools such as dmesg and journalctl can be used to automatically collect logs related to each stage, including information logs and error logs. System logs may include logs related to hardware and drivers; application logs may include logs related to the operation of artificial intelligence models (e.g., inference processes). In some implementations, the analysis of the collected logs includes keyword search analysis, with keywords including general error keywords (e.g., error), execution failure keywords (e.g., fail), and critical error keywords (e.g., coredump).
[0038] In some implementations, method 200 may also include: exiting the test process and providing error feedback if any of the tests in the first to fourth phases fails.
[0039] The method for testing artificial intelligence systems according to embodiments of this application utilizes a "fail-fast" mechanism based on "infrastructure-first verification" to quickly identify and report problems before investing significant resources in testing, saving resources (such as the expensive computing costs and time associated with GPUs / accelerator cards in the cloud). This method systematically covers the entire lifecycle quality dimensions of an artificial intelligence system, from hardware environment, software stack, core logic, integration services to performance, ensuring the overall reliability of the system. The entire process can be seamlessly embedded into CI / CD pipelines, enabling full verification upon submission and significantly improving testing speed. Furthermore, the method provides a solid foundation for performance regression analysis and online problem troubleshooting through automated performance metric data collection and analysis, and complete log archiving.
[0040] Figure 3 This is an application diagram illustrating a method for testing an artificial intelligence system according to an embodiment of this application.
[0041] like Figure 3 As shown, method 300 begins in the test system (such as...) Figure 1 The continuous integration / continuous deployment (CI / CD) pipeline is triggered at the test management platform 101 shown. After the CI / CD pipeline is triggered, the first phase of testing is executed.
[0042] In the first phase of testing, infrastructure verification is performed. This involves driving the AI system to install the software stack via a script such as `prepare.sh`. After the software stack is installed, a testing framework such as `shellspec` is used to test the readiness of the AI system's infrastructure (such as accelerator cards, drivers, software stack, system resources, dependencies, etc.). If the first phase of testing passes, the second phase of testing is executed; otherwise, the testing process is terminated, and error feedback is provided.
[0043] In the second phase of testing, core logic testing is performed, driving the AI system to test the functionality of its core logic (such as accelerator card bandwidth, various utility functions in model code units, and key business logic) using testing frameworks such as pytest and shellspec. If the second phase of testing passes, the third phase of testing is executed; otherwise, the testing process is terminated and error feedback is provided.
[0044] In the third phase of testing, integration testing is performed to drive the deployment of the AI model in the AI system. This phase tests the smooth operation of the AI model across its entire lifecycle (e.g., through end-to-end calls). If the third phase of testing passes, the fourth phase of testing is executed; otherwise, the testing process is terminated and error feedback is provided.
[0045] In the fourth phase of testing, performance testing is performed to test whether the AI system's performance under stress meets the standards. During this phase, performance metrics data are collected through stress testing to determine if performance data meets standards and to pinpoint performance issues. If the fourth phase of testing is passed, the process proceeds to the fifth phase; otherwise, the testing process exits and error feedback is provided.
[0046] In the fifth phase, log collection and analysis are performed. The entire process logs (including system logs and application logs) from the first to the fourth phase of testing are collected in the testing system, and the collected logs are analyzed to generate a comprehensive test report.
[0047] The method for testing an artificial intelligence system according to the embodiments of this application provides a progressive testing scheme combined with a CI / CD pipeline. This scheme enables a systematic interconnected framework for testing the infrastructure, core logic, integration and operation status, performance stress testing, and log reporting of the artificial intelligence system. This allows for deep integration into the CI / CD pipeline, achieving efficient resource utilization, comprehensive coverage of the quality dimensions of the artificial intelligence system, and improved testing efficiency.
[0048] Figure 4 This is a block diagram of a system for testing an artificial intelligence system according to an embodiment of this application. Figure 4 As shown, the system 400 for testing the artificial intelligence system according to an embodiment of this application includes:
[0049] Trigger module 401 is configured to trigger the continuous integration / continuous deployment (CI / CD) pipeline at the test system.
[0050] Infrastructure testing module 402 is configured to drive the artificial intelligence system through a testing framework to perform the first phase of testing on whether the infrastructure of the artificial intelligence system is ready.
[0051] The core logic test module 403 is configured to drive the artificial intelligence system to perform a second-stage test on whether the core logic of the artificial intelligence system functions normally, provided that the first-stage test is passed.
[0052] The integration test module 404 is configured to drive the AI system to deploy the AI model after the second phase test is passed, and to conduct the third phase test to check whether the AI model runs smoothly across the entire chain.
[0053] Performance testing module 405 is configured to drive the AI system to conduct a fourth-stage test to assess whether its performance under stress meets the standards, after the third-stage test is passed; and
[0054] The log analysis module 406 is configured to collect the full-process logs from the first to the fourth stage of testing in the testing system after the fourth stage test is passed, and to analyze the collected logs to generate a comprehensive test report.
[0055] In some implementations, system 400 also includes a feedback module configured to exit the process and provide error feedback if any of the tests in the first to fourth phases of the test fails.
[0056] It should be understood that Figure 4 Each module can be referred to in the preceding method embodiments, and will not be repeated here. The system for testing artificial intelligence systems according to embodiments of this application provides a progressive testing scheme combined with a CI / CD pipeline, implementing the fail-fast principle of testing. Through phased screening, low-level errors are intercepted as early as possible, saving resources. Furthermore, this method provides full-stack verification from infrastructure to business performance, and automates data collection and analysis, improving the quality and efficiency of artificial intelligence system development and delivery.
[0057] Figure 5 This is a system 400 that can perform testing of an artificial intelligence system according to embodiments of this application. Figure 1 A schematic diagram of the computing device of the test management platform 100 shown. Figure 5 As shown, computing device 500 may include bus 502 or other communication mechanism for transmitting information, and one or more processors 504 coupled to bus 502 for processing information. The one or more processors 504 may include, for example, one or more general-purpose microprocessors.
[0058] like Figure 5As shown, in some implementations, computing device 500 may further include main memory 506 coupled to bus 502. Main memory 506 is used to store information (e.g., a historical database) and instructions executed by one or more processors 504, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 504. When these instructions are stored in storage media accessible to one or more processors 504, they can cause computing device 500 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 508 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical disks or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tapes, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.
[0059] like Figure 5 As shown, in some embodiments, computing device 500 may further include one or more communication interfaces or network interfaces 510 coupled to bus 502. Network interface 510 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 510 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.
[0060] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.
[0061] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.
[0062] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.
[0063] According to embodiments of this application, a system is provided that includes a processor and a non-transitory computer-readable storage medium storing instructions, which are executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. According to embodiments of this application, a non-transitory computer-readable storage medium or computer program product storing instructions, which are executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.
[0064] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Certain blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, certain components may be added to, removed from, or rearranged compared to the disclosed example embodiments.
[0065] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly turn the computer into an execution function but may learn from training data to produce a predictive model of the execution function.
[0066] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.
[0067] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).
[0068] Throughout this specification, although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configurations may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document. Therefore, this specification and its drawings should be considered illustrative rather than restrictive.
[0069] As used herein, “or” is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, “A, B, or C” means “A, B, A and B, A and C, B and C, or A, B, and C” unless explicitly indicated by the context. Furthermore, “and” is combined and separate unless explicitly indicated by the context. Therefore, in this document, “A and B” means “A and B, combined or separate” unless explicitly indicated by the context.
[0070] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not exclude the presence of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments necessarily require a certain feature, component, and / or step in any way, or that one or more embodiments must include such features, components, and / or steps.
[0071] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.
[0072] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, the term "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their equivalents.
Claims
1. A method for testing an artificial intelligence system, characterized by include: After triggering the continuous integration / continuous deployment (CI / CD) pipeline at the test system, the AI system is driven to conduct a first-phase test on whether the infrastructure of the AI system is ready through the test framework. The test framework used in the first-phase test is the shellspec test framework. If the first phase of testing is passed, the AI system is driven to conduct a second phase of testing on whether the core logic of the AI system functions normally through a testing framework. The testing framework used in the second phase of testing includes the pytest testing framework or the shellspec testing framework. After the second phase of testing is passed, the artificial intelligence system is driven to deploy an artificial intelligence model, and the third phase of testing is conducted to check whether the entire chain operation of the artificial intelligence model is smooth. After the third stage test is passed, the artificial intelligence system is driven to conduct a fourth stage test to determine whether the performance of the artificial intelligence system under pressure meets the standard. as well as After the fourth phase of testing is passed, the entire process log from the first phase of testing to the fourth phase of testing is collected in the testing system, and the collected logs are analyzed to generate a comprehensive test report. The method further includes: If any stage of the test, from the first to the fourth stage, fails, the testing process will be terminated and error feedback will be provided. The second phase of testing includes: Accelerator card bandwidth testing is used to verify whether the actual computing and memory access bandwidth of the accelerator card meets expectations; Model code unit tests are used to verify the correctness of the execution of model building functions, weight loading functions, and forward propagation functions; The utility function test is used to verify the correct functionality of the data preprocessing module, data postprocessing module, and indicator performance calculation module; and Critical business logic testing is used to verify the correct execution of multimedia decoding logic.
2. The method of claim 1, wherein, The CI / CD pipeline is triggered in one of the following ways: when a tester submits test code to the CI / CD pipeline, automatically at a predetermined time, or manually by the tester.
3. The method according to claim 1, characterized in that, The first phase of testing includes: conducting preliminary checks on the hardware, software, system resources, and dependencies of the artificial intelligence system.
4. The method according to claim 3, characterized in that, The hardware information includes: the presence, model, and health status of the accelerator card; The software information includes: the version and loading status of the accelerator card driver; The system resource information includes: available disk space, memory capacity, and network connectivity; and The dependencies include: the operating system version, kernel header files, compiler, and the installation status of dependency packages.
5. The method according to claim 1, characterized in that, The fourth stage of testing includes: collecting performance index data of the artificial intelligence system and comparing the collected data with predefined quality access control thresholds to verify whether the performance indexes meet the standards.
6. The method according to claim 5, characterized in that, The collected performance metrics include: throughput, average response latency, P99 latency, GPU / accelerator card utilization, video memory usage, accelerator card frequency, accelerator card voltage, accelerator card power consumption, accelerator card temperature, and accelerator card fan speed.
7. The method according to claim 1, characterized in that, The logs include system logs and application logs.
8. The method according to claim 1, characterized in that, The analysis of the collected logs includes: performing keyword retrieval analysis on the logs, whereby the keywords include general error keywords, execution failure keywords, and critical error keywords.
9. A system for testing artificial intelligence systems, characterized in that... include: The trigger module is configured to trigger the continuous integration / continuous deployment (CI / CD) pipeline at the test system. The infrastructure testing module is configured to drive the artificial intelligence system to perform a first-phase test on whether the infrastructure of the artificial intelligence system is ready through a testing framework. The testing framework used in the first-phase test is the shellspec testing framework. The core logic testing module is configured to drive the artificial intelligence system to perform a second-stage test on the functionality of the core logic of the artificial intelligence system through a testing framework if the first-stage test is passed. The testing framework used in the second-stage test includes the pytest testing framework or the shellspec testing framework. The integration testing module is configured to drive the AI system to deploy an AI model after the second phase test is passed, and to conduct a third phase test to check whether the AI model runs smoothly across the entire chain. The performance testing module is configured to drive the artificial intelligence system to conduct a fourth-stage test on whether the performance of the artificial intelligence system under stress meets the standard after the third-stage test is passed. as well as The log analysis module is configured to collect the complete logs from the first to the fourth stage of testing in the testing system after the fourth stage test is passed, and to analyze the collected logs to generate a comprehensive test report. The system also includes: The feedback module is configured to exit the process and provide error feedback if any stage of the test (from the first stage to the fourth stage) fails. The second phase of testing includes: Accelerator card bandwidth testing is used to verify whether the actual computing and memory access bandwidth of the accelerator card meets expectations; Model code unit tests are used to verify the correctness of the execution of model building functions, weight loading functions, and forward propagation functions; The utility function test is used to verify the correct functionality of the data preprocessing module, data postprocessing module, and indicator performance calculation module; and Critical business logic testing is used to verify the correct execution of multimedia decoding logic.