Configurable NPU general test framework and test method
By building a configurable NPU general testing framework, the problem of independent development of NPU test programs from different manufacturers has been solved, and a standardized testing process across manufacturers has been realized, improving testing efficiency and compatibility, and supporting NPU hardware testing in diverse scenarios.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-04-10
AI Technical Summary
Existing NPU testing solutions require the development of independent test programs for different manufacturers, resulting in low development efficiency, long adaptation cycles, lack of comparability of test results, difficulty in supporting multi-batch inference and multi-threaded parallel testing, and inability to meet the testing needs of different application scenarios.
This paper provides a configurable general testing framework and method for NPUs. It receives user requirements through the application layer, encapsulates common logic through the core framework layer, defines standardized interfaces through the adaptation layer, and provides an SDK through the hardware layer to achieve cross-vendor hardware adaptation, shield the underlying differences, and provide a standardized testing process.
It has achieved standardization of cross-vendor NPU testing, reduced testing costs, improved adaptation efficiency, enhanced testing efficiency and compatibility, and supported multi-threaded and multi-instance parallel testing to meet diverse scenario requirements.
Smart Images
Figure CN121833360A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence hardware testing technology, and more specifically, to a configurable general-purpose NPU testing framework and testing method, electronic devices, and computer-readable storage media. Background Technology
[0002] With the development of artificial intelligence technology, NPUs, as hardware accelerators specifically designed for neural network computing, have been widely applied in scenarios such as autonomous driving, smart terminals, and edge computing. Currently, there are numerous NPU manufacturers on the market, each equipped with its own proprietary software development kit (SDK), resulting in significant differences in their model loading, inference execution, and hardware interaction interfaces. Existing NPU testing solutions generally suffer from the following problems: Independent test programs need to be developed for different manufacturers' NPUs, requiring repetitive coding of identical test logic, leading to low development efficiency. Adding a new NPU model necessitates modifying the core logic of the test program, such as replacing the model loading interface and adjusting hardware parameter configurations, resulting in a long adaptation cycle. Inconsistent performance statistics dimensions and functional verification logic across different test programs lead to a lack of comparability in test results between different NPUs. Furthermore, they struggle to flexibly support complex scenarios such as multi-batch inference and multi-threaded parallel testing, failing to meet the testing needs of diverse application scenarios.
[0003] Therefore, there is an urgent need for a universal testing framework that can extract common logic for NPU testing and adapt to hardware from different manufacturers through configurable interfaces, in order to reduce testing costs, improve adaptation efficiency, and achieve a unified testing standard.
[0004] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0005] The purpose of this disclosure is to provide a configurable general-purpose NPU test framework and test method, electronic device and computer-readable storage medium, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.
[0006] According to one aspect of this disclosure, a configurable general testing method for NPUs is provided, comprising:
[0007] The application layer receives user test requirements, parses the configuration file, and initializes the scheduling, logging, and performance statistics modules of the core framework.
[0008] The hardware layer provides a software development kit (SDK) for the neural network processor (NPU) hardware;
[0009] The adaptation layer dynamically loads the manufacturer's adaptation library based on the configuration information parsed from the user's testing requirements and calls the hardware initialization interface of the hardware layer.
[0010] The core framework layer performs common logic tests based on task requirements.
[0011] In one exemplary embodiment of this disclosure, the method further includes implementation based on the adaptation layer:
[0012] Create the device and initialize the memory pool;
[0013] Bind the computing core and start the hardware scheduler;
[0014] It provides feedback on hardware connectivity, computing unit health, and memory integrity.
[0015] Configure the hardware operating mode and start the NPU device.
[0016] In one exemplary embodiment of this disclosure, the method further includes:
[0017] The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances;
[0018] The core framework layer scheduling module allocates inference tasks through a thread pool based on the "test task type";
[0019] The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU;
[0020] The core framework layer calls the inference execution interface to complete the calculation;
[0021] The core framework layer calls the output tensor acquisition interface and saves the result to the output path;
[0022] The core framework layer performance module summarizes metrics such as FPS, latency, and utilization; the core framework layer verification module compares the output with the reference results to determine the correctness of the inference.
[0023] The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
[0024] In one aspect of this disclosure, a configurable general-purpose NPU testing framework is provided, the general-purpose NPU testing framework comprising an application layer, a core framework layer, an adaptation layer, and a hardware layer, wherein:
[0025] The application layer receives user test requirements, parses the configuration file, and initializes the scheduling, logging, and performance statistics modules of the core framework.
[0026] The hardware layer is the target NPU to be tested, consisting of NPU hardware and SDK provided by the manufacturer.
[0027] The adaptation layer dynamically loads the vendor's adaptation library based on the application layer configuration information, calls the hardware initialization interface of the hardware layer, and starts the NPU device.
[0028] The core framework layer is used to encapsulate the common logic of NPU testing, provide a standardized testing process, and shield the differences in underlying hardware. The core framework layer consists of a test task scheduling module, an input / output processing module, a performance statistics module, a result verification module, and a log monitoring module.
[0029] In one exemplary embodiment of this disclosure, the application layer of the NPU general testing framework includes:
[0030] The application layer receives user test requests via a configuration file and adjusts test parameters accordingly. The configuration file contains the following core configuration items:
[0031] The adaptation layer type corresponds to different NPU manufacturers, NPU device IDs, multi-device scenarios, model paths, input data paths, output result paths, reference output paths, inference batch size, number of parallel instances, total number of inferences, result verification type, accuracy threshold, thread pool size, and test task type.
[0032] In one exemplary embodiment of this disclosure, the core framework layer of the NPU general testing framework includes:
[0033] The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances;
[0034] The core framework layer scheduling module allocates inference tasks through a thread pool based on the "test task type";
[0035] The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU;
[0036] The core framework layer calls the inference execution interface to complete the calculation; the core framework layer calls the output tensor acquisition interface to save the result to the output path;
[0037] The core framework layer performance modules summarize metrics such as FPS, latency, and utilization.
[0038] The core framework layer verification module compares the output with the reference result to determine the correctness of the reasoning.
[0039] The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
[0040] In one exemplary embodiment of this disclosure, the core framework layer of the NPU general testing framework is designed as a common module, encapsulating all the logic required for NPU testing, including:
[0041] The test task scheduling module schedules test tasks based on the number of instances and models in the configuration file, using a thread pool or asynchronous task scheduling; it provides a task interruption mechanism, supports stopping the test via a signal, and ensures that resources are released normally; it enables multi-threaded and multi-instance parallel testing, and supports single-model multi-instance and multi-model parallel scenarios;
[0042] The input / output processing module reads test data from configuration parameters and automatically parses the data dimensions; it calls the "tensor format conversion interface" of the adaptation layer to convert the input data into a format supported by the target NPU; it saves the inference output to the output path in a unified format and supports configuration of whether to save intermediate results; by uniformly processing the loading, format conversion and result saving of test data, it shields the tensor format differences of different NPUs.
[0043] The performance statistics module includes core metrics such as throughput, latency, resource utilization, memory bandwidth, and timing accuracy.
[0044] The result verification module compares the inference output file with the standard file in the reference output path to check whether the file size and binary content are consistent; for tasks such as classification and detection, it calculates the accuracy and mAP of the inference results to verify the correctness of the NPU inference results.
[0045] The log monitoring module defines a unified log macro, and the output log includes timestamp, module name, and log level content, providing standardized log output and supporting different levels of log control.
[0046] In one exemplary embodiment of this disclosure, the adaptation layer of the NPU general testing framework includes:
[0047] The adaptation layer features a standardized interface design, defining standardized interfaces for hardware initialization and destruction, model loading and unloading, inference instance creation and destruction, input data setting, inference execution, and obtaining output tensors.
[0048] In one aspect of this disclosure, an electronic device is provided, comprising:
[0049] Processor; and
[0050] A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of the preceding claims.
[0051] In one aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method according to any one of the preceding claims.
[0052] An exemplary embodiment of this disclosure provides a configurable general-purpose NPU testing method. The method includes: an application layer receiving a user test requirement parsing configuration file and initializing the scheduling, logging, and performance statistics modules of the core framework; a hardware layer providing a software development kit (SDK) for the neural network processor (NPU) hardware; an adaptation layer dynamically loading a vendor adaptation library based on the configuration information parsed from the user test requirements and calling the hardware initialization interface of the hardware layer; and a core framework layer performing common logic testing based on task requirement parsing. This disclosure addresses the pain points of existing NPU testing through parameterized requirement configuration, common module encapsulation, and differentiated interface adaptation testing methods. It provides a standardized testing process, shields underlying hardware differences, and can be applied to NPU hardware testing in numerous scenarios, reducing testing costs, improving adaptation efficiency, achieving unified testing standards, and significantly improving the efficiency and compatibility of NPU testing in complex scenarios.
[0053] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description
[0054] The above and other features and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments thereof with reference to the accompanying drawings.
[0055] Figure 1 A flowchart illustrating a configurable general testing method for an NPU according to an exemplary embodiment of the present disclosure is shown;
[0056] Figure 2 A logic flowchart of a configurable general testing method for an NPU according to an exemplary embodiment of the present disclosure is shown;
[0057] Figure 3 A schematic block diagram of a configurable general-purpose NPU testing framework according to an exemplary embodiment of the present disclosure is shown;
[0058] Figure 4 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown schematically;
[0059] Figure 5 The illustration shows a schematic diagram of a computer-readable storage medium according to an exemplary embodiment of the present disclosure. Detailed Implementation
[0060] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.
[0061] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details described, or other methods, components, materials, apparatuses, steps, etc., can be employed. In other instances, well-known structures, methods, apparatuses, implementations, materials, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.
[0062] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more software-hardened modules, or in different network and / or processor devices and / or microcontroller devices.
[0063] In this example embodiment, a configurable general testing method for NPUs is first provided; refer to Figure 1 As shown, this configurable general-purpose NPU testing method may include the following steps:
[0064] Step S110: The application layer receives the user's test requirement parsing configuration file and initializes the scheduling, logging, and performance statistics modules of the core framework.
[0065] Step S120: The hardware layer provides a software development kit (SDK) for the neural network processor (NPU) hardware.
[0066] Step S130: The adaptation layer dynamically loads the manufacturer's adaptation library based on the configuration information parsed from the user's testing requirements and calls the hardware initialization interface of the hardware layer.
[0067] Step S140: The core framework layer performs common logic tests based on the task requirements.
[0068] An exemplary embodiment of this disclosure provides a configurable general-purpose NPU testing method. The method includes: an application layer receiving a user test requirement parsing configuration file and initializing the scheduling, logging, and performance statistics modules of the core framework; a hardware layer providing a software development kit (SDK) for the neural network processor (NPU) hardware; an adaptation layer dynamically loading a vendor adaptation library based on the configuration information parsed from the user test requirements and calling the hardware initialization interface of the hardware layer; and a core framework layer performing common logic testing based on task requirement parsing. This disclosure addresses the pain points of existing NPU testing through parameterized requirement configuration, common module encapsulation, and differentiated interface adaptation testing methods. It provides a standardized testing process, shields underlying hardware differences, and can be applied to NPU hardware testing in numerous scenarios, reducing testing costs, improving adaptation efficiency, achieving unified testing standards, and significantly improving the efficiency and compatibility of NPU testing in complex scenarios.
[0069] The following will further explain a configurable general testing method for NPU in this example embodiment.
[0070] Example 1:
[0071] In step S110, the application layer can receive the user test requirement parsing configuration file and initialize the scheduling, logging, and performance statistics modules of the core framework.
[0072] In this example embodiment, the method further includes implementation based on the adaptation layer:
[0073] Create the device and initialize the memory pool;
[0074] Bind the computing core and start the hardware scheduler;
[0075] It provides feedback on hardware connectivity, computing unit health, and memory integrity.
[0076] Configure the hardware operating mode and start the NPU device.
[0077] In step S120, a software development kit (SDK) for the neural network processor (NPU) hardware can be provided at the hardware layer.
[0078] In step S130, the adaptation layer can dynamically load the manufacturer's adaptation library based on the configuration information parsed from the user's testing requirements and call the hardware initialization interface of the hardware layer.
[0079] In this example embodiment, the method further includes:
[0080] The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances;
[0081] The core framework layer scheduling module allocates inference tasks through a thread pool based on the "test task type";
[0082] The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU;
[0083] The core framework layer calls the inference execution interface to complete the calculation;
[0084] The core framework layer calls the output tensor acquisition interface and saves the result to the output path;
[0085] The core framework layer performance module summarizes metrics such as FPS, latency, and utilization; the core framework layer verification module compares the output with the reference results to determine the correctness of the inference.
[0086] The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
[0087] In step S140, the core framework layer can perform common logic tests based on the task requirements.
[0088] Example 2:
[0089] In the implementation of this example, as Figure 2 The diagram shown is a test flowchart of the present invention.
[0090] Step 1.1 describes the application layer receiving the user's test requirement parsing configuration file and initializing the core framework's scheduling, logging, and performance statistics modules;
[0091] The hardware layer described in step 1.2 is provided as an NPU hardware SDK.
[0092] Step 1.3, the adaptation layer dynamically loads the vendor's adaptation library based on the configuration information parsed from the user's testing requirements, and calls the hardware initialization interface of the hardware layer. Step 1.3 includes the following steps:
[0093] Step 1.3.1 describes the adaptation layer creating a device and initializing the memory pool;
[0094] Step 1.3.2 describes binding the adaptation layer to the computing core and starting the hardware scheduler;
[0095] Step 1.3.3 describes the adaptation layer providing feedback on hardware connectivity, computing unit health, and memory integrity.
[0096] Step 1.3.4 describes configuring the hardware operating mode of the adaptation layer and starting the NPU device.
[0097] Step 1.4 describes the core framework layer performing common logic tests based on task requirements. Step 1.4 includes the following steps:
[0098] Step 1.4.1 describes the core framework layer calling the model loading interface to load the target model, and calling the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances;
[0099] Step 1.4.2 describes the core framework layer scheduling module allocating inference tasks through a thread pool based on the "test task type";
[0100] Step 1.4.3 describes the core framework layer input processing module loading data, converting formats, and calling the input data setting interface to pass it to the NPU;
[0101] Step 1.4.4 describes the core framework layer calling the inference execution interface to complete the computation;
[0102] Step 1.4.5 describes the core framework layer calling the output tensor acquisition interface and saving the result to the output path;
[0103] Step 1.4.6 summarizes performance metrics such as FPS, latency, and utilization in the core framework layer performance module; the core framework layer verification module compares the output with the reference results to determine the correctness of the inference.
[0104] Step 1.4.7 describes the core framework layer releasing resources in the order of "destroy instance → unload model → hardware destruction".
[0105] In this example implementation, the present invention provides a general testing framework and method for the aforementioned configurable NPU. The application layer receives user test requests, provides configuration entry points and test result output paths, while the core framework layer encapsulates the common logic of NPU testing, provides a standardized testing process, and shields the underlying hardware differences. The core framework layer consists of a test task scheduling module, an input / output processing module, a performance statistics module, a result verification module, and a log monitoring module. An adaptation layer defines standardized adaptation interfaces, implemented by adaptation code for NPUs from different vendors, to complete the interface between the core framework and the hardware, including hardware adaptation interfaces, model loading and execution interfaces, and tensor format conversion interfaces. The hardware layer provides the NPU hardware and the vendor-provided SDK.
[0106] In this example implementation, the present invention addresses the pain points of existing NPU testing through parameterized requirement configuration, common module encapsulation, and differentiated interface adaptation testing framework and method. The present invention provides a standardized testing process, shields underlying hardware differences, and can be widely applied to NPU hardware testing in many scenarios, reducing testing costs, improving adaptation efficiency, and achieving unified testing standards, significantly improving the efficiency and compatibility of NPU testing in complex scenarios.
[0107] It should be noted that although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.
[0108] Furthermore, this example embodiment also provides a configurable general-purpose NPU testing framework. (Refer to...) Figure 4 As shown, this configurable NPU general testing framework includes an application layer, a core framework layer, an adaptation layer, and a hardware layer, wherein:
[0109] The application layer receives user test requirements, parses the configuration file, and initializes the scheduling, logging, and performance statistics modules of the core framework.
[0110] The hardware layer is the target NPU to be tested, consisting of NPU hardware and SDK provided by the manufacturer.
[0111] The adaptation layer dynamically loads the vendor's adaptation library based on the application layer configuration information, calls the hardware initialization interface of the hardware layer, and starts the NPU device.
[0112] The core framework layer is used to encapsulate the common logic of NPU testing, provide a standardized testing process, and shield the differences in underlying hardware. The core framework layer consists of a test task scheduling module, an input / output processing module, a performance statistics module, a result verification module, and a log monitoring module.
[0113] In this example implementation, the application layer of the NPU general testing framework includes:
[0114] The application layer receives user test requests via a configuration file and adjusts test parameters accordingly. The configuration file contains the following core configuration items:
[0115] The adaptation layer type corresponds to different NPU manufacturers, NPU device IDs, multi-device scenarios, model paths, input data paths, output result paths, reference output paths, inference batch size, number of parallel instances, total number of inferences, result verification type, accuracy threshold, thread pool size, and test task type.
[0116] In this example implementation, the core framework layer of the NPU general testing framework includes:
[0117] The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances;
[0118] The core framework layer scheduling module allocates inference tasks through a thread pool based on the "test task type";
[0119] The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU;
[0120] The core framework layer calls the inference execution interface to complete the calculation; the core framework layer calls the output tensor acquisition interface to save the result to the output path;
[0121] The core framework layer performance modules summarize metrics such as FPS, latency, and utilization.
[0122] The core framework layer verification module compares the output with the reference result to determine the correctness of the reasoning.
[0123] The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
[0124] In this example implementation, the core framework layer of the NPU general testing framework is designed as a common module, encapsulating all the logic required for NPU testing, including:
[0125] The test task scheduling module schedules test tasks based on the number of instances and models in the configuration file, using a thread pool or asynchronous task scheduling; it provides a task interruption mechanism, supports stopping the test via a signal, and ensures that resources are released normally; it enables multi-threaded and multi-instance parallel testing, and supports single-model multi-instance and multi-model parallel scenarios;
[0126] The input / output processing module reads test data from configuration parameters and automatically parses the data dimensions; it calls the "tensor format conversion interface" of the adaptation layer to convert the input data into a format supported by the target NPU; it saves the inference output to the output path in a unified format and supports configuration of whether to save intermediate results; by uniformly processing the loading, format conversion and result saving of test data, it shields the tensor format differences of different NPUs.
[0127] The performance statistics module includes core metrics such as throughput, latency, resource utilization, memory bandwidth, and timing accuracy.
[0128] The result verification module compares the inference output file with the standard file in the reference output path to check whether the file size and binary content are consistent; for tasks such as classification and detection, it calculates the accuracy and mAP of the inference results to verify the correctness of the NPU inference results.
[0129] The log monitoring module defines a unified log macro, and the output log includes timestamp, module name, and log level content, providing standardized log output and supporting different levels of log control.
[0130] In this example implementation, the adaptation layer of the NPU general testing framework includes:
[0131] The adaptation layer features a standardized interface design, defining standardized interfaces for hardware initialization and destruction, model loading and unloading, inference instance creation and destruction, input data setting, inference execution, and obtaining output tensors.
[0132] Example 3:
[0133] In the embodiments of this example, as Figure 3 As shown, the testing framework includes an application layer, a core framework layer, an adaptation layer, and a hardware layer.
[0134] The application layer receives user test requirements, parses the configuration file, and initializes the scheduling, logging, and performance statistics modules of the core framework.
[0135] The hardware layer is the target NPU to be tested, consisting of the NPU hardware and the SDK provided by the manufacturer.
[0136] The adaptation layer dynamically loads the vendor's adaptation library based on the application layer configuration information, calls the hardware initialization interface of the hardware layer, and starts the NPU device.
[0137] The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances. The core framework layer scheduling module allocates inference tasks through a thread pool according to the "test task type". The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU. The core framework layer calls the inference execution interface to complete the calculation. The core framework layer calls the output tensor acquisition interface to save the results to the output path. The core framework layer performance module summarizes indicators such as FPS, latency, and utilization. The core framework layer verification module compares the output with the reference results to determine the correctness of the inference. The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
[0138] In this example embodiment, addressing the problems of "low reusability, high adaptation cost, and inconsistent standards" in existing NPU testing solutions, this invention provides a configurable general NPU testing framework and method. It extracts common modules in NPU testing to form a reusable core framework; designs an extensible adaptation interface, requiring only the development of adaptation layer code for NPUs from different manufacturers without modifying the core framework; and supports flexible definition of test parameters through configuration files to adapt to different testing scenarios.
[0139] In this example embodiment, the present invention constructs a four-layer architecture of "application layer - core framework layer - adaptation layer - hardware layer", and achieves general testing through "common module encapsulation + differential interface adaptation".
[0140] In this example embodiment, the application layer receives user test requests, provides configuration entry points and test result display, and consists of a configuration file parsing module and a test result output module. The core framework layer encapsulates the common logic of NPU testing, provides a standardized test process, and shields the underlying hardware differences. The core framework layer consists of a test task scheduling module, an input / output processing module, a performance statistics module, a result verification module, and a log monitoring module. The adaptation layer defines standardized adaptation interfaces, implemented by adaptation code for NPUs from different vendors, to complete the interface between the core framework and the hardware, including hardware adaptation interfaces, model loading and execution interfaces, and tensor format conversion interfaces. The hardware layer is the target NPU to be tested, consisting of the NPU hardware and the SDK provided by the vendor.
[0141] In this example embodiment, the application layer receives user test requirements through a configuration file and adjusts test parameters. The configuration file contains the following core configuration items: adaptation layer type (corresponding to different NPU vendors), NPU device ID (corresponding to multi-device scenarios), model path, input data path, output result path, reference output path, inference batch size, number of parallel instances, total inference times, result verification type, accuracy threshold, thread pool size, and test task type.
[0142] In this example embodiment, the core framework layer is designed with common modules that encapsulate all the logic required for NPU testing, specifically including:
[0143] (1) The test task scheduling module schedules test tasks based on the number of instances and models in the configuration file, using a thread pool or asynchronous task scheduling mechanism; it provides a task interruption mechanism, supporting stopping the test via signals to ensure normal resource release. It enables multi-threaded and multi-instance parallel testing, supporting scenarios such as single-model multi-instance and multi-model parallel testing.
[0144] (2) The input / output processing module reads test data from configuration parameters and automatically parses the data dimensions; it calls the "tensor format conversion interface" of the adaptation layer to convert the input data into a format supported by the target NPU; and it saves the inference output to the output path in a unified format, supporting configuration of whether to save intermediate results. By uniformly processing the loading, format conversion, and result saving of test data, the differences in tensor formats between different NPUs are shielded.
[0145] (3) Performance statistics module, the core indicators include hardware indicators such as throughput, latency, resource utilization, memory bandwidth and timing accuracy.
[0146] (4) Result verification module: compare the inference output file with the standard file in the reference output path to check whether the file size and binary content are consistent; for tasks such as classification and detection, calculate the accuracy and mAP of the inference results to verify the correctness of the NPU inference results.
[0147] (5) The log monitoring module defines a unified log macro, and the output log includes "timestamp, module name, log level and content", providing standardized log output and supporting different levels of log control.
[0148] In this example embodiment, the adaptation layer features a standardized interface design, defining six standardized interfaces. NPUs from different vendors only need to implement the corresponding interface adaptation code to access the core framework. Specifically, this includes:
[0149] (1) Hardware initialization and destruction
[0150] Hardware initialization: Parse the vendor type, determine the corresponding SDK initialization interface to call, initialize the device memory pool, bind the computing core, start the hardware scheduler, check the device connection status, computing unit health and memory integrity, and enable specific hardware modes.
[0151] Hardware destruction: Confirm that all models have been unloaded and inference instances have been destroyed (force release any remnants); release the device memory pool, shut down the computing core power, and stop the hardware scheduler; call the vendor SDK's destruction interface to disconnect from the hardware.
[0152] (2) Model loading and unloading
[0153] Model loading reads the corresponding file based on the model path, parses the network structure, weight data, and input / output tensor specifications; converts the model into the NPU native instruction set; moves the model weights from host memory to NPU device memory and quantizes them according to hardware support; generates a model handle, and records metadata such as the model's memory address, computation graph information, and input / output dimensions on the device.
[0154] Model unloading first checks the validity of the model handle (whether it has been loaded or not referenced by an instance); it reclaims the model's weight data, computation graph structure, and instruction cache in the device's memory; and it marks the model handle as invalid to prevent secondary use.
[0155] (3) Creation and destruction of reasoning instances
[0156] Allocate independent input / output buffers (device memory), temporary computation stacks, and instruction queues for loaded model instances; adjust buffer sizes according to batch size; bind instances to the NPU's hardware flow to ensure inference tasks are executed in sequence; and set parameters such as queue length and task priority according to instance configuration parameters.
[0157] After the test is completed, ensure that the instance is not performing inference (if it is running, wait for it to finish or force termination); release the instance's input / output buffers, temporary memory, hardware stream bindings, and mark the instance handle as invalid.
[0158] (4) Input data settings
[0159] Check if the tensor shape, data type, and data arrangement order are consistent with the model input specifications; call the vendor's SDK quantization interface to perform conversion according to the data requirements of the host and model sides. Copy the input data from the host memory to the instance's device input buffer via direct memory access; verify the consistency between the transmitted data and the source data.
[0160] (5) Reasoning execution
[0161] Confirm that the instance is ready, check that the input data has been set and no tasks are currently executing; submit the model's computation instruction sequence to the hardware stream bound to the instance; block and wait for inference to complete; check for any computational errors that occurred during the inference process.
[0162] (6) Obtain the output tensor (shape, data)
[0163] After inference is complete, the type and shape of the output tensor are read from the instance metadata, the data in the device output buffer is copied to the host memory pointed to by the output data through direct memory access, and then converted into a common format according to the host requirements.
[0164] This invention provides a configurable general-purpose NPU testing framework based on the above-mentioned features. By constructing a four-layer architecture of "application layer - core framework layer - adaptation layer - hardware layer," and combining the concepts of common encapsulation and differential adaptation, it achieves standardized testing of NPUs from different vendors. The framework receives user configuration requirements and displays results through the application layer. The core framework layer encapsulates common testing logic to implement functions such as task scheduling, IO processing, performance statistics, result verification, and log monitoring. The adaptation layer defines six standardized interfaces (hardware initialization and destruction, model loading and unloading, inference instance management, input settings, inference execution, and output acquisition). The hardware layer connects to the target NPU and the vendor's SDK, forming a complete testing chain. The standardized interfaces of the adaptation layer shield hardware differences, allowing for seamless integration upon adaptation. The core framework layer unifies testing processes and metrics, supporting multi-threaded, multi-instance parallel testing and diverse task scenarios. Configuration files allow for flexible customization of test parameters to meet different scenario requirements. A comprehensive performance statistics and result verification mechanism ensures the comprehensiveness and accuracy of the tests. This invention effectively solves the problems of non-universal NPU testing tools from multiple vendors, high adaptation costs, and low code reuse, significantly improving NPU testing efficiency and compatibility, and can be widely applied to functional verification and performance evaluation scenarios of various NPUs.
[0165] It should be noted that although several modules or units of a configurable NPU general testing framework 0 are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0166] Furthermore, in an exemplary embodiment of this disclosure, an electronic device capable of implementing the above-described method is also provided.
[0167] Those skilled in the art will understand that various aspects of the present invention can be implemented as systems, methods, or program products. Therefore, various aspects of the present invention can be specifically implemented as entirely hardware embodiments, entirely software embodiments (including firmware, microcode, etc.), or embodiments combining hardware and software aspects, collectively referred to herein as “circuit,” “module,” or “system.”
[0168] The following reference Figure 4 To describe an electronic device 500 according to such an embodiment of the present invention. Figure 4 The electronic device 500 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.
[0169] like Figure 4 As shown, the electronic device 500 is manifested in the form of a general-purpose computing device. The components of the electronic device 500 may include, but are not limited to: at least one processing unit 510, at least one storage unit 520, a bus 530 connecting different system components (including storage unit 520 and processing unit 510), and a display unit 540.
[0170] The storage unit stores program code that can be executed by the processing unit 510, causing the processing unit 510 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention. For example, the processing unit 510 can perform actions such as... Figure 1 Steps S110 to S140 are shown in the diagram.
[0171] Storage unit 520 may include a readable medium in the form of a volatile storage unit, such as random access memory (RAM) 5201 and / or cache memory 5202, and may further include a read-only memory (ROM) 5203.
[0172] Storage unit 520 may also include a program / utility 5204 having a set (at least one) program module 5205, such program module 5205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.
[0173] Bus 530 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.
[0174] Electronic device 500 can also communicate with one or more external devices 570 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 500, and / or with any device that enables electronic device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 550. Furthermore, electronic device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 560. As shown, network adapter 560 communicates with other modules of electronic device 500 via bus 530. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.
[0175] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.
[0176] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored. In some possible embodiments, various aspects of the invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps of the various exemplary embodiments of the invention described in the "Exemplary Methods" section above.
[0177] refer to Figure 5 As shown, a program product 600 for implementing the above-described method according to an embodiment of the present invention is described. It may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.
[0178] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0179] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with an instruction execution system, apparatus, or device.
[0180] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.
[0181] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages such as Java and C++, and conventional procedural programming languages such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).
[0182] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0183] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.
[0184] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.
Claims
1. A configurable general testing method for NPUs, characterized in that, The method includes: The application layer receives user test requirements, parses the configuration file, and initializes the scheduling, logging, and performance statistics modules of the core framework. The hardware layer provides a software development kit (SDK) for the neural network processor (NPU) hardware; The adaptation layer dynamically loads the manufacturer's adaptation library based on the configuration information parsed from the user's testing requirements and calls the hardware initialization interface of the hardware layer. The core framework layer performs common logic tests based on task requirements.
2. The method as described in claim 1, characterized in that, The method also includes implementation based on the adaptation layer: Create the device and initialize the memory pool; Bind the computing core and start the hardware scheduler; It provides feedback on hardware connectivity, computing unit health, and memory integrity. Configure the hardware operating mode and start the NPU device.
3. The method as described in claim 1, characterized in that, The method further includes: The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances; The core framework layer scheduling module allocates inference tasks through a thread pool based on the "test task type"; The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU; The core framework layer calls the inference execution interface to complete the calculation; The core framework layer calls the output tensor acquisition interface and saves the result to the output path; The core framework layer performance module summarizes metrics such as FPS, latency, and utilization; the core framework layer verification module compares the output with the reference results to determine the correctness of the inference. The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
4. A configurable general-purpose NPU testing framework, characterized in that, The NPU general testing framework includes an application layer, a core framework layer, an adaptation layer, and a hardware layer, wherein: The application layer receives user test requirements, parses the configuration file, and initializes the scheduling, logging, and performance statistics modules of the core framework. The hardware layer is the target NPU to be tested, consisting of NPU hardware and SDK provided by the manufacturer. The adaptation layer dynamically loads the vendor's adaptation library according to the application layer configuration information, calls the hardware initialization interface of the hardware layer, and starts the NPU device. The core framework layer is used to encapsulate the common logic of NPU testing, provide a standardized testing process, and shield the differences in underlying hardware. The core framework layer consists of a test task scheduling module, an input / output processing module, a performance statistics module, a result verification module, and a log monitoring module.
5. The NPU general testing framework as described in claim 4, characterized in that, The application layer of the NPU general testing framework includes: The application layer receives user test requests through a configuration file and adjusts test parameters. The configuration file contains the following core configuration items: The adaptation layer type corresponds to different NPU manufacturers, NPU device IDs, multi-device scenarios, model paths, input data paths, output result paths, reference output paths, inference batch size, number of parallel instances, total number of inferences, result verification type, accuracy threshold, thread pool size, and test task type.
6. The NPU general testing framework as described in claim 4, characterized in that, The core framework layer of the NPU general testing framework includes: The core framework layer calls the model loading interface to load the target model, and calls the inference instance creation interface according to the "number of parallel instances / batch size" to generate multiple independent inference instances; The core framework layer scheduling module allocates inference tasks through a thread pool based on the "test task type"; The core framework layer input processing module loads data, converts formats, and calls the input data setting interface to pass it to the NPU; The core framework layer calls the inference execution interface to complete the calculation; the core framework layer calls the output tensor acquisition interface to save the result to the output path; The core framework layer performance modules summarize metrics such as FPS, latency, and utilization. The core framework layer verification module compares the output with the reference result to determine the correctness of the reasoning. The core framework layer releases resources in the order of "destroy instance → unload model → hardware destruction".
7. The NPU general testing framework as described in claim 6, characterized in that, The core framework layer of the NPU general testing framework is designed as a common module, encapsulating all the logic required for NPU testing, including: The test task scheduling module schedules test tasks based on the number of instances and models in the configuration file, using a thread pool or asynchronous task scheduling; it provides a task interruption mechanism, supports stopping the test via a signal, and ensures that resources are released normally; it enables multi-threaded and multi-instance parallel testing, and supports single-model multi-instance and multi-model parallel scenarios; The input / output processing module reads test data from configuration parameters and automatically parses the data dimensions; it calls the "tensor format conversion interface" of the adaptation layer to convert the input data into a format supported by the target NPU; it saves the inference output to the output path in a unified format and supports configuration of whether to save intermediate results; by uniformly processing the loading, format conversion and result saving of test data, it shields the tensor format differences of different NPUs. The performance statistics module includes core metrics such as throughput, latency, resource utilization, memory bandwidth, and timing accuracy. The result verification module compares the inference output file with the standard file in the reference output path to check whether the file size and binary content are consistent; for tasks such as classification and detection, it calculates the accuracy and mAP of the inference results to verify the correctness of the NPU inference results. The log monitoring module defines a unified log macro, and the output log includes timestamp, module name, and log level content, providing standardized log output and supporting different levels of log control.
8. The NPU general testing framework as described in claim 4, characterized in that, The adaptation layer of the NPU general testing framework includes: The adaptation layer features a standardized interface design, defining standardized interfaces for hardware initialization and destruction, model loading and unloading, inference instance creation and destruction, input data setting, inference execution, and obtaining output tensors.
9. An electronic device, characterized in that, include Processor; and A memory storing computer-readable instructions that, when executed by the processor, implement the method according to any one of claims 1 to 3.
10. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 3.