A repeatability verification method and system for computing power token benchmarking

Abstract

This invention discloses a repeatability verification method and system for computing power token benchmark testing, belonging to the field of automated verification technology for computing power hardware benchmark testing. Addressing the industry pain points of existing token benchmark testing environments, such as the lack of hardware-level fixed standards, subjective judgment of data fluctuations, lagging adaptation to new hardware, and poor stability under extreme conditions, this invention innovatively establishes a fully automated verification architecture with read-only firmware at the hardware level. It integrates measured data fitting to a general threshold, embedded algorithm multi-dimensional fluctuation verification, Bayesian adaptive iterative updates, and a full-condition fault-tolerant mechanism. Through hardware-level environment locking, unmanned, closed-loop testing throughout the entire process, and dynamic adaptive threshold adaptation, this invention achieves standardized technical verification of the entire computing power hardware benchmark testing process, environment, and results for all categories of computing power, significantly improving the repeatability, reproducibility, and credibility of test data, and adapting to the commercial deployment scenarios of large-scale AI computing power trusted benchmark testing.

Description

Technical Field

[0001] This invention relates to the field of automated verification technology for benchmark testing of computing power hardware, and is particularly applicable to standardized testing of large-scale model inference computing power hardware of all categories, automated verification of multi-dimensional data fluctuations, reliable verification of test results, adaptation of new computing power hardware and stable operation in complex working conditions. Background Technology

[0002] The current token benchmarking system in the industry relies on manually formulated testing specifications, human operating procedures, and experience-based judgment standards. It is a typical example of intellectual activity rules and human management processes. It lacks a dedicated software and hardware automated technology verification architecture and has many inherent technical defects, making it impossible to achieve standardized, highly reliable, and comprehensive benchmarking.

[0003] The shortcomings of existing technologies are mainly reflected in four dimensions: First, the testing environment lacks a unified technical standard, relying entirely on manual system constraints and human review and verification. The hardware and software parameters of the testing equipment and the network operating status lack a hardware-level read-only locking mechanism, resulting in significant environmental differences and poor data comparability. Second, the testing process is based on human operation specifications. Key aspects such as warm-up time, number of samplings, and log recording methods depend on manual operation, which is prone to human error and cannot achieve process standardization. Third, the judgment of test results relies on human experience, with a single verification dimension and a lack of multi-dimensional quantitative support. It is highly subjective, lacks credibility, and lacks standardized thresholds for fitting actual test data as a basis for judgment. Fourth, hardware compatibility and operating condition stability are poor. It is only compatible with a small number of conventional computing hardware and cannot be compatible with new and niche computing chips. At the same time, it does not have a dedicated fault tolerance mechanism for extremely complex operating conditions, making the testing process prone to interruption and test data prone to invalidation, and unable to adapt to large-scale commercial testing scenarios.

[0004] In summary, existing benchmark testing solutions are all based on manual management logic rather than closed-loop technical solutions. They suffer from problems such as non-standard processes, inconsistent environments, lack of basis for judgment, narrow applicability, and insufficient stability, and cannot meet the standardized, automated, and highly reliable testing technology requirements of all types of computing power hardware. Summary of the Invention

[0005] (a) Technical problems to be solved This invention completely abandons the traditional intellectual activity mode of manual standardization, human processes, and system judgment. It solves the core technical problems of existing token benchmark testing, such as the lack of hardware-level standardized locking environment, lack of fully automated programmatic execution logic, lack of multi-dimensional algorithm automated verification, lack of full-category hardware adaptation, and lack of extreme working condition fault tolerance. It constructs a fully technical, fully automated, fully covered, and highly stable benchmark testing and verification architecture, realizing the comprehensive standardization of testing process, testing environment, result judgment, hardware adaptation, and working condition compatibility.

[0006] (II) Technical Solution To achieve the aforementioned technical objectives, this invention provides a fully automated, closed-loop testing and verification solution with deep hardware and software collaboration. It eliminates any manual management rules and operational procedures, completely avoiding the risks associated with intellectual property review. This invention establishes a read-only, fixed, dedicated standardized computing power testing environment parameter model at the underlying firmware level of the testing hardware. All core testing environment parameters are fixed through hardware-level locking technology, preventing manual configuration modifications and environmental disturbances. Before testing, the hardware autonomously performs full-dimensional parameter verification and anomaly interception, achieving unique standardized control of the testing environment.

[0007] This invention deploys an embedded, dedicated automated testing program adapted to all types of token inference scenarios. It solidifies standardized execution logic for hardware warm-up, batch inference, multi-dimensional data sampling, and full log retention, achieving unmanned, autonomous, closed-loop operation of the testing process. Based on massive amounts of steady-state test data from all types of computing power hardware, it specifically fits and generates multi-dimensional standardized verification thresholds. Through an embedded multi-dimensional fluctuation analysis algorithm, it automatically quantifies multi-dimensional test data fluctuations, achieving an objective technical judgment of test repeatability and reproducibility.

[0008] Simultaneously, this invention adds a hardware iteration adaptive adaptation module based on Bayesian updates. Using a general, fixed threshold as a priori, it fits the specific deviation correction coefficients of new hardware through multiple rounds of standardized retesting, covering all categories of computing power devices. A comprehensive extreme condition fault tolerance mechanism is established, enabling local cache retransmission during network outages, recovery from hardware anomalies, and high-concurrency data balancing compensation, ensuring stable operation of the testing process under complex conditions. Finally, the device autonomously integrates multi-dimensional test data to generate a standardized report, which is then stored and publicized using blockchain technology, achieving a purely technical, automated, and highly stable closed-loop verification process for benchmark testing.

[0009] (III) Beneficial Effects Compared to traditional manual management testing solutions, this invention represents a substantial technological advancement and comprehensive scenario optimization: First, this invention pioneers a hardware firmware read-only solidified environment parameter locking mechanism, which differs from traditional testing schemes that are configurable and manually adjustable by general software. By permanently solidifying and locking the core test environment constraint logic at the hardware level, it eliminates the problems of human configuration differences and arbitrary environmental disturbances from the hardware source, breaks through the limitations of traditional manual system constraints, and realizes standardized, unique, and tamper-proof technical control of the test environment, providing underlying environmental protection for the repeatability of test data.

[0010] Secondly, this invention constructs a full-link embedded programmable fully automated execution system, which solidifies the entire process logic of hardware steady-state warm-up, multi-dimensional inference operation, full data sampling, operation log retention, and abnormal environment interception into a low-level program that the hardware can execute autonomously. There is no human intervention port, which completely eliminates random errors and process non-standardization problems caused by manual operation, and realizes the standardization, unmanned operation, and closed-loop implementation of the test process, which is superior to the existing semi-automated and manually assisted test solutions.

[0011] Third, this invention features a unique technical architecture based on fitting and solidifying thresholds using massive steady-state measured data, combined with multi-dimensional algorithm-based fluctuation verification. It is not a simple application of general data statistical algorithms. Instead, it customizes verification logic and threshold fitting rules for the specific characteristics of large-scale token reasoning tests, covering multiple core test dimensions. This completely eliminates the industry's subjective judgment model that relies on human experience, filling the gap in the industry's lack of a technically sound and standardized multi-dimensional fluctuation verification system, and significantly improving the objectivity and credibility of test results.

[0012] Fourth, this invention establishes a hardware iteration adaptive architecture based on Bayesian updates and a fault-tolerant system for all-dimensional extreme working conditions. It overcomes the technical shortcomings of traditional testing solutions, such as narrow hardware adaptability and poor stability under complex working conditions. It can independently adapt to various new and niche heterogeneous computing hardware, while ensuring the stability of the testing process and the validity of data under complex working conditions such as network outages, high concurrency, and instantaneous hardware anomalies. It achieves adaptability to all types of hardware and all scenarios, significantly improving the commercialization adaptability and reliability of the solution. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the hardware execution process for automated benchmark testing and verification of the present invention; Figure 2 This is a module architecture diagram of the automatic hardware verification test environment of the present invention; Figure 3 This is the algorithm logic diagram for the automated data fluctuation verification of this invention. Detailed Implementation

[0014] The following detailed description of the technical solution of this invention, in conjunction with the technical implementation of hardware devices and embedded algorithms, illustrates the pure technical solution of this invention. The embodiments described below are all implemented using automated technology and do not involve any manual control processes, manual rule settings, or manual parameter adjustments. They are only used to explain the technical solution of this invention and do not constitute a limitation on the scope of protection of this invention.

[0015] Example 1: Implementation of Automated Calibration of Testing Thresholds for All Types of Hardware In this embodiment, multiple mainstream heterogeneous computing hardware platforms from various industries are selected as calibration samples to conduct standardized benchmark tests for large-scale model inference. Through multiple batches and durations of continuous repeated testing, comprehensive test data under various hardware steady-state operating conditions is collected. The raw data is standardized and preprocessed to remove invalid sample data such as transient startup, idle standby, and overload anomalies. Statistical noise reduction, outlier screening, and least-squares fitting are used to iteratively optimize and generate a universal multi-dimensional verification threshold system adapted to all types of computing hardware, ensuring the universality and scientific validity of the thresholds.

[0016] Through batch steady-state data fitting and optimization, multi-dimensional standardized verification thresholds are formed and solidified in layers. These thresholds cover repeatability verification thresholds within the same environment, reproducibility verification thresholds across environments, inference latency fluctuation thresholds, and operating energy consumption fluctuation thresholds, constructing a three-dimensional and comprehensive test data fluctuation verification standard. All threshold parameters are permanently solidified into the verification terminal firmware with read-only lock permissions, eliminating the channels for manual modification, custom adjustments, and background tampering, thus ensuring the uniformity, stability, and authority of the full-domain test verification standard.

[0017] For novel architectures and new computing hardware not included in the initial fitting samples, the system incorporates adaptive iterative adaptation logic: using a universally fixed threshold as a priori benchmark, the device automatically initiates multiple rounds of standardized full-process retesting to fully collect multi-dimensional inference steady-state operation data of the new hardware under standard test environments. Relying on the built-in Bayesian update framework, combined with the likelihood distribution of measured data and the prior threshold distribution, iterative fitting yields a specific deviation correction coefficient for the corresponding hardware. This correction coefficient is then fused with the universal threshold for adaptation, generating a specific verification threshold adapted to the new hardware and automatically updating the firmware, achieving seamless adaptation across all categories of computing hardware.

[0018] Among them, the core formula for Bayesian update is a universally applicable and exemplary algorithm formula in the industry: P(θ|D)=(P(D|θ)·P(θ)) / P(D) In the formula, P(θ) is the prior distribution of the general fixed threshold, P(D|θ) is the likelihood function of the measured data of the new hardware, P(θ|D) is the updated posterior threshold distribution, and P(D) is the marginal probability, which is used to normalize the posterior distribution and ensure the scientificity and accuracy of the threshold iterative fitting.

[0019] Example 2: Implementation of Full-Dimensional Automatic Closed-Loop Verification of Hardware Environment The test terminal's underlying firmware permanently embeds a standardized environment adaptation model, which integrates standard adaptation ranges for core operating parameters across multiple dimensions, including the hardware, system, and network layers. Before each benchmark test begins, the terminal automatically calls the underlying hardware scanning interface to scan all dimensions of the current test device's real-time operating parameters without any blind spots, generating a real-time environment parameter snapshot.

[0020] The system will scan parameters in real time and compare them precisely with the standardized parameter ranges fixed in the firmware. The entire process is executed autonomously by the embedded program, without the need for manual verification or calibration. If all parameters fall within the standard adaptation range, the system will automatically determine that the environment is compliant and allow subsequent testing processes. If any core parameter deviates from the fixed standard range, the system will immediately and automatically lock the testing permissions, block invalid testing processes, and simultaneously generate a standardized environment anomaly log. The abnormal data will be retained for subsequent traceability and verification, realizing a purely technical and fully automated closed-loop verification of the testing environment, eliminating test data deviations caused by environmental differences from the source.

[0021] Example 3: Implementation of Full-Process Automated Testing and Verification The test terminal has built-in firmware-encapsulated automated test timing logic, supporting the periodic automatic initiation of benchmark test tasks. It can adapt to both conventional fixed-cycle iteration and new hardware-triggered expedited iteration operation modes. After the test starts, the device autonomously executes a standardized hardware warm-up process. Once the hardware reaches a steady-state standard, it automatically starts batch large-model inference operations, continuously collecting core test data such as throughput, inference latency, and operating power consumption in multiple batches and time periods. Simultaneously, it completes the retention and storage of full operation logs and test snapshots, all without manual intervention.

[0022] After the testing process is completed, the data verification terminal calls the underlying fixed fluctuation analysis algorithm, using relative standard deviation as the core quantitative indicator. The exemplary calculation formula is: RSD=(S / X̄)×100% In the formula, S represents the standard deviation of multiple batches of test data, and X̄ represents the average value of multiple batches of test data. This formula can accurately quantify the fluctuation range of various test data. The system compares the calculated multidimensional fluctuation index with the standardized thresholds for repeatability and reproducibility of firmware one by one, and automatically outputs objective and quantitative test result judgments, completing the fully automated verification process.

[0023] Example 4: Fault-Tolerant Implementation under Extreme Operating Conditions The system's underlying firmware incorporates a multi-dimensional fault tolerance mechanism for extreme operating conditions, covering three typical complex operating conditions: network outages / offline scenarios, instantaneous hardware anomalies, and high-concurrency overload. The entire process is automatically triggered and autonomously repaired, ensuring continuous and stable testing processes and complete and valid data.

[0024] In the case of network outage and offline fault tolerance, the system automatically encrypts and stores real-time test data in a dedicated local partition, using a first-in-first-out circular overwrite storage mechanism to avoid data overflow and loss. After the network link is restored, the system automatically re-uploads offline cached data to the evidence storage node in batches according to the order of test timestamps, ensuring that all data is completely uploaded to the blockchain and retained.

[0025] Under abnormal hardware conditions, the system monitors hardware operating status parameters in real time. When it detects faults such as abnormal hardware temperature, driver malfunctions, or computational errors, it automatically marks the fault and pauses the current test task to avoid invalid data collection. After the hardware operating status returns to a steady state, the system uses the moving median of the most recent valid test data as a benchmark to complete the missing test data and performs a second verification of data validity to ensure the integrity of the test batch.

[0026] Under high concurrency conditions, the system activates a fixed data balancing compensation algorithm. Using the median of the data in the current test window as a benchmark, and combined with the median absolute deviation statistical threshold, it automatically identifies and removes abnormal data points such as instantaneous peaks and troughs, and uses the optimized mean of effective data as the final test result, thus eliminating the impact of high-concurrency instantaneous disturbances on test accuracy.

[0027] Example 5: Implementation of a Novel Hardware Adaptive System The system possesses intelligent hardware iteration recognition capabilities, automatically identifying new architecture computing hardware not included in the firmware standard library. Once a new hardware is detected, it immediately triggers an adaptive adaptation process autonomously, eliminating the need for manual identification and configuration. The system first collects the basic architecture parameters of the new hardware, matches them with the common fixed thresholds of the corresponding hardware category as prior standards, and then automatically initiates multiple rounds of standardized retesting tasks to comprehensively collect multi-dimensional data on the steady-state inference operation of the hardware.

[0028] Based on the built-in Bayesian update framework, the system autonomously calculates the deviation correction coefficient between the new hardware and the general standard, iteratively optimizes and generates a unique adaptation threshold, and solidifies the new threshold parameters into the terminal firmware, updates the standard database, completes the adaptation and entry of the new hardware into the database, and realizes the autonomous synchronous update of the test standard in the scenario of computing power hardware iteration.

[0029] Example 6: Implementation of Blockchain Evidence Storage and Standardized Report Generation After the entire testing, verification, and judgment process is completed, the terminal autonomously integrates all information, including test environment snapshots, raw test data, multi-dimensional fluctuation analysis results, repeatability and reproducibility judgment conclusions, test timestamps, and runtime exception logs, to automatically generate a standardized and structured test verification report.

[0030] After the report is generated, the system automatically encrypts and pushes the report data to the blockchain notarization node, completing irreversible notarization and public disclosure, and achieving traceability, verifiability, and tamper-proofness of the entire testing process and all data. The entire report generation and notarization process is executed entirely automatically by technology, without any manual editing, review, or intervention, ensuring the credibility and authority of the test results.

Claims

1. A method for verifying the repeatability of computing power token benchmark testing, characterized in that, Based on the proprietary verification logic embedded in the terminal firmware, the test terminal, environment verification equipment, and data verification terminal work together in a fully automated manner. The entire process requires no manual configuration, judgment, or intervention, thus constructing a closed-loop verification system characterized by environment lock-in, standardized data verification, adaptive hardware iteration, and controllable fault tolerance. This system includes the following steps: S1. Permanently solidify the standardized adaptation parameters of the hardware, software and network corresponding to the benchmark test at the underlying firmware of the test terminal, build a unique standardized environment adaptation model at the device level, and lock the parameters at the underlying level so that they cannot be modified, adapting to all types of AI computing hardware. S2. Before the test starts, the environment verification equipment automatically scans all dimensions of the current test equipment's operating parameters and compares them precisely with the standardized adaptation parameters fixed in the firmware. It automatically completes the environment compliance verification and anomaly interception, eliminating manual parameter adaptation deviations. S3. After the environment verification is passed, the test terminal loads the standardized test program with firmware and autonomously completes hardware warm-up, batch inference calculation, multi-dimensional data sampling, and full log retention operations. The entire process is automated and closed-loop. S4. The data verification terminal uses a built-in solidified fluctuation analysis algorithm to automatically count the multi-dimensional fluctuation indicators of test data from multiple time periods and batches, and combines the measured fitting threshold of the solidified firmware to complete the automated and objective determination of test repeatability. S5. The data verification terminal collects test data from multiple devices, multiple environments, and multiple time periods, calculates the cross-environmental comprehensive fluctuation value, and combines it with a preset fixed threshold to complete the automatic determination of test reproducibility. S6. After the verification is passed, the terminal automatically integrates test data, operation logs, and multi-dimensional verification results to generate a standardized verification report and complete the blockchain notarization and publicity. Relying on the underlying firmware solidification logic, it is equipped with a dual-dimensional update and fault tolerance mechanism. Through regular periodic iteration and new hardware-triggered expedited iteration, it adapts to hardware architecture updates. Combined with extreme working condition fault tolerance capabilities, it ensures the long-term stability, reliability, and controllability of the test and verification system.

2. The method according to claim 1, characterized in that, The standardized adaptation parameters are read-only and tamper-proof universal adaptation parameters at the firmware level. They are pre-fixed and stored in the terminal's underlying firmware and are exclusively used for compliance verification in the computing power token benchmark testing environment. They cover hardware operating constraint range, software version adaptation range, network stability judgment conditions, and hardware preheating steady-state standards. The fully automatic verification and matching is completed only through the hardware's dedicated interface, without any manual configuration, modification, or custom adjustment permissions. This unifies the testing environment access standards for all types of heterogeneous computing power hardware.

3. The method according to claim 1, characterized in that, It adopts a firmware-fixed proprietary relative standard deviation algorithm as the core fluctuation judgment logic. The algorithm's operation logic is fixed and cannot be modified. It automatically collects and statistically analyzes test data from multiple time periods and batches and quantifies the fluctuation amplitude. It uses massive amounts of stable-state test data from heterogeneous computing power hardware across all categories to generate a universal fixed threshold as the sole judgment criterion. This eliminates human experience intervention and achieves standardized, objective, and fully automatic judgment of test repeatability.

4. The method according to claim 1, characterized in that, The solidified threshold system is a dedicated standard adapted to all categories of computing power testing. It is generated by fitting massive amounts of steady-state test data of heterogeneous computing power hardware, and the firmware is locked and stored at the underlying level and cannot be tampered with. For new architectures and new computing power hardware that are not included in the adaptation system, the system uses the general solidified threshold as a priori benchmark, and autonomously completes multiple rounds of retesting, deviation fitting and coefficient correction through the built-in Bayesian update framework to adaptively generate dedicated adaptation thresholds, so as to achieve full coverage adaptation of all categories of computing power hardware, both new and old.

5. The method according to claim 1, characterized in that, The system has a built-in firmware-embedded fault tolerance mechanism for all-dimensional extreme working conditions. All fault tolerance logic is embedded at the underlying level, automatically triggered, and requires no manual intervention. It includes closed-loop logic such as encrypted caching and retention of offline data after network outages, automatic time-series retransmission after network recovery, automatic task suspension due to hardware malfunctions, and data completion and secondary validity verification after device recovery. This ensures that the testing process is continuous and uninterrupted under complex working conditions, and that the test data is complete and valid.

6. The method according to claim 1, characterized in that, For high-concurrency testing scenarios with instantaneous disturbances, the system calls upon a proprietary data balancing compensation algorithm embedded in the firmware. Relying on statistically fixed anomaly judgment thresholds, it automatically identifies and filters abnormal data points generated by instantaneous disturbances, optimizes and fits the effective test data, and outputs accurate test results. This eliminates test deviations caused by high-concurrency operating condition disturbances from the data level, ensuring the stability and accuracy of test results under high-load scenarios.

7. A repeatability verification system for computing power token benchmark testing, characterized in that, The method for performing any one of claims 1-6 includes: an automatic hardware environment verification module, an automated test execution module, a data fluctuation verification module, a hardware iteration adaptation module, an extreme working condition fault tolerance module, and a report generation and storage module; the core operating logic of all modules is solidified in the underlying firmware of the device and cannot be manually tampered with or adjusted. The modules work together precisely to form a complete benchmark test verification closed loop with hardware environment lock-in, unmanned testing process, threshold adaptive iteration, full working condition fault tolerance, and reliable result storage.

Images