NVMe solid state disk abnormal power-off read-write test system and method

Abstract

The application discloses a kind of NVMe solid state disk abnormal power-off read-write test system and method, the system includes: control server, test node and power control unit, control server is connected test node and power control unit respectively, power control unit is electrically connected test node.Test node captures NVMe drive layer I / O timeout event and obtains the I / O performance index of to-be-tested SSD.Control server is based on historical performance index to establish baseline according to I / O timeout event and performance index, when performance index significantly deviates from baseline and is accompanied by I / O timeout event reaches severe abnormal threshold, judge hardware exception, reduce test frequency by power control unit, if continuous not trigger exception after a number of cycles are observed, then gradually restore, again determine abnormality in recovery, then back off.The application realizes test frequency self-adaptive adjustment, shortens test cycle.

Description

Technical Field

[0001] This application relates to the field of solid-state drive testing technology, and in particular to an NVMe solid-state drive abnormal power failure read / write testing system and method. Background Technology

[0002] NVMe solid-state drives (SSDs that use the NVMe protocol / specification and communicate with the host via the PCIe bus) are widely used in data centers, enterprise storage, and consumer electronics due to their superior performance of high bandwidth and low latency. However, in real-world use, unstable power supplies and subsequent power outages are frequent occurrences, which can easily lead to data loss or corruption on SSDs.

[0003] Existing NVMe SSD abnormal power failure testing solutions typically employ a fixed-frequency cyclic power-down strategy, repeatedly performing power-off and power-on operations at preset fixed time intervals. This fixed-frequency testing method cannot dynamically adjust test parameters based on the real-time operating status of the SSD. When the SSD is under high load or in an abnormal state, fixed-frequency testing may subject the SSD to additional stress, even masking the true manifestation of the fault; when the SSD is idle or in a stable state, fixed-frequency testing may lead to overtesting, reducing testing efficiency.

[0004] Furthermore, existing testing solutions primarily focus on whether there are hardware-level failures in the solid-state drive (SSD), such as judging test results by checking whether the SSD can be recognized normally or whether the SMART information is abnormal. These solutions lack system-level monitoring of the operating system's storage driver layer behavior during abnormal power outages, and cannot capture and quantify the "system lag" or I / O response latency that users can actually perceive in abnormal power outage scenarios, resulting in a disconnect between test results and user experience. Summary of the Invention

[0005] In view of this, this application provides an NVMe solid-state drive abnormal power failure read / write test system and method to solve the above problems.

[0006] To address the aforementioned technical problems, this application adopts the following technical solution: providing an NVMe solid-state drive abnormal power failure read / write testing system, comprising: The system includes a control server, test nodes, and a power control unit. The control server is communicatively connected to both the test nodes and the power control unit, and the power control unit is electrically connected to the test nodes. The test node is used to capture NVMe driver layer I / O timeout events in the Linux kernel logs and obtain the I / O performance metrics of the NVMe SSD connected to the test node. The control server is used to establish a baseline based on the historical I / O performance metrics of the NVMe SSD under test, according to the I / O timeout events and I / O performance metrics reported by the test nodes. When the I / O performance metrics of the current round deviate significantly from the baseline and the I / O timeout events reach the severe anomaly threshold, it is determined that the NVMe SSD under test is hardware abnormal. The power control unit reduces the power-off cycle test frequency of the test nodes. After reduction, it observes for several cycles at low frequency. If no abnormality is triggered continuously, it gradually restores to the initial frequency. If the hardware abnormality of the NVMe SSD under test is determined again during the restoration process, it reverts to the previous frequency.

[0007] To address the aforementioned technical problems, another technical solution adopted in this application is: providing a method for testing NVMe solid-state drives during abnormal power outages, which is applied to the aforementioned NVMe solid-state drive abnormal power outage read / write testing system. The method includes: The test node captures NVMe driver layer I / O timeout events in the Linux kernel logs and obtains the I / O performance metrics of the NVMe SSD connected to the test node. The control server establishes a baseline based on the historical I / O performance metrics of the NVMe SSD under test, according to the I / O timeout events and I / O performance metrics reported by the test nodes. When the I / O performance metrics of the current round deviate significantly from the baseline and the I / O timeout events reach the severe abnormality threshold, it is determined that the NVMe SSD under test is hardware abnormal. In response to the determination that the NVMe SSD under test is experiencing a hardware malfunction, the control server reduces the power-off cycle test frequency of the test node through the power control unit. It observes for several cycles at the low frequency. If no malfunction is triggered continuously, it gradually restores to the initial frequency. If the NVMe SSD under test is determined to be experiencing a hardware malfunction again during the restoration process, it reverts to the previous frequency.

[0008] Compared to existing technologies, this invention achieves direct monitoring of the system storage driver layer behavior during abnormal power outages by capturing NVMe driver layer I / O timeout events in the Linux kernel logs in real time and obtaining I / O performance metrics. This accurately reflects the system response status perceptible to users during abnormal power outages. By establishing a baseline based on historical I / O performance metrics, frequency reduction is triggered when I / O performance metrics significantly deviate from the baseline and are accompanied by I / O timeout events reaching a severe anomaly threshold, avoiding misjudgments caused by normal system performance fluctuations. Through a three-stage adaptive closed-loop control of "frequency reduction—observation—gradual recovery / rollback," the power outage test frequency is dynamically adjusted according to the real-time status of the solid-state drive. This ensures a high fault detection rate while avoiding over-testing, effectively shortening the test cycle compared to traditional fixed-frequency solutions. Attached Figure Description

[0009] Figure 1This is a schematic diagram of an embodiment of the NVMe solid-state drive abnormal power failure read / write test system of the present invention.

[0010] Figure 2 This is a flowchart illustrating an embodiment of the NVMe solid-state drive abnormal power failure read / write test method of the present invention. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0012] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] Figure 1 A schematic diagram illustrating the structure of one embodiment of the NVMe solid-state drive abnormal power-off read / write test system of the present invention is shown. Figure 1As shown, the NVMe solid-state drive abnormal power failure read / write test system includes: a control server 100, a test node 200, and a power control unit 300. The control server 100 is communicatively connected to the test node 200 and the power control unit 300, and the power control unit 300 is electrically connected to the test node 200.

[0015] Test node 200 is used to capture NVMe driver layer I / O timeout events in the Linux kernel log and obtain the I / O performance metrics of the NVMe SSD connected to test node 200. Control server 100 is used to establish a baseline based on the historical I / O performance metrics of the NVMe SSD under test according to the I / O timeout events and I / O performance metrics reported by test node 200. When the I / O performance metrics of the current round deviate significantly from the baseline and the I / O timeout events reach the severe anomaly threshold, it is determined that the NVMe SSD under test has a hardware anomaly. The power control unit 300 reduces the power-off cycle test frequency of test node 200. After reduction, it observes for several cycles in the low-frequency state. If no anomaly is triggered continuously, it gradually restores to the initial frequency. If the hardware anomaly of the NVMe SSD under test is determined again during the restoration process, it reverts to the previous frequency.

[0016] Specifically, the control server 100, as the decision-making core of the entire testing system, is responsible for receiving monitoring data reported by the test node 200, performing multi-indicator comprehensive judgment, and issuing frequency adjustment commands. The control server 100 can be an independent physical server, virtual machine, or cloud server instance, and is configured with at least one network interface for communication with the test node 200 and the power control unit 300. An adaptive test execution module runs on the control server 100, which is the core software component for implementing dynamic frequency adjustment.

[0017] Test node 200 is the physical computing device that executes test cases. It is equipped with an x86 architecture processor and runs a Linux operating system (e.g., Ubuntu 22.04 LTS, kernel version 5.15 or later). Test node 200 is connected to the NVMe SSD 400 under test via a PCIe 4.0 interface. A monitoring agent is deployed on test node 200. This monitoring agent is a user-space service program responsible for performing system-level I / O behavior monitoring.

[0018] The control server 100 establishes a communication connection with the test node 200 via Ethernet or other communication networks. Through this connection, the control server 100 sends test configurations to the test node 200 and receives monitoring data reported by the monitoring agent. The control server 100 also establishes a communication connection with the power control unit 300 via a communication network, sending power-off or power-on commands to the power control unit 300. The power control unit 300 is electrically connected to the test node 200 via power cables. After receiving commands from the control server 100, the power control unit 300 performs the power supply switching operation on and off for the test node 200.

[0019] The monitoring agent captures NVMe driver I / O timeout events in the Linux kernel logs in real time. In Linux systems, NVMe driver I / O timeout events generate specific formatted log entries in the kernel ring buffer, typically appearing as "nvme nvmeX: I / O tag ... timeout". The monitoring agent captures these log entries by reading / dev / kmsg in real time or periodically checking dmesg output, and extracts the I / O timeout information, including the device where the timeout occurred, the I / O tag, and the timeout duration.

[0020] The monitoring agent also polls the raw I / O counters in the system disk statistics interface ( / proc / diskstats). / proc / diskstats is a block device I / O statistics interface provided by the Linux kernel, which records raw cumulative counters for each block device, such as the number of read / write operations, the number of read / write merges, and the milliseconds spent on each read / write operation. The monitoring agent reads these raw counters at a preset sampling period (e.g., 1 second) and performs a secondary calculation on the difference between two samples to obtain the average I / O wait time and device activity level. The average I / O wait time reflects the average time from I / O request issuance to completion, including the time spent waiting in the queue and the actual processing time by the device; device activity level reflects the proportion of time the device is in a non-idle state.

[0021] The monitoring agent synchronizes kernel logs to a remote log server in real time via a system log synchronization mechanism to prevent local log loss due to abnormal power outages. Simultaneously, the monitoring agent reports captured I / O timeout events and calculated I / O performance metrics to the control server 100 via the communication network.

[0022] After receiving the data reported by the monitoring agent, the control server 100 performs a comprehensive multi-indicator judgment. The control server 100 maintains I / O performance data for each test node 200 over several recent normal cycles and calculates the average and standard deviation based on this data as a baseline. A normal cycle refers to a complete power-off-power-on cycle without I / O timeout events. The control server 100 compares the average I / O latency received in this round with the baseline. When the average I / O latency in this round significantly deviates from the baseline (e.g., exceeding the average plus three times the standard deviation) and the device busy level exceeds a preset high threshold (e.g., 90%), accompanied by I / O timeout events reaching a severe abnormality threshold, the tested NVMe SSD 400 is determined to have a hardware malfunction.

[0023] When a hardware anomaly is detected, the control server 100 reduces the power-off cycle test frequency of the test node 200 via the power control unit 300. Specifically, the control server 100 sends a command to the power control unit 300 to extend the interval of the power-off cycle from its current value to a larger value, for example, from once per minute to once every 5 minutes. After reducing the frequency, the system observes several complete test cycles at the low frequency, continuously monitoring I / O timeout events and I / O performance indicators. If no anomalies are triggered continuously during the observation period, and the I / O performance indicators fall back to the normal threshold range (e.g., average I / O wait time less than 200ms and device busy level less than 80%), the control server 100 gradually restores the test frequency, for example, first restoring it to once every 3 minutes, and then restoring it to once per minute. During the gradual restoration process, if a hardware anomaly is detected again, the control server 100 reverts the test frequency to the previous lower frequency and re-enters the observation state.

[0024] This embodiment uses a three-stage adaptive closed-loop control of frequency reduction, observation, and gradual recovery / rollback to enable the system to dynamically adjust the test frequency according to the real-time status of the solid-state drive (SSD). When the SSD exhibits abnormal behavior, the test intensity is reduced to protect the SSD and avoid overtesting. After the SSD returns to normal, the test efficiency is gradually restored, solving the problems of low test efficiency and overtesting in fixed-frequency test schemes.

[0025] Furthermore, to increase the richness of test data and the granularity of data integrity verification, in addition to the above embodiments, in other embodiments, the control server 100 also includes a cold and hot data construction module, which is used to construct cold data area, hot data area and warm data area on the NVMe solid-state drive under test to simulate the access mode of different types of data in real use scenarios; the cold data area occupies a preset proportion of capacity, and only read verification is performed after writing is completed. Its data file is divided into multiple cold data blocks of fixed size, and each cold data block independently calculates a hash value as the integrity verification baseline. The hash value is used to compare and verify with the read data in the subsequent power-off cycle.

[0026] Specifically, the cold and hot data construction module divides the NVMe SSD 400 under test into three data areas: a cold data area, a hot data area, and a warm data area. The three areas are allocated in terms of capacity according to a preset ratio. A preferred allocation is: approximately 50% for the cold data area, approximately 20% for the hot data area, and approximately 30% for the warm data area. These ratios can be adjusted according to actual testing needs.

[0027] The hot data area is used to simulate frequent read and write operations by the operating system and applications, and its access pattern is continuous high-frequency read and write. In one specific implementation, a general IO load generation tool is used to perform 4K random read and write operations in this area, with a queue depth set to 32, to simulate solid-state drive access behavior under high-concurrency I / O scenarios.

[0028] The warm data area is used to simulate data accessed by users at a moderate frequency, and its access mode is periodic read-refresh. In one specific implementation, the warm data area is filled with preset test data, and a read-only refresh operation is performed every 24 hours. After the refresh, the CRC32 check value is recalculated and compared with the previous check value to verify data integrity.

[0029] The cold data zone is used to simulate the stability of long-term archived user data under abnormal power outage scenarios. Its access mode involves only reading verification after a write operation is completed, without further write operations. The construction process of the cold data zone is as follows: First, the cold data partition is formatted as a file system. In one specific implementation, the cold data partition is formatted using the ext4 file system and then mounted to a preset mount point. Second, cold data files are generated. In one specific implementation, cold data files are generated using a system data generation tool in conjunction with a random number source; the data volume is calculated based on the SSD capacity and the proportion of the cold data zone. Then, the cold data files are divided into multiple cold data blocks of a fixed size. In one specific implementation, the cold data block size is set to 64MB. Finally, a hash value is independently calculated for each cold data block as an integrity verification baseline. In one specific implementation, the SHA-256 hash algorithm is used to calculate the hash value for each cold data block, and the hash value is stored in the database using a "block number - hash value" mapping relationship. After the cold data zone is constructed, the cold data file is only read-verified in all subsequent test loops and is not rewritten.

[0030] In each subsequent power outage cycle, test node 200 reads the cold data block data corresponding to the current cycle, calculates its SHA-256 hash value, and compares it with the baseline hash value of the cold data block stored in the database.

[0031] This embodiment simulates the access patterns of different types of data in real-world usage scenarios by constructing cold data areas, hot data areas, and warm data areas, making the test results more representative. The cold data area adopts a "one-time construction, block hash verification" strategy, which ensures the reliability of data integrity verification while limiting the write volume of the entire test cycle to the range of a single full write, effectively avoiding excessive consumption of the solid-state drive's write lifespan.

[0032] Furthermore, in order to accurately locate the error when verification fails, in other embodiments based on the above embodiments, the hot and cold data construction module is also used to record the physical logic block address range corresponding to each cold data block.

[0033] In one specific implementation, after the cold data file is generated and mounted, the list of contiguous physical blocks of the cold data file on the block device is obtained through the physical block mapping query tool provided by the file system. For the ext4 file system, its built-in file fragment query function can be used to obtain the physical block mapping of the file, outputting the starting sector number and length of each physical block occupied by the file, where the sector number is in units of 512 bytes.

[0034] After obtaining the complete list of contiguous physical blocks, and combining it with the preset cold data block size (e.g., 64MB), the physical logical block address range corresponding to each cold data block is calculated. Specifically, based on the logical offset of the cold data block in the file (the offset of the Nth block is N × block size), the physical block corresponding to that offset is searched in the list of contiguous physical blocks, thereby determining the starting physical sector number and length covered by the cold data block. In the scenario where the logical block address of the NVMe SSD is 512 bytes, the sector number output by filefrag can be directly used as the LBA address; in the scenario where the LBA is 4KB, the sector number needs to be divided by 8 before conversion.

[0035] After recording the physical logical block address range corresponding to each cold data block, the mapping relationship is stored in the database in the format of "block number - start LBA - end LBA".

[0036] This embodiment records the physical logical block address range corresponding to each cold data block, enabling precise location of the error within the physical address range of the solid-state drive when a subsequent verification error occurs, thus providing an accurate data foundation for error distribution analysis and root cause tracing.

[0037] Furthermore, in order to deeply analyze the root cause of the fault when a verification error occurs, in other embodiments based on the above embodiments, when a verification error occurs, the control server 100 classifies and determines the distribution of the erroneous logical block addresses according to the following rules: continuous block distribution is determined to be a physical bad block, random discrete distribution is determined to be a firmware mapping error, and concentrated in a specific capacity area is determined to be a specific physical unit characteristic degradation.

[0038] Specifically, in the post-verification phase after the test, if the SHA-256 verification of a certain cold data block fails, the control server 100 classifies and determines the distribution pattern of the erroneous logical block address based on the physical logical block address range recorded in the cold data block and the verification results of adjacent cold data blocks.

[0039] Category 1: Consecutive Block Distribution. When multiple faulty LBAs are consecutive or clustered in adjacent intervals in the address space, it is determined to be a contiguous block distribution. This distribution pattern usually corresponds to physical bad blocks, because physical bad blocks in NAND flash memory often exhibit the characteristics of appearing consecutively or in clusters. For example, if cold data blocks numbered 100 to 120 have consecutive check errors, and the LBA intervals corresponding to these cold data blocks are also consecutive, then the system determines it to be a contiguous block distribution and infers that the cause of the failure is a physical bad block.

[0040] The second category: Random discrete distribution. When error LBAs are uniformly and randomly distributed in the address space without obvious clustering characteristics, they are classified as random discrete distribution. This distribution pattern usually corresponds to firmware mapping errors (i.e., FTL logic errors) or random bit flips caused by read interference. For example, if check errors are scattered in different locations in the address space without obvious continuity or clustering characteristics, the system is classified as having a random discrete distribution, and the cause of the fault is inferred to be a firmware mapping error or read interference problem.

[0041] The third category: Concentrated in a specific capacity area. When error LBAs are highly concentrated within a certain capacity range, they are determined to be concentrated in a specific capacity area. This distribution pattern usually corresponds to the physical characteristic degradation of a specific die or plane. For example, if all verification errors are concentrated in the first 10% of the SSD's capacity, the system, combined with the physical topology parameters obtained from the NVMe IdentifyController, infers that the specific die or plane corresponding to that area may have physical characteristic degradation.

[0042] If all cold data blocks pass the SHA-256 checksum, the system directly determines that the hardware and firmware are in normal condition and will not be included in the above distribution mode classification.

[0043] This embodiment classifies and determines the distribution of erroneous logical block addresses, enabling in-depth diagnosis from "data verification failure" to "fault root cause classification." It can attribute errors to physical bad blocks, firmware mapping errors, or degradation of specific physical unit characteristics, providing a powerful analytical tool for solid-state drive reliability assessment.

[0044] Furthermore, in order to provide differentiated responses to anomalies of different severity, in other embodiments based on the above embodiments, the severe anomaly threshold is when the I / O suspension time exceeds a preset suspension time threshold, or when the number of I / O timeout events within a preset time window reaches a preset number threshold; the control server 100 is also used to mark an event as a mild anomaly and record the event when the number of I / O timeout events within a preset time window reaches a preset mild anomaly number threshold.

[0045] Specifically, the severe anomaly threshold includes two scenarios: The first scenario is that the I / O pending time exceeds a preset pending time threshold. In one specific implementation, the preset pending time threshold is set to 30 seconds. When the monitoring agent detects that an I / O request has not been completed after waiting for more than 30 seconds, the control server 100 determines it as a severe anomaly. The second scenario is that the number of I / O timeout events reaches a preset threshold within a preset time window. In one specific implementation, the preset time window is set to 60 seconds, and the preset number threshold is set to 10 times. When the number of NVMe driver layer I / O timeout events captured within 60 seconds reaches 10 or more, the control server 100 determines it as a severe anomaly.

[0046] The threshold for minor anomalies is defined as the number of I / O timeout events reaching a preset threshold within a preset time window. In one specific implementation, the preset time window is set to 60 seconds, and the preset threshold for the number of minor anomalies is set to 3. When the number of I / O timeout events captured within 60 seconds reaches 3 but is less than 10, the control server 100 marks it as a minor anomaly and records the event, but does not trigger a frequency reduction operation.

[0047] This embodiment establishes a progressive judgment system through two levels of thresholds: mild anomalies and severe anomalies. Mild anomalies serve as early warning signals, indicating that the solid-state drive may experience performance fluctuations but are not serious enough to trigger frequency throttling. Severe anomalies serve as action signals, triggering frequency throttling and deep scanning. Through this hierarchical design, the system can respond differently to anomalies of varying severity, avoiding frequent frequency throttling due to oversensitivity and missed fault detection due to slow response.

[0048] Furthermore, in order to solve the problem of the inability to establish a baseline due to the lack of historical data in the early stage of system deployment, in other embodiments based on the above embodiments, the control server 100 is also used to enter a baseline calibration period after system deployment, and run continuously until a preset number of normal cycles are accumulated. A normal cycle refers to a complete power-off-power-on cycle without I / O timeout events. No abnormal judgment is triggered during the calibration period. If a sufficient number of normal cycles cannot be accumulated after the preset time, the preset default baseline is forcibly used.

[0049] Specifically, the goal of the baseline calibration period is to accumulate a preset number of normal cycles. A normal cycle refers to a complete power-down-power-on cycle without any I / O timeout events. In one specific implementation, the preset number is set to 10 normal cycles.

[0050] During the baseline calibration period, the monitoring agent of test node 200 normally collects I / O timeout events and I / O performance metrics data and reports them to control server 100. However, control server 100 does not trigger any anomaly detection during this phase. Even if the monitoring agent reports I / O timeout events, control server 100 only logs the data and does not perform frequency reduction operations or trigger deep scans. All I / O performance metrics data during the calibration period is used to build the initial baseline, not to trigger alarms.

[0051] If a continuous anomaly prevents the accumulation of a sufficient number of normal cycles within a preset time, the control server 100 will force the use of a preset default baseline. In one specific implementation, the preset time is set to 60 minutes, and the preset default baseline is the average of the average I / O latency. Standard deviation When using the default baseline, the control server 100 marks "Calibration period abnormal - insufficient baseline" in the logs so that subsequent analysis can take note of potential biases in the calibration period data.

[0052] After accumulating a preset number of normal cycles, the control server 100 calculates the mean of the initial baseline. and standard deviation Starting from the (N+1)th normal cycle, the control server 100 will be officially activated. The threshold is used to determine anomalies, and a rolling window update mechanism is enabled—whenever new normal periodic data is added, the oldest data is automatically discarded, and the window size is always kept at the preset number.

[0053] This embodiment solves the problem of missing baseline data during system cold start by using a baseline calibration period mechanism, ensuring that the anomaly detection logic can work normally from the initial stage of system operation; at the same time, the timeout fallback strategy avoids the problem of the calibration period being extended indefinitely due to continuous anomalies.

[0054] Furthermore, in order to still be able to perform cold data verification in the case of file system corruption, in other embodiments based on the above embodiments, the test node 200 is also used to attempt to mount the cold data area after each power-on. If the mounting is successful, the verification is performed by reading the file path and executing the file fragment query tool to verify the consistency of continuous physical blocks; if the mounting fails, the raw partition is read segment by segment based on the pre-recorded list of continuous physical blocks for verification.

[0055] Specifically, in each power-down-power-up cycle, after the system powers on, test node 200 attempts to mount the cold data partition. Test node 200 first waits for the Linux system's default.target to be achieved (indicating that the system boot target has been completed), and then executes the mount command.

[0056] If the mount is successful, test node 200 reads the cold data block corresponding to the current round through the file path, calculates its SHA-256 hash value, and compares it with the baseline. Simultaneously, test node 200 executes a file fragment query tool to verify whether the current list of contiguous physical blocks matches the historical list of contiguous physical blocks stored in the database. Because the file system may perform log replay or repair operations after an abnormal power outage, the physical block mapping of files may be slightly adjusted. If the current list of contiguous physical blocks matches the historical record, the historical record is used; if they do not match, the database is updated with the new list of contiguous physical blocks.

[0057] If mounting fails (e.g., the file system is corrupted due to an abnormal power outage), test node 200 initiates a degraded read strategy. Test node 200 reads the historical list of consecutive physical blocks for the cold data file from the database, and, combined with the block number corresponding to the current round, locates the physical segment corresponding to the cold data block. Then, test node 200 directly reads the data of the corresponding physical segment in the raw partition using a block device raw read tool, calculates the SHA-256 hash value, and compares it with the baseline.

[0058] If a physical segment fails to be read (returning an I / O error), test node 200 marks that cold data block as "physically unreadable" and records the error LBA range. If all physical segments fail to be read, test node 200 marks the solid-state drive as "physically unusable" and terminates the test.

[0059] This embodiment employs a two-tiered strategy: file path reading verification upon successful mounting and raw partition degradation reading verification upon mounting failure. This ensures the feasibility of cold data verification in file system corruption scenarios and improves the robustness of the test system under abnormal power outage scenarios.

[0060] Furthermore, to avoid the I / O generated by cold data verification from polluting the performance monitoring data, in other embodiments based on the above embodiments, the period during which the test node 200 performs cold data block verification is a verification window. During the verification window, the anomaly judgment of I / O performance indicators is suspended, and the data in the verification window is not included in the historical baseline. After the verification window ends, a clean monitoring window is entered, and the monitoring data in the clean monitoring window participates normally in baseline updates and anomaly judgment.

[0061] Specifically, the monitoring agent of test node 200 generates a large number of sequential read I / O operations during the cold data block verification process. These I / O operations significantly affect I / O performance metrics—average I / O latency may increase due to verification reads, and device busyness may approach 100% due to verification reads. If performance monitoring is performed and anomaly detection is triggered during the verification process, the verification behavior itself may lead to false alarms.

[0062] The verification window is defined as the time interval during which the monitoring agent performs cold data block verification for the current round. In one specific implementation, the size of a single cold data block is 64MB. Under the condition that the sequential read speed of the NVMe solid-state drive is about 3GB / s, the verification time for a single cold data block is about 0.02 seconds. Within the verification window, the monitoring agent performs the following operations: (1) reads the cold data block data corresponding to the current round; (2) calculates the SHA-256 hash value; (3) compares it with the baseline hash value and records the result.

[0063] During the verification window, the monitoring agent suspends all anomaly detection and processing for I / O performance metrics: average I / O latency and device activity data within the verification window are not included in the historical baseline window; I / O performance metrics within the verification window do not participate in any anomaly detection; and no alarms are triggered within the verification window. The raw I / O performance metric data within the verification window is only archived as raw records and is not used for any subsequent analysis.

[0064] After the verification window ends, the monitoring agent enters the clean monitoring window. The clean monitoring window remains open until the next power outage. During this period, the monitoring agent collects I / O performance metrics normally, and the collected data is used for baseline updates and anomaly detection.

[0065] This embodiment effectively avoids interference from verification I / O on performance monitoring indicators by using a timing isolation mechanism between the verification window and the clean monitoring window, thus ensuring the accuracy of anomaly detection.

[0066] Furthermore, in order to enable remote controllable operation of the power supply of the test node, in other embodiments based on the above embodiments, the power control unit 300 is an intelligent rack-mounted power distribution unit that supports the Simple Network Management Protocol.

[0067] Specifically, the control server 100 sends instructions to the intelligent rack-mounted power distribution unit via Simple Network Management Protocol (SNMP) to perform power on / off operations on the designated test node 200. Specifically, the control server 100 constructs an SNMPSET request to set the output status (ON or OFF) of the corresponding output port on the intelligent rack-mounted power distribution unit, thereby enabling remote control of the power-on and power-off of the test node 200.

[0068] SNMP control commands are transmitted over a network, and response time is affected by network conditions, potentially resulting in delays ranging from milliseconds to seconds. For most abnormal power outage test scenarios, the power outage duration is typically on the order of seconds (e.g., 5 seconds), which is sufficient for testing requirements.

[0069] In special testing scenarios requiring higher timing accuracy, hardware triggering methods with more controllable delays can be used as supplementary solutions. For example, more precise power-off timing control can be achieved by controlling solid-state relays via GPIO or by controlling PCIe slot power. However, as a standard configuration, the SNMP-based intelligent rack-mount power distribution unit solution is suitable for most abnormal power-off testing scenarios due to its advantages of easy deployment, moderate cost, and batch controllability.

[0070] This embodiment uses an intelligent rack-mounted power distribution unit that supports the SNMP protocol as the power control unit 300, which enables remote and controllable operation of the mains power supply to the test node 200, providing convenience for centralized control of multiple nodes in batch testing scenarios.

[0071] Figure 2 This document illustrates a flowchart of an embodiment of the NVMe solid-state drive (SSD) abnormal power-loss read / write testing method of the present invention. This NVMe SSD abnormal power-loss read / write testing method is applied to an NVMe SSD abnormal power-loss read / write testing system, such as… Figure 2 As shown, the test method for abnormal power outage read / write operations of this NVMe solid-state drive includes: Step S1: The test node captures the NVMe driver layer I / O timeout event in the Linux kernel log and obtains the I / O performance metrics of the NVMe solid-state drive under test connected to the test node.

[0072] Specifically, a monitoring agent running on the test node captures NVMe driver layer I / O timeout events in the Linux kernel logs in real time. The monitoring agent identifies log entries containing the format "nvmenvmeX: I / O tag ... timeout" by reading ` / dev / kmsg` in real time or periodically checking `dmesg` output, extracting information such as the occurrence time, device identifier, I / O tag, and timeout duration of the timeout event. Simultaneously, the monitoring agent polls the raw I / O counters in ` / proc / diskstats`, performs secondary calculations on the difference between two samples, and derives the average I / O wait time and device activity. The monitoring agent then reports the captured I / O timeout events and the calculated I / O performance metrics to the control server via the communication network.

[0073] Step S2: The control server establishes a baseline based on the historical I / O performance metrics of the NVMe SSD under test, according to the I / O timeout events and I / O performance metrics reported by the test nodes. When the I / O performance metrics of the current round deviate significantly from the baseline and the I / O timeout events reach the severe abnormality threshold, it is determined that the NVMe SSD under test is hardware abnormal.

[0074] Specifically, the control server maintains I / O performance data for each test node over several recent normal cycles and calculates the average and standard deviation as a baseline based on this data. The control server compares the average I / O latency received in the current round with the baseline. When the average I / O latency in the current round deviates significantly from the baseline (e.g., exceeding the average plus three times the standard deviation) and the device busy level exceeds a preset high threshold (e.g., 90%), accompanied by I / O timeout events reaching a severe anomaly threshold, the control server determines that the NVMe SSD under test has a hardware malfunction.

[0075] Step S3: In response to the determination that the NVMe SSD under test is hardware abnormal, the control server reduces the power-off cycle test frequency of the test node through the power control unit. It observes for several cycles in the low-frequency state. If no abnormality is triggered continuously, it gradually restores to the initial frequency. If the NVMe SSD under test is determined to be hardware abnormal again during the restoration process, it reverts to the previous frequency.

[0076] Specifically, when a hardware malfunction is detected, the control server sends a command to the power control unit to extend the power-off cycle interval from its current value to a larger value. After reducing the frequency, the system observes several complete test cycles at the low frequency, continuously monitoring I / O timeout events and I / O performance metrics. If no malfunctions are triggered continuously during the observation period, and the I / O performance metrics fall back to the normal threshold range, the control server gradually restores the test frequency. During the gradual restoration process, if a hardware malfunction is detected again, the control server reverts the test frequency to the previous lower frequency and re-enters the observation state.

[0077] The above steps constitute a complete "monitoring-judgment-adjustment" closed loop. Through the continuous operation of this closed loop, the system can dynamically adjust the testing frequency according to the real-time status of the solid-state drive, ensuring the fault detection rate while avoiding over-testing, thus achieving a balance between testing efficiency and testing reliability.

[0078] The method provided in this embodiment is based on the same inventive concept as the aforementioned system embodiment. The specific implementation details of each step can be found in the corresponding descriptions in the aforementioned system embodiment, and will not be repeated here.

[0079] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A test system for abnormal power failure read / write operations of NVMe solid-state drives, characterized in that, include: The system includes a control server, a test node, and a power control unit. The control server is communicatively connected to the test node and the power control unit, respectively, and the power control unit is electrically connected to the test node. The test node is used to capture NVMe driver layer I / O timeout events in the Linux kernel log and obtain the I / O performance metrics of the NVMe solid-state drive under test connected to the test node. The control server is used to establish a baseline based on the historical I / O performance indicators of the NVMe SSD under test, according to the I / O timeout events and I / O performance indicators reported by the test nodes. When the I / O performance indicators of the current round deviate significantly from the baseline and the I / O timeout events reach the severe abnormality threshold, it is determined that the NVMe SSD under test is hardware abnormal. The power control unit reduces the power-off cycle test frequency of the test nodes. After reduction, it observes for several cycles in the low-frequency state. If no abnormality is triggered continuously, it gradually restores to the initial frequency. If the hardware abnormality of the NVMe SSD under test is determined again during the restoration process, it reverts to the previous frequency. The period during which the test node performs cold data block verification is called the verification window. During the verification window, the anomaly detection of I / O performance indicators is suspended, and the data in the verification window is not included in the historical baseline. After the verification window ends, the clean monitoring window begins, and the monitoring data in the clean monitoring window participates normally in baseline updates and anomaly detection.

2. The NVMe solid-state drive abnormal power failure read / write test system according to claim 1, characterized in that, The control server also includes a cold and hot data construction module, which is used to construct cold data area, hot data area and warm data area on the NVMe solid-state drive under test; The cold data area occupies a preset proportion of capacity. After writing is completed, only read verification is performed. Its data file is divided into multiple cold data blocks of a fixed size. Each cold data block independently calculates a hash value as the integrity verification baseline. The hash value is used to compare and verify with the read data in subsequent power-off cycles.

3. The NVMe solid-state drive abnormal power failure read / write test system according to claim 2, characterized in that, The hot and cold data construction module is also used to record the physical logic block address range corresponding to each cold data block.

4. The NVMe solid-state drive abnormal power failure read / write test system according to claim 3, characterized in that, When a verification error occurs, the control server classifies and determines the distribution of erroneous logical block addresses according to the following rules: continuous block distribution is determined to be a physical bad block, random discrete distribution is determined to be a firmware mapping error, and concentration in a specific capacity area is determined to be a specific physical unit characteristic degradation.

5. The NVMe solid-state drive abnormal power failure read / write test system according to claim 1, characterized in that, The severe anomaly threshold is defined as an I / O suspension time exceeding a preset suspension time threshold, or an I / O timeout event reaching a preset number threshold within a preset time window; the control server is also used to mark an event as a mild anomaly and record the event when an I / O timeout event reaches a preset mild anomaly number threshold within a preset time window.

6. The NVMe solid-state drive abnormal power failure read / write test system according to claim 1, characterized in that, The control server is also used to enter the baseline calibration period after system deployment, and continue to run until a preset number of normal cycles are accumulated. The normal cycle refers to a complete power-off-power-on cycle without I / O timeout events, and no abnormal judgment is triggered during the calibration period. If a sufficient number of normal cycles cannot be accumulated within the preset time, the preset default baseline will be used.

7. The NVMe solid-state drive abnormal power failure read / write test system according to claim 2, characterized in that, The test node is also used to attempt to mount the cold data area after each power-on. If the mounting is successful, it reads the verification through the file path and executes the file fragment query tool to verify the consistency of continuous physical blocks. If mounting fails, the raw partition is read segment by segment based on a pre-recorded list of consecutive physical blocks for verification.

8. The NVMe solid-state drive abnormal power failure read / write test system according to claim 2, characterized in that, The period during which the test node performs the verification of the cold data blocks is a verification window. During the verification window, the anomaly detection of I / O performance indicators is suspended, and the data in the verification window is not included in the historical baseline. After the verification window ends, the clean monitoring window is entered. The monitoring data in the clean monitoring window participates normally in baseline updates and anomaly detection.

9. The NVMe solid-state drive abnormal power failure read / write test system according to claim 1, characterized in that, The power control unit is an intelligent rack-mounted power distribution unit that supports a simple network management protocol.

10. A method for testing read / write operations of an NVMe solid-state drive during abnormal power outages, characterized in that... It is applied to the NVMe solid-state drive abnormal power failure read / write test system according to any one of claims 1-9, the method comprising: The test node captures NVMe driver layer I / O timeout events in the Linux kernel log and obtains the I / O performance metrics of the NVMe SSD connected to the test node. The control server establishes a baseline based on the historical I / O performance metrics of the NVMe SSD under test, according to the I / O timeout events and I / O performance metrics reported by the test nodes. When the I / O performance metrics of the current round deviate significantly from the baseline and the I / O timeout events reach the severe abnormality threshold, it is determined that the NVMe SSD under test is hardware abnormal. In response to the determination that the NVMe SSD under test is experiencing a hardware malfunction, the control server reduces the power-off cycle test frequency of the test node through the power control unit. It observes for several cycles at the low frequency. If no malfunction is triggered continuously, it gradually restores to the initial frequency. If the NVMe SSD under test is determined to be experiencing a hardware malfunction again during the restoration process, it reverts to the previous frequency.

Images