Automatic verification method of storage device system and storage device system
By proactively downgrading and repairing the firmware version of storage devices through management nodes, the problem of insufficient automated detection and repair capabilities during firmware upgrades in existing technologies is solved. This enables second-level repair of firmware anomalies and lossless business recovery, thereby improving the reliability and autonomy of the storage device system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INSPUR SUZHOU INTELLIGENT TECH CO LTD
- Filing Date
- 2026-06-24
- Publication Date
- 2026-07-21
AI Technical Summary
Existing technologies lack the ability to automatically detect and repair firmware upgrades for storage devices, making it impossible to effectively verify the success or abnormal state of firmware upgrades. This results in state drift issues being difficult to detect and repair in a timely manner, and the automation level of firmware reliability verification is low, leading to inefficiency.
By executing downgrade operations, anomaly detection, and anomaly recovery processes through the management node, the firmware version of the storage device is proactively downgraded, firmware status anomalies are monitored and repaired, and verification reports are generated, thus achieving automated firmware status management.
It enables instantaneous detection and second-level repair of firmware status anomalies, significantly shortening fault recovery time, improving the reliability and autonomy of storage device systems, and ensuring uninterrupted business operation.
Smart Images
Figure CN122431944A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of verification technology for storage device systems, and in particular to an automatic verification method and storage device system for a storage device system. Background Technology
[0002] In related technologies, the new firmware image is manually uploaded by the administrator through the storage device's own out-of-band management interface or a dedicated management network, triggering the upgrade process. During the upgrade, the system typically enters a firmware update mode and automatically restarts upon completion.
[0003] However, the relevant technologies can only perform targeted upgrades from one version to another, lacking proactive detection and automatic repair capabilities. Furthermore, the success of the upgrade and the system's ability to withstand firmware anomalies (such as unexpected version rollbacks) depend entirely on additional manual testing after the upgrade or production failures reported by end users. There is a lack of effective verification mechanisms, and the automation level of firmware reliability verification (such as rollback testing and compatibility testing) is extremely low. Summary of the Invention
[0004] This application provides an automatic verification method and storage device system for a storage device system, so as to at least solve the technical problem of lack of automated verification system firmware in the production environment in the related art.
[0005] This application provides an automatic verification method for a storage device system, executed by a management node, the method comprising: Degradation operation: Actively perform a degradation operation on the firmware of at least one target storage device in the storage device system. The degradation operation includes at least setting the firmware version of the target storage device to a specific firmware version that is lower than the expected firmware version. Anomaly detection: In response to the completion of the downgrade operation, monitor the current firmware version of the target storage device. If the current firmware version is inconsistent with the expected firmware version, determine that the target storage device is in an abnormal firmware state. Anomaly Recovery: In response to an anomaly in the firmware state of the target storage device, the target firmware image matching the expected firmware version is obtained based on the device identifier of the target storage device, and the target firmware image is deployed to the target storage device through an online flashing operation to restore the firmware of the target storage device to the expected firmware version. Report generation: Collect time-series data and performance metrics during the execution of degradation operations, anomaly detection, and anomaly recovery processes, and generate a verification report based on the time-series data and performance metrics.
[0006] This application also provides a storage device system, which includes a management node and multiple storage devices. The management node includes a processor, a memory, and a network interface. The multiple storage devices are communicatively connected to the management node. The memory of the management node stores a computer program. When the processor executes the program, it configures the management node to perform the steps of the automatic verification method of the storage device system described in the following embodiments on at least one of the multiple storage devices.
[0007] Degradation operation: Actively perform a degradation operation on the firmware of at least one target storage device in the storage device system. The degradation operation includes at least setting the firmware version of the target storage device to a specific firmware version that is lower than the expected firmware version. Anomaly detection: In response to the completion of the downgrade operation, monitor the current firmware version of the target storage device. If the current firmware version is inconsistent with the expected firmware version, determine that the target storage device is in an abnormal firmware state. Anomaly Recovery: In response to an anomaly in the firmware state of the target storage device, the target firmware image matching the expected firmware version is obtained based on the device identifier of the target storage device, and the target firmware image is deployed to the target storage device through an online flashing operation to restore the firmware of the target storage device to the expected firmware version. Report generation: Collect time-series data and performance metrics during the execution of degradation operations, anomaly detection, and anomaly recovery processes, and generate a verification report based on the time-series data and performance metrics.
[0008] The automatic verification method for storage device systems provided in this application transforms uncontrollable accidental degradation operations into proactive and controllable ones by setting the firmware of the storage device to perform degradation operations. This method can trigger and verify the system's behavior when the firmware version unexpectedly rolls back, creating realistic fault scenarios and solving the problem that traditional testing cannot safely simulate version drift. After the degradation operation, version comparison is triggered, enabling instantaneous perception of state drift and overcoming the lag and omission risks of manual inspection, ensuring that any version deviation can be detected in milliseconds. Based on the device identifier, the correct firmware image is automatically matched and flashed online, enabling end-to-end automated repair. This compresses the traditional multi-step, multi-person recovery process into a second-level autonomous operation, significantly shortening the average recovery time and achieving lossless business recovery. Hot upgrade technology avoids business system restarts, giving the storage system true autonomy. Through end-to-end data collection, key indicators such as fault detection time, fault recovery time, and business impact are transformed into analyzable indicators, generating verification reports and improving the reliability of the storage device system. Attached Figure Description
[0009] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0010] Figure 1 A flowchart illustrating an automatic verification method for a storage device system provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of a storage device system provided in an embodiment of this application; Figure 3 A flowchart illustrating an automatic verification method for a storage device system provided in another embodiment of this application; Figure 4 A flowchart illustrating an automatic verification method for a storage device system provided in another embodiment of this application; Figure 5 This is a flowchart illustrating an automatic verification method for a storage device system provided in another embodiment of this application. Detailed Implementation
[0011] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0012] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0013] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0014] As cloud storage and data centers continue to expand, the reliability, availability, and maintainability of storage systems become paramount. The firmware of storage controllers (such as the firmware in various types of PCIe card hardware modules) is the cornerstone of their stable operation, and firmware upgrades are a key aspect of their lifecycle management. Currently, firmware upgrades face two main challenges: 1) Risks and Inconsistencies in Firmware Upgrade Status: Traditional online firmware upgrade solutions are prone to bricking or functional degradation when faced with upgrade interruptions, power failures, or compatibility issues. More commonly, after device replacement or maintenance, a "configuration drift" problem can occur, where the firmware version is inconsistent with the expected system version, resulting in a suboptimal or potentially unstable system state. Related technologies lack the ability to automatically detect and repair such anomalies.
[0015] 2) Shortcomings of Firmware Reliability Verification: Traditional firmware testing methods are mostly passive, planned functional verifications. Testers simulate faults through tedious manual steps (such as flashing old firmware and then plugging it back into the device), which is inefficient, non-repeatable, and difficult to cover complex production environment scenarios. On the other hand, manual upgrades are subject to the risk of human error, such as selecting the wrong upgrade package or performing the operation at the wrong time. This cannot effectively verify the system's resilience and self-healing capabilities under real fault conditions.
[0016] The relevant technologies primarily rely on traditional online upgrade solutions based on out-of-band management. This approach involves the administrator manually uploading the new firmware image and triggering the upgrade process via the storage device's own out-of-band management interface, such as the BMC (Baseboard Management Controller), IPMI (Intelligent Platform Management Interface), or a dedicated management network. During the upgrade process, the system typically enters a firmware update mode and automatically restarts upon completion.
[0017] The related technologies have the following technical defects: 1) Passive response, lacking state drift detection and self-healing capabilities: This solution can only perform targeted upgrades from version A to version B. It lacks proactive detection and automatic repair capabilities for state drift issues caused by hardware replacement, accidental downgrades, or configuration errors leading to firmware version discrepancies with system expectations. Maintenance personnel must manually identify and intervene, which is inefficient and prone to oversight.
[0018] 2) Lack of an effective resilience verification mechanism: The process itself lacks self-verification capabilities. Whether the upgrade is successful, and the system's behavior in abnormal states after a successful upgrade, depends entirely on subsequent manual testing or user-reported faults. This is a passive, reactive verification model.
[0019] 3) Low automation of rollback and compatibility testing, resulting in high verification costs: To test the rollback mechanism or system compatibility after an upgrade failure, testers must perform tedious manual operations (e.g., manually refreshing the old firmware and restoring the physical connection for observation). This process is difficult to automate, non-repeatable, and cannot be integrated into a continuous integration / continuous delivery (CI / CD) pipeline, thus becoming a key bottleneck restricting the improvement of firmware quality and delivery efficiency.
[0020] To address the aforementioned technical issues, such as Figure 1 As shown, embodiments of this application provide an automatic verification method for a storage device system, executed by a management node. The method specifically includes the following steps: Degradation operation: Actively perform a degradation operation on the firmware of at least one target storage device in the storage device system. The degradation operation includes at least setting the firmware version of the target storage device to a specific firmware version that is lower than the expected firmware version.
[0021] First, before actively performing a firmware downgrade operation on at least one target storage device in the storage device system, the firmware status information of the target components of each storage device in the storage device system can be scanned; a expected firmware status table is constructed based on the firmware status information of the target components of each storage device; wherein, the firmware status information includes at least the bus address of the target component, the serial number of the target component, the current firmware version, and the expected firmware version.
[0022] A storage device system refers to a collection of multiple independent storage devices (such as storage servers, disk arrays, and server nodes with storage controllers) connected via a network, which work together to provide data storage services.
[0023] The target storage device refers to one or more specific storage devices in the storage device system that are selected to perform verification operations such as firmware downgrade on.
[0024] The target component refers to a specific hardware part inside the storage device that requires firmware management. In this application, it specifically refers to a storage controller or host bus adapter, such as a Serial Attached SCSI (SAS) card (i.e., a Small Computer System Interface (Serial Attached SCSI) card), an NVMe (Non-Volatile Memory Fast Channel) controller, etc., which have independent, writable firmware.
[0025] Firmware status information refers to a set of key data describing the current status of the target component's firmware and its management objectives, which may include: Bus address of the target component: The physical or logical address of the target component on the computer bus, used to uniquely locate the component in the system. For example, the PCIe bus address (High-Speed Peripheral Component Interconnect Bus address), in the format 0000:03:00.0.
[0026] Serial number of the target component: A unique hardware identifier assigned by the target component manufacturer to distinguish this specific hardware instance globally.
[0027] The current firmware version refers to the version number of the firmware currently running within the target component.
[0028] The expected firmware version refers to the correct or latest firmware version that the target component should run on, as specified by the system administrator or policy.
[0029] The expected firmware status table is a structured data set (usually in the form of a database table). Each record corresponds to a target component and stores the component's identification information (bus address, serial number), actual firmware status (current firmware version), and management target (expected firmware version).
[0030] Specifically, the management node polls all storage devices by executing pre-built scanning scripts or calling device management tools (e.g., using the sas3flash-listall command for SAS cards, which lists all SAS devices) to discover the target components (such as SAS HBA cards) that need to be managed within each device. For each discovered target component, its key firmware status information is collected, all collected information is structured, and persistently stored on the management node to form a expected firmware status table.
[0031] Next, through the baseboard management controller interface of the target storage device, a boot image file containing a firmware downgrade tool and a firmware image of a specific firmware version is remotely mounted to the virtual optical drive device of the target storage device; the next boot device of the target storage device is remotely configured to be the virtual optical drive device, and the target storage device is triggered to restart; in response to the target storage device booting from the virtual optical drive device, the firmware downgrade tool is run, and the firmware image of the specific firmware version is flashed to the target component; in response to the completion of flashing the firmware image of the specific firmware version to the target component, the target storage device is triggered to restart again, and the original operating system boot configuration of the target storage device is restored; a downgrade operation completion notification is generated to trigger anomaly detection, and the notification contains at least the identification information of the target storage device.
[0032] The Baseboard Management Controller (BMC) here refers to an independent microprocessor subsystem (usually called a BMC) integrated on the motherboard of hardware such as servers and storage devices. It provides out-of-band management capabilities, meaning it can remotely control power, monitor status, and mount virtual media without relying on the device's main operating system, via a network (such as a dedicated management port). Common protocols are IPMI or Redfish. Both IPMI and Redfish are standardized protocols for out-of-band server management, used for remotely monitoring and controlling the server's hardware status (power supply, temperature, fans, sensors, etc.).
[0033] Firmware downgrade tools refer to dedicated command-line programs or software tools provided by the manufacturer of hardware components, such as SAS cards and RAID cards (disk array cards), used to flash or modify the firmware of those components. Examples include the sas3flash tool for LSI SAS cards and the storcli tool for Broadcom RAID cards.
[0034] A firmware image for a specific firmware version refers to a known firmware binary file (usually in .bin, .rom, or similar format) with a version number lower than the expected firmware version of the current system. This file is the payload for performing a downgrade operation, used to artificially and controllably set the firmware state of the target component from an expected older version.
[0035] A bootable image file is a disk image file (usually in .iso format) that contains a lightweight operating system (such as FreeDOS or Linux) and its runtime environment, firmware downgrade tools, and a firmware image of a specific version. It is designed to be mounted and booted from a virtual drive.
[0036] A virtual optical drive device refers to an optical disc drive simulated on a target storage device through the virtual media function of the baseboard management controller. The management node can "mount" local boot image files to this virtual optical drive over the network, making the target device appear as if a physical optical disc is inserted.
[0037] The original operating system boot configuration refers to the normal boot order set in the BIOS (Basic Input / Output System) / UEFI (Unified Extensible Firmware Interface) of the target storage device before executing this downgrade process. It is usually to boot from the local hard drive to load the host operating system (such as CentOS (Community Enterprise Operating System) or Windows Server Operating System).
[0038] The management node (which can be a server) selects at least one target storage device from the storage device system. Then, remote operation is performed via the target storage device's baseboard management controller interface (e.g., using standard IPMI commands). The controller transfers and mounts a pre-made boot image file to the target device's virtual optical drive. This image file is a self-contained operating environment, internally encapsulating a firmware downgrade tool (such as sas3flash) compatible with the target hardware and a firmware image of a specific firmware version (i.e., an older firmware file) as the downgrade target. After successful mounting, the boot order of the target device is modified via the BMC interface, setting its next boot device to the aforementioned virtual optical drive, remotely triggering a restart of the target device. At this point, the target device's main operating system shuts down, and a lightweight operating system from the boot image is loaded from the virtual optical drive. When the target device boots from the virtual optical drive, a pre-set automated script in the boot image runs automatically. The core of this script is to execute the encapsulated firmware downgrade tool, pointing to the specific firmware version image, and performing a firmware flashing operation on the target component (such as a specified SAS card). This process takes place in an independent, clean environment, completely unaffected by the original host operating system, ensuring the reliability and security of the firmware flashing process and successfully downgrading the firmware version to the preset older version. After the firmware flashing operation is complete, the script automatically commands the device to reboot again. Before rebooting or during the reboot process via BMC, the system restores the device's boot order to its original operating system boot configuration (i.e., booting from the hard drive). After rebooting, the device will normally enter the original host operating system, but at this point, the firmware of its internal target components has been confirmed as an older version. Finally, the management system generates a downgrade completion notification. This notification is a clear internal event, indicating that a controllable firmware state anomaly has been successfully created, and carries the target device identifier to automatically trigger downstream anomaly detection processes. Thus, agentless, cross-operating system proactive downgrade operations are achieved.
[0039] Anomaly Detection: In response to the completion of the downgrade operation, monitor the current firmware version of the target storage device. If the current firmware version is inconsistent with the expected firmware version, determine that the target storage device is in an abnormal firmware state.
[0040] Specifically, the current firmware version of the target component in the target storage device is obtained; the expected firmware version corresponding to the target component is queried from the expected firmware status table; the current firmware version is compared with the expected firmware version; and in response to the current firmware version being lower than the expected firmware version, the target storage device is determined to be in an abnormal firmware state.
[0041] When the management node receives a notification that the downgrade operation is complete or starts detection according to the established inspection cycle, the anomaly detection process is initiated. First, a device query command is executed through the in-band management channel (e.g., logging into the target storage device's operating system using the SSH protocol (Secure Shell protocol)) to obtain the current firmware version of the target component in the target storage device. This operation directly and accurately reads the runtime state of the hardware component. After obtaining the current version, the expected firmware state table established during the initialization phase is accessed. Based on the unique identifier of the target component (such as a PCIe address or serial number), the expected firmware version corresponding to the target component is retrieved from the table. Subsequently, the current firmware version is automatically compared with the expected firmware version. A judgment is made based on the comparison result. In a preferred embodiment of the invention, in response to the current firmware version being lower than the expected firmware version, the target storage device is determined to be in a firmware state anomaly. Here, "lower than" is used to define a version rollback type fault caused by the downgrade operation. Once the condition is met, the target device is marked as an anomaly state, and a structured anomaly confirmation event is generated. This event not only contains the anomaly facts but also carries contextual information (such as component identifier, current version, and expected version), providing precise input for subsequent recovery operations.
[0042] In this way, problems that might have taken hours or even days of manual inspection in the traditional model can be automatically detected in seconds, achieving instantaneous perception and accurate diagnosis of "state drift".
[0043] Anomaly Recovery: In response to an anomaly in the firmware state of the target storage device, the target firmware image matching the expected firmware version is obtained based on the device identifier of the target storage device, and the target firmware image is deployed to the target storage device through an online flashing operation to restore the firmware of the target storage device to the expected firmware version.
[0044] Specifically, based on the device identification information of the target storage device, a target firmware image matching the target component model and the expected firmware version in the target storage device is obtained from the firmware repository; the target firmware image is deployed to the target storage device, and while the operating system of the target storage device is running, an online firmware flashing command is executed to flash the target firmware image to the target component.
[0045] Once the target storage device is confirmed to be in an abnormal firmware state, the recovery process begins immediately. First, based on the target storage device's device identification information (which can be associated with the specific model and hardware ID of the target component), the firmware repository is accessed. From this repository, a target firmware image that exactly matches the target component model in the target storage device and whose version number is strictly equal to the expected firmware version is obtained. After obtaining the correct target firmware image, it is securely transferred or deployed to the target storage device via an in-band management channel (such as SSH). While the target storage device's operating system remains running, an online firmware flashing command is remotely executed. This command invokes a tool provided by the hardware vendor that supports hot upgrades (e.g., specific parameters of sas3flash) to flash the target firmware image into the target component's flash memory. By flashing online, the business interruption caused by the need to restart the entire server to repair firmware is avoided, achieving lossless or minimally damaged business recovery. This makes it possible to perform repairs and verifications during critical business periods, greatly improving system availability. Firmware updates can be performed without restarting the host server, ensuring continuous operation of the business operating system and the services running on it.
[0046] After the flashing command is executed successfully, the firmware of the target component is updated to the expected correct version. Once the entire recovery process is complete, a simplified verification (such as a quick firmware version check) can be optionally triggered again to confirm the recovery result. Finally, the internal state is updated to mark the target device as recovered, thus completing the full convergence from the abnormal to the normal state.
[0047] In this way, the traditional complex recovery operation of manually searching for firmware, selecting a version, and performing flashing is transformed into a fully automated, policy-based standardized process, which significantly shortens the mean time to recovery. Furthermore, by automatically matching and obtaining verified correct images from the firmware repository, the risks of version misselection and file corruption that may occur during manual operation are completely eliminated, thus improving the reliability of the recovery operation.
[0048] In response to flashing the target firmware image to the target component, obtain the actual firmware version of the target component; query the expected firmware version corresponding to the target component from the expected firmware status table; compare the actual firmware version with the expected firmware version; in response to the actual firmware version being consistent with the expected firmware version, confirm that the firmware of the target storage device has been restored to the expected firmware version.
[0049] Immediately after the completion signal of the operation of flashing the target firmware image to the target component, or within a very short delay, a query is initiated again through the in-band management channel to obtain the actual firmware version of the target component. This step collects the real, real-time state of the firmware within the hardware component after the recovery operation is executed. The expected firmware status table, which serves as the authoritative benchmark, is accessed again to query the expected firmware version corresponding to the target component. A final comparison is performed between the actual firmware version and the expected firmware version. The purpose of this comparison is to verify whether the actual version has aligned with the expected version. In response to the comparison result that the actual firmware version matches the expected firmware version, it is confirmed that the firmware of the target storage device has been successfully restored to the expected firmware version. The global status can then be updated, the device can be marked as healthy or verified, and an internal event for this successful recovery can be generated to update the monitoring view or trigger report generation. In this way, it is ensured that the result of the recovery operation meets expectations, thereby truly realizing intelligent and reliable autonomous management of the storage system firmware status.
[0050] Report generation: Collect time-series data and performance metrics during the execution of degradation operations, anomaly detection, and anomaly recovery processes, and generate a verification report based on the time-series data and performance metrics.
[0051] Specifically, in response to the commencement of the degradation operation, time-series data of key events and business performance indicators of the target storage device are collected until the anomaly recovery is completed; in response to the completion of the anomaly recovery, key timeliness indicators are calculated based on the collected time-series data. The key timeliness indicators include at least: the fault detection time from the completion of the degradation operation to the detection of the anomaly, and the fault recovery time from the detection of the anomaly to the completion of the anomaly recovery; in response to the calculation of the key timeliness indicators, the key timeliness indicators and business performance indicators are comprehensively analyzed to generate a verification report. The verification report is used to evaluate the firmware fault self-healing capability and business impact of the storage device system.
[0052] In response to the commencement of the degradation operation, a full-cycle data acquisition task is automatically initiated. On one hand, it continuously collects time-series data of key events, accurately recording the time-series data of each key event, such as degradation completion, anomaly confirmation, and recovery completion. On the other hand, it synchronously collects the target storage device's business performance metrics, such as real-time IOPS and latency. This data acquisition continues until the anomaly recovery is complete, ensuring the capture of end-to-end data throughout the verification process. Upon completion of anomaly recovery, based on the collected complete time-series data, the metric calculation engine is launched. Key time-sensitive metrics are calculated, with core metrics including at least: fault detection time, which is the time interval from the completion of the degradation operation to the detection of the anomaly; fault detection is used to measure the system's sensitivity to state deviations. Fault recovery time is the time interval from the detection of the anomaly to the completion of anomaly recovery, used to measure the efficiency of the system's repair actions. In response to the calculated key time-sensitive metrics, a correlation analysis is performed between the calculated key time-sensitive metrics and the business performance metrics collected throughout the process. For example, the peak and duration of business latency during firmware flashing are analyzed. Based on this comprehensive analysis, a verification report is generated.
[0053] The verification report enables the quantification and visualization of resilience capabilities. Indicators such as Time to Detect Fault (MTTD) and Time to Recover Fault (MTTR) in the report provide benchmarks for setting service level goals and evaluating the effectiveness of improvements.
[0054] The verification report can also achieve an objective assessment of business impact. By comparing the fluctuations in business performance indicators during the verification process, the report can clearly indicate the actual impact of this verification on business services (such as no impact, slight fluctuations, or brief interruptions), enabling the operations and maintenance team to accurately weigh the verification frequency against business risks.
[0055] In one embodiment, when the target storage device is configured with multiple target components of the same functional type, performing a firmware downgrade operation includes: In response to the target storage device's hardware architecture supporting parallel firmware flashing and the current business load being below the first threshold, a component-level batch degradation strategy is executed. In response to the execution of the component-level batch degradation strategy, a firmware degradation execution environment is created through a single out-of-band boot operation, and in this environment, a specific firmware version firmware image is flashed to multiple target components of the same functional type simultaneously through multi-instance concurrent execution. In response to the completion of the flashing operations of all target components, a unified device-level reboot and configuration recovery operation is executed.
[0056] For target storage devices configured with multiple target components of the same functional type (e.g., two or more identical SAS HBA cards (Serial Attached SCSI Host Bus Adapters) installed in a storage server), firmware downgrade operations can select either a component-level batch downgrade strategy or a component-level step-by-step downgrade strategy based on the device's hardware capabilities and real-time load, to achieve the best balance between efficiency and security.
[0057] The management node first assesses the status of the target storage device. In response to the target storage device's hardware architecture supporting parallel firmware flashing (e.g., the device's baseboard management controller or firmware flashing tool allows simultaneous programming operations on multiple similar components, each with its own independent flashing channel, without interference), and the current service load being below a first threshold (i.e., the management node monitors the target device's current service input / output volume, CPU utilization, and other metrics, determining that they are below a preset safety threshold), this indicates that the device is in a relatively idle state and has a strong ability to withstand potential performance impacts or temporary unavailability risks from parallel operations.
[0058] When the above conditions are met simultaneously, a component-level batch downgrade strategy is executed. Specifically, a unified firmware downgrade execution environment is created for the target storage devices through a single out-of-band boot operation (i.e., only one configuration and restart via the BMC interface). This environment is typically a lightweight operating system mounted from the network, containing the necessary tools and firmware images. Within this unified execution environment, a specific firmware version image is simultaneously flashed to multiple target components of the same functional type through concurrent execution of multiple instances. This means that multiple SAS cards receive and update firmware in parallel within the same flashing cycle, significantly reducing the overall operation time. In response to the completion of the flashing operation for all target components (the system monitors and confirms that all parallel flashing instances have successfully completed), a unified device-level restart and configuration recovery operation is executed. That is, the device is controlled to restart once and its normal boot sequence is restored uniformly, allowing the device to re-enter the operating system with the downgraded firmware for all components. Subsequently, a batch downgrade completion notification is generated.
[0059] In this way, under the premise of controllable risks, the hardware potential is maximized, compressing the original serial, multi-component degradation operation that required multiple restarts into a single, efficient parallel operation. This significantly reduces the overall verification task time window, making it particularly suitable for large-scale scenarios with short maintenance windows or a large number of device components. At the same time, its business load-based triggering conditions ensure that the operation will not have an unacceptable impact on critical business operations, achieving a balance between automation and security.
[0060] In response to the target storage device's hardware architecture not supporting parallel firmware flashing or the current service load exceeding a second threshold, a component-level step-by-step degradation strategy is executed. In response to this strategy, each target component is sequentially processed through an independent out-of-band degradation operation loop, ordered by its physical location priority or logical service priority within the target storage device. Each out-of-band degradation operation loop includes: performing a dedicated firmware flashing operation on a single target component; performing a single-component functional self-test on the target component upon completion of the dedicated firmware flashing operation; and performing a partial service traffic switching verification upon successful completion of the single-component functional self-test. This verification includes: temporarily switching some or all of the service traffic currently carried by the degraded target component to other undegraded or recovered target components within the same device, while maintaining the overall online status of the target storage device, and verifying service continuity and performance stability. Upon successful completion of the partial service traffic switching verification, the next sequence of target components is determined and executed based on its physical location priority or logical service priority.
[0061] When the target storage device is not suitable for the batch strategy due to hardware limitations or heavy business, a more prudent and secure component-level step-by-step degradation strategy is adopted. This strategy performs orderly and controlled verification of multiple components one by one while ensuring business continuity.
[0062] Specifically, when the target storage device's hardware architecture does not support parallel firmware flashing (e.g., resource limitations in the flashing channel or management tools only supporting serial operations) or the current business load exceeds the second threshold (the device is processing critical business flows), a component-level step-by-step degradation strategy is executed. In response to this strategy, the target components are first sorted according to their physical location priority or logical service priority (e.g., prioritizing spare cards or components with lower loads in a redundant architecture), and then an independent out-of-band degradation operation loop is executed for each target component in sequence. Each loop constitutes a complete degradation-verification-switch micro-process: first, a dedicated firmware flashing operation is performed on a single target component; immediately after this operation, a single-component functional self-check (e.g., link status, basic read / write tests) is performed to ensure the basic functions of the component are normal after degradation; after the self-check passes, the critical business assurance stage begins, performing partial business traffic switching verification. That is, while keeping the device online as a whole, some or all of the business traffic originally carried by the currently degraded component is temporarily switched to other undegraded or recovered components within the same device, verifying business continuity and performance stability, thereby testing the device's redundancy and load balancing capabilities when some components are abnormal. In response to the successful verification of partial switching of business traffic, the out-of-band degradation operation of the next target component is determined and executed in a loop according to the established order, and this process is repeated until all components have been degraded.
[0063] In this way, by isolating risks from multiple components and distributing them into independent micro-loops, and by verifying verification through local traffic switching, verifiable destruction rather than service destruction can be achieved. This approach can adapt to hardware architectures that do not support parallel writing and can also adopt a serial approach that causes less business interference under high load.
[0064] In one feasible implementation, before performing the degradation operation, the topology information of the target storage device in the storage system can be obtained. This topology information includes the physical rack identifier of the target storage device, the logical fault domain identifier it belongs to, and the service level of the services it carries. Based on the obtained topology information, the execution methods of the degradation operation and anomaly detection steps are adapted and adjusted. Specifically, when the service level is high, the execution time of the degradation operation is adjusted to a preset off-peak business period. When a logical fault domain contains multiple storage devices, all storage devices within that fault domain are divided into multiple verification batches, and only one storage device is verified at a time. The device performs a downgrade operation and subsequent steps; when the physical rack identifier indicates that the target storage device is located at the network edge, the timeout threshold for waiting for the version query response in the anomaly detection step is extended; after completing the anomaly recovery step, the correspondence between the topology relationship information and the key timeliness indicators calculated during this verification process is recorded, and the records in the topology health database are updated based on the correspondence; the topology health database is used to record the historical performance of the self-healing capability of storage devices in different topology locations, and to provide a reference for the subsequent selection of new target storage devices. The reference includes prioritizing storage devices corresponding to topology locations with historical self-healing capability performance lower than the preset level.
[0065] Before initiating a downgrade operation, you can proactively obtain the topology information of the target storage device within the storage system. This topology information is a multi-dimensional set of environmental tags, specifically including: Physical rack identifier: refers to the physical rack or server location number where the target device is located, used to identify its physical deployment location.
[0066] Logical fault domain identifier: refers to the logical isolation unit (such as availability zone, rack group, power group) to which the target device belongs. Devices within a fault domain share the same fault risk (such as being on the same switch or using the same power supply), and its identifier is used to control the fault blast radius.
[0067] Service level: refers to the importance level of the service carried by the device (such as core, important, general), which is usually determined based on the recovery time objective (RTO) and recovery point objective (RPO) of the service system.
[0068] After obtaining the above information, the execution methods of the degradation operation and anomaly detection steps are adapted and adjusted based on the obtained topology information. For example, when the business service level is high, in order to avoid interfering with critical business, the system automatically adjusts the execution time of the degradation operation to a preset off-peak business period (such as late at night), thereby verifying the system's resilience while ensuring absolute stability during peak business periods.
[0069] When a logical fault domain contains multiple storage devices, in order to prevent the simultaneous verification of multiple devices sharing a risk boundary from causing cascading risks, the system automatically divides all storage devices in the fault domain into multiple verification batches and controls that only one storage device is degraded and subsequent steps are performed at any given time.
[0070] When the physical rack identifier indicates that the target storage device is located at the network edge (such as cross-regional data centers or sites with high network latency), considering that network fluctuations may affect the response speed of detection commands, the system automatically extends the timeout threshold for waiting for version query responses in the anomaly detection step, avoiding misjudgment of detection failure due to network latency and improving the robustness of the verification process in complex network environments.
[0071] After completing the anomaly recovery steps, the system records the correspondence between topology relationship information and key timeliness indicators (such as MTTD and MTTR) calculated during the verification process. This correlated data is used to update the records in the topology health database. The topology health database is a structured knowledge base specifically designed to store and analyze the historical verification performance of devices under different topology dimensions (such as specific racks or fault domains). The core value of this database lies in providing data-driven decision support for subsequent verifications: its recorded historical self-healing capabilities (e.g., a longer average recovery time for devices within a rack) can serve as a reference for selecting new target storage devices. Specifically, storage devices corresponding to topology locations with historical self-healing capability performance below a preset level can be prioritized for the next round of verification.
[0072] For example, if historical data shows that equipment recovery times in rack-A are generally slow, the system can prioritize verifying the equipment in that rack to proactively expose and resolve potential performance bottlenecks or configuration issues. In this way, by utilizing topology information, the system achieves synergy between verification tasks and business SLAs, respect for infrastructure risk boundaries, and adaptation to real-world network conditions.
[0073] The automatic verification method for the storage device system in this application is executed by a management node. The management node includes a processor, a memory, and a network interface. The memory stores a computer program. When the processor executes the program, it configures the management node to implement the automatic verification method for the storage device system.
[0074] Please see Figure 2 The management node in this application can be configured with five modules: a monitoring and reporting module (including an indicator collector and a report generator), a fault injection module (including an experiment scheduler and an automated degradation executor), a firmware status management module (including an expected firmware status table and a device discovery service), and a status recovery module (including an anomaly detector and an intelligent recovery executor).
[0075] The fault injection module is used to actively perform a downgrade operation on the firmware of at least one target storage device in the storage device system. The downgrade operation includes at least setting the firmware version of the target storage device to a specific firmware version that is lower than the expected firmware version.
[0076] The firmware status management module is used to monitor the current firmware version of the target storage device in response to the completion of the downgrade operation, and to determine that the target storage device is in an abnormal firmware state in response to the current firmware version being inconsistent with the expected firmware version.
[0077] The status recovery module is used to respond to the target storage device being in an abnormal firmware state. Based on the device identifier of the target storage device, it obtains the target firmware image that matches the expected firmware version, and deploys the target firmware image to the target storage device through an online flashing operation to restore the firmware of the target storage device to the expected firmware version.
[0078] The monitoring and reporting module is used to collect time-series data and performance indicators during the execution of degradation operations, anomaly detection, and anomaly recovery processes, and to generate verification reports based on the time-series data and performance indicators.
[0079] The storage device system of this application includes multiple storage devices (storage device cluster) and a management node. The multiple storage devices are communicatively connected to the management node. The management node is configured to execute an automatic verification method of the storage device system on at least one of the multiple storage devices.
[0080] In a specific example, this application also configures a first network and a second network. Specifically, the first network is a management network through which management nodes connect to the baseboard management controller of each storage device in the storage cluster in an out-of-band management manner; the second network is a service / management data network through which management nodes connect to the operating system of each storage device in the storage cluster in an in-band management manner; wherein, the management node is configured to: perform firmware downgrade operations on selected target storage devices through the first network, and perform firmware state anomaly detection and anomaly recovery operations on target storage devices through the second network.
[0081] In a specific example, each storage device is equipped with one or more SAS cards for connecting to the back-end JBOD expansion enclosure.
[0082] Regarding system configuration and environment, this application adopts a combined software and hardware configuration. For example, the hardware environment can be configured with three storage devices, each equipped with two SAS chip HBA cards. One management server (management node) is also configured. All storage devices and the management server are equipped with BMC chips and connected to the management network.
[0083] In terms of software environment, a management server can be configured to execute the verification method of the storage device system provided in this application. The management server components are written in Python and remote task scheduling is performed using Ansible (an open-source automated operation and maintenance tool). The storage node operating system can be CentOS (a community enterprise operating system). The firmware operation tool can use the sas3flash tool. Regarding firmware version settings: the expected firmware version is 10.23.01.00. The old firmware version used for downgrading is 10.22.00.00.
[0084] Please see Figure 3 In one specific embodiment, the automatic verification method for the storage device system provided in this application specifically includes the following steps: Step 1: System initialization and construction of expected firmware state table.
[0085] After the storage device system boots up, the firmware status management module automatically scans all SAS cards in the cluster using a script. It executes the command `sas3flash -listall` to obtain the PCIe address, serial number, and current firmware version of each card. This information is then stored in a local database, forming the expected firmware status table. The initial state of the expected firmware status table can be shown in Table 1. Table 1
[0086] Step 2: Configure and trigger the fault injection strategy (degradation operation).
[0087] The experimental scheduler for the fault injection module is configured with a strategy: for example, every Monday at 2 a.m., one or more storage nodes are randomly selected and their SAS cards are degraded.
[0088] Please see Figure 4 Storage device B1 was selected as the target storage device. The automated degradation executor of the fault injection module was triggered, executing the following automated process: 1) Connect to the Baseboard Management Controller (BMC) of storage device B1 using the Intelligent Platform Management Interface Tool (ipmitool) command. The command can be as follows: ipmitool -H 192.168.1.102 -U admin -P password.
[0089] 2) Using the virtual media function of the Baseboard Management Controller (BMC), a pre-built disk operating system (DOS) boot image containing a serially connected small computer system interface flash tool (sas3flash) and old firmware fw_v10.22.00.00.bin is mounted to storage device B1.
[0090] 3) Set the next boot device for storage device B1 to the virtual optical drive and perform a restart: Intelligent Platform Management Interface Tool (ipmitool) ... Power Reset.
[0091] 4) After storage device B1 boots from the disk operating system (DOS) image, it automatically executes an automatic downgrade batch file (auto_downgrade.bat), whose core command can be: sas3flash -c 0 -f -o -firmware.
[0092] 5) After the downgrade is complete, the script automatically restarts the node. The system restores the boot order to hard disk boot via the Baseboard Management Controller (BMC).
[0093] 6) Storage device B1 boots normally into the CentOS (Central Enterprise Operating System) system. At this time, its Serial Connect Small Computer System Interface (SAS) card firmware version has been downgraded to 10.22.00.00.
[0094] 7) The fault injection module sends a message to the state recovery module: "Fault injection complete, target: storage device B1".
[0095] Step 3: Automatic detection and recovery, please refer to [link / reference]. Figure 5 .
[0096] Upon receiving notification from the fault injection module, the anomaly detector of the state recovery module immediately initiates a rapid verification of storage device B1. It connects to storage device B1 via Secure Shell Protocol (SSH), executes a command to list all serially connected small computer system interface flash memory tools, and confirms that its firmware version is indeed 10.22.00.00, lower than the expected 10.23.01.00. The anomaly is confirmed, and storage device B1 is marked as state-abnormal. The intelligent recovery executor of the state recovery module then begins operation: Compatibility check: Based on the Serial Connect Small Computer System Interface (SAS) card model (Logic System Integration 3508, LSI 3508) of storage device B1, retrieve the corresponding correct firmware fw_v10.23.01.00.bin from the firmware repository.
[0097] 2) Perform an online upgrade: Distribute the firmware file to storage device B1 using Ansible and execute the online upgrade command: sas3flash-cf -o-firmware fw_v10.23.01.00.bin. During this process, the operating system of storage device B1 does not need to be restarted, and business input / output (I / O) only experiences a brief fluctuation during the upgrade.
[0098] 3) Result verification: After the upgrade is completed, check the firmware version again to confirm that it has been successfully upgraded to 10.23.01.00.
[0099] Step 4: Monitoring and Reporting.
[0100] Throughout the process, the indicator collector in the monitoring and reporting module continuously collects data: For example, a timeline could be: 02:00:00 - Fault injection experiment begins.
[0101] 02:07:30 - Storage device B1 has completed its downgrade and entered the operating system.
[0102] 02:07:35 - The status recovery module detected an anomaly.
[0103] 02:09:15 - Automatic upgrade process completed.
[0104] Key metrics: Fault Detection Time (MTTD): 5 seconds (from system launch to detection of an anomaly).
[0105] Fault recovery time (MTTR): 1 minute 40 seconds (from the detection of the fault to the complete repair).
[0106] Total experiment time: 9 minutes and 15 seconds.
[0107] Business impact: Virtual machines on storage device B1 experienced a brief loss of connectivity during the downgrade process (during the restart). There were slight fluctuations in I / O performance during the upgrade process, but no service interruption.
[0108] The report generator generates a report: The system automatically generates a resilience assessment report, which includes the above timeline, indicators, experimental results (success), and concludes that "the system performs robustly under the current fault injection event, the automatic recovery mechanism is effective, and the business impact is within an acceptable range."
[0109] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in the embodiments of the automatic verification method of any of the above storage device systems when it is run.
[0110] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), mobile device, magnetic disk, or optical disk.
[0111] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0112] The above provides a detailed description of an automatic verification method for a storage device system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.
Claims
1. An automatic verification method for a storage device system, characterized in that, The method is executed by the management node, and the method includes: Degradation operation: Actively perform a degradation operation on the firmware of at least one target storage device in the storage device system, the degradation operation including at least setting the firmware version of the target storage device to a specific firmware version that is lower than the expected firmware version; Anomaly detection: In response to the completion of the downgrade operation, monitor the current firmware version of the target storage device; in response to the current firmware version being inconsistent with the expected firmware version, determine that the target storage device is in an abnormal firmware state. Anomaly Recovery: In response to an abnormal firmware state of the target storage device, a target firmware image matching the expected firmware version is obtained based on the device identifier of the target storage device, and the target firmware image is deployed to the target storage device through an online flashing operation to restore the firmware of the target storage device to the expected firmware version. Report generation: Collect time-series data and performance metrics during the execution of the degradation operation, anomaly detection, and anomaly recovery process, and generate a verification report based on the time-series data and performance metrics.
2. The automatic verification method for a storage device system according to claim 1, characterized in that, Prior to actively performing a firmware downgrade operation on at least one target storage device in the storage device system, the following steps are included: Scan the firmware status information of the target components of each storage device in the storage device system; A desired firmware status table is constructed based on the firmware status information of the target components of each storage device. The firmware status information includes at least the bus address of the target component, the serial number of the target component, the current firmware version, and the expected firmware version.
3. The automatic verification method for a storage device system according to claim 1, characterized in that, The active degradation operation on the firmware of at least one target storage device in the storage device system includes: The boot image file, which includes a firmware downgrade tool and a firmware image of a specific firmware version, is remotely mounted to the virtual optical drive device of the target storage device through the baseboard management controller interface of the target storage device. The system remotely configures the next boot device for the target storage device to be the virtual optical drive, and triggers the target storage device to restart. In response to the target storage device booting from the virtual optical drive device, the firmware downgrade tool is run to flash the specific firmware version image to the target component; Upon completion of flashing the specific firmware version image to the target component, the target storage device is triggered to restart again, and the original operating system boot configuration of the target storage device is restored. A downgrade operation completion notification is generated to trigger anomaly detection. The notification contains at least the identification information of the target storage device.
4. The automatic verification method for a storage device system according to claim 1, characterized in that, The step of monitoring the current firmware version of the target storage device and determining that the target storage device is in an abnormal firmware state in response to a discrepancy between the current firmware version and the expected firmware version includes: Obtain the current firmware version of the target component in the target storage device; Query the expected firmware version corresponding to the target component from the expected firmware status table; Compare the current firmware version with the expected firmware version; In response to the current firmware version being lower than the expected firmware version, it is determined that the target storage device is in an abnormal firmware state.
5. The automatic verification method for a storage device system according to claim 1, characterized in that, The step of obtaining a target firmware image matching the expected firmware version based on the device identifier of the target storage device, and deploying the target firmware image to the target storage device through an online flashing operation includes: Based on the device identification information of the target storage device, obtain the target firmware image from the firmware repository that matches the target component model in the target storage device and has the expected firmware version; The target firmware image is deployed to the target storage device, and while the operating system of the target storage device is running, an online firmware flashing command is executed to flash the target firmware image to the target component.
6. The automatic verification method for a storage device system according to claim 5, characterized in that, The method further includes: In response to flashing the target firmware image to the target component, the actual firmware version of the target component is obtained; Query the expected firmware version corresponding to the target component from the expected firmware status table; Compare the actual firmware version with the expected firmware version; In response to the actual firmware version being consistent with the expected firmware version, the firmware of the target storage device is confirmed to be restored to the expected firmware version.
7. The automatic verification method for a storage device system according to claim 1, characterized in that, The process of collecting time-series data and performance metrics during the degradation operation, anomaly detection, and anomaly recovery, and generating a verification report based on the time-series data and performance metrics, includes: In response to the commencement of the degradation operation, time-series data of key events and business performance indicators of the target storage device are collected until the anomaly recovery is complete; In response to the completion of anomaly recovery, based on the collected time series data, key timeliness indicators are calculated. The key timeliness indicators include at least: the fault detection time from the completion of the degradation operation to the detection of the anomaly, and the fault recovery time from the detection of the anomaly to the completion of the anomaly recovery. In response to the calculation of the key timeliness indicators, the key timeliness indicators and the business performance indicators are comprehensively analyzed to generate a verification report. The verification report is used to evaluate the firmware fault self-healing capability and business impact of the storage device system.
8. The automatic verification method for a storage device system according to claim 1, characterized in that, When the target storage device is configured with multiple target components of the same functional type, performing a firmware downgrade operation includes: In response to the target storage device's hardware architecture supporting parallel firmware flashing and the current service load being below a first threshold, a component-level batch degradation strategy is executed. In response to the execution of the component-level batch downgrade strategy, a firmware downgrade execution environment is created through a single out-of-band boot operation, and the specific firmware version firmware image is simultaneously flashed to the multiple target components of the same functional type through a multi-instance concurrent execution method within the firmware downgrade execution environment. Once the flashing operation of all target components is completed, a unified device-level reboot and configuration recovery operation will be performed. In response to the target storage device’s hardware architecture not supporting parallel firmware flashing or the current business load being higher than the second threshold, a component-level step-by-step degradation strategy is executed. In response to the execution of the component-level step-by-step degradation strategy, the target components are sorted according to their physical location priority or logical service priority in the target storage device, and an independent out-of-band degradation operation loop is executed for each target component in turn.
9. The automatic verification method for a storage device system according to claim 8, characterized in that, The out-of-band degradation operation cycle includes: Perform a dedicated firmware flashing operation on a single target component; In response to the completion of the dedicated firmware flashing operation, a single-component function self-test of the target component is performed; In response to the successful self-test of the single component function, a partial switching verification of service traffic is performed. The partial switching verification of service traffic includes: while keeping the target storage device online as a whole, temporarily switching some or all of the service traffic originally carried by the currently degraded target component to other undegraded or recovered target components within the same device, and verifying service continuity and performance stability. In response to the successful verification of the partial switching of the service traffic, the out-of-band degradation operation loop for the next target component is determined and executed according to the physical location priority or logical service priority.
10. A storage device system, characterized in that, The system includes: The management node includes a processor, memory, and a network interface; Multiple storage devices are communicatively connected to the management node; The memory of the management node stores a computer program, and when the processor executes the program, the management node is configured to perform an automatic verification method for the storage device system as described in any one of the plurality of storage devices on at least one of the storage devices as claimed in any one of claims 1 to 9.