Firmware upgrading method and device, electronic equipment and readable storage medium
By building a hard disk array in a distributed storage system to upgrade SATA SSD firmware in parallel, and combining consistency verification tasks with load, the business interruption problem caused by firmware upgrades in existing technologies is solved, achieving business continuity and high availability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA TELECOM CLOUD TECH CO LTD
- Filing Date
- 2025-12-02
- Publication Date
- 2026-05-05
AI Technical Summary
Existing technologies require pausing the business services of the entire storage node or resource pool when upgrading SATA SSD firmware, resulting in business interruption and failing to meet the high availability requirements of modern data centers.
By building hard disk arrays in a distributed storage system, the hardware foundation for offline upgrades of individual hard disks is provided. A parallel upgrade mode for a single hard disk on multiple storage nodes is adopted, and the upgrade process is dynamically adjusted by associating consistency verification tasks with the load of storage nodes to ensure business continuity.
It enables firmware upgrades for SATA SSDs during business operations without interrupting services. By using multi-node parallel upgrades and dynamic verification tasks, the upgrade cycle is shortened, the upgrade difficulty is reduced, and the high availability and stability of services are ensured.
Smart Images

Figure CN121979548A_ABST
Abstract
Description
Technical Field
[0001] This application belongs to the field of computer technology, and specifically relates to a firmware upgrade method, apparatus, electronic device, and readable storage medium. Background Technology
[0002] With the continuous growth of data volume, distributed storage systems have become a core infrastructure of modern data centers due to their high scalability, high reliability, and high performance. These systems typically employ Serial Advanced Technology Attached Solid State Drives (SATA SSDs) as storage media. SATA SSDs require regular firmware updates to fix potential defects, improve performance, or enhance compatibility.
[0003] Current technology requires pausing all business services on a storage node or resource pool before upgrading the firmware on the SSDs of that node. This approach leads to service interruptions and fails to meet the high availability requirements of modern data centers. Summary of the Invention
[0004] The purpose of this application is to provide a firmware upgrade method, apparatus, electronic device, and readable storage medium that can reduce the impact on the normal operation of front-end services when upgrading firmware on various storage nodes in a distributed storage system, and reduce the difficulty of firmware upgrades.
[0005] In a first aspect, embodiments of this application provide a firmware upgrade method, the method comprising: For each first storage node in the distributed storage system, all hard disks on the first storage node are assembled into a hard disk array; In each round of the loop, the target storage node for each parallel upgrade is determined from all the second storage nodes to be upgraded; For each target storage node, the firmware on a single hard disk to be upgraded in the hard disk array is upgraded, and after the firmware upgrade is completed, a consistency check task is dynamically performed on all hard disks according to the load of the target storage node. After the consistency verification task of all the target storage nodes is completed, the next cycle begins until all the second storage nodes to be upgraded are upgraded.
[0006] Secondly, embodiments of this application provide a firmware upgrade device, the device comprising: The module is used to assemble all the hard disks on each first storage node in a distributed storage system into a hard disk array. The node determination module is used to determine the target storage node for each parallel upgrade from all the second storage nodes to be upgraded in each round of the loop. The upgrade module is used to upgrade the firmware on a single hard disk to be upgraded in the hard disk array for each target storage node, and after the firmware upgrade is completed, to dynamically perform a consistency check task on all hard disks according to the load of the target storage node. The loop module is used to enter the next loop after all the consistency verification tasks of the target storage nodes have been completed, until all the second storage nodes to be upgraded have been upgraded.
[0007] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, and a program or instructions stored in the memory and executable on the processor, wherein the program or instructions, when executed by the processor, implement the steps of the method described in the first aspect.
[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.
[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.
[0010] In this embodiment, for each storage node in the distributed storage system, all hard drives on the storage node are grouped into a redundant array logical disk. During the upgrade process where one physical hard drive is offline, upper-layer services can still use the other physical hard drives in the array logical disk normally. That is, even when one physical hard drive is offline, the array logical disk can still be accessed normally. After the firmware upgrade on the physical hard drive is completed, the physical hard drive is brought back online. Throughout this process, upper-layer services do not block access to the array logical disk, and the upgrade is virtually imperceptible to the upper-layer services, enabling simultaneous service operation and physical hard drive upgrades. Furthermore, after the firmware upgrade on the physical hard drive, this application performs consistency checks between the physical hard drives to ensure data consistency. Moreover, during the consistency check task, this application dynamically adjusts the progress and resource usage of the consistency check by monitoring the system's input / output load in real time, balancing the upgrade efficiency. Attached Figure Description
[0011] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a flowchart illustrating the steps of an embodiment of a firmware upgrade method according to this application; Figure 2 This is a flowchart illustrating the steps of another embodiment of the firmware upgrade method of this application; Figure 3 This is a flowchart illustrating the steps of another embodiment of the firmware upgrade method of this application; Figure 4 This is a structural block diagram of a firmware upgrade device according to this application; Figure 5 This is a structural block diagram of an electronic device provided in this application. Detailed Implementation
[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0015] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and are not limited in number; for example, a first object can be one or more. Furthermore, the term "and / or" in the specification and claims is used to describe the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. In the embodiments of this application, the term "multiple" refers to two or more, and other quantifiers are similar.
[0016] Method Implementation Examples Reference Figure 1 The diagram illustrates a flowchart of an embodiment of a firmware upgrade method according to this application, the method comprising: Step 101: For each first storage node in the distributed storage system, assemble all the hard disks on the first storage node into a hard disk array; Step 102: In each round of the loop, determine the target storage node for each parallel upgrade from all the second storage nodes to be upgraded; Step 103: For each target storage node, upgrade the firmware on the individual hard disk to be upgraded in the hard disk array, and after the firmware upgrade is completed, dynamically perform a consistency check task on all hard disks according to the load of the target storage node. Step 104: After the consistency verification task of all the target storage nodes is completed, enter the next round of loop until all the second storage nodes to be upgraded are upgraded.
[0017] Regarding steps 101 to 104, it should be noted that the firmware upgrade method provided in this application is applied to distributed storage systems to upgrade the firmware on each storage node within the system. Modern high-capacity distributed storage systems commonly employ a business model where hard disk drives (HDDs) are used for data storage, and SSDs are used to accelerate metadata storage across multiple HDDs. Although SSD failure rates are much lower than HDD failure rates, if SSD firmware issues arise, the annual failure rate (AFR) can surge to 3-5% or even higher, posing a significant challenge to business stability. Furthermore, SSD firmware issues are not limited to AFR; they can also cause premature wear and tear, performance degradation, and other serious business problems. Therefore, it is typically necessary to obtain new firmware versions from SSD manufacturers and upgrade the SSDs.
[0018] Currently, SSDs generally include SATA SSDs or Non-Volatile Memory Host Controller Interface Specification (NVME) SSDs. The firmware upgrade principles and upgrade times differ between these two interface protocols. For NVME SSDs, the host sends a Firmware ImageDownload management command to the hard drive device, transferring the firmware to the device's non-volatile memory. After completion, the host sends a Firmware Commit management command to the device, activating the firmware online or through delayed activation. Currently, advanced high-speed serial computer expansion bus standards from various manufacturers, such as Peripheral Component Interconnect Express (PCIe GEN4) or even NVME SSDs, can generally guarantee uninterrupted service during upgrades. Online firmware activation is completed within 1-2 seconds. During this time, the kernel pauses the multi-queue input / output (IO) transmission of the NVME block device, waiting for the firmware activation to complete before resuming transmission. Due to the low latency of NVME, the upper-layer service experiences only about a 1-second IO spike, which is generally imperceptible.
[0019] SATA SSDs upgrade firmware by sending a Download Microcode command from the host to the hard drive. Common modes for this command are 05, 07, 0E, and 0F, corresponding to whether to immediately activate the firmware, download the entire firmware, and save the firmware, respectively. If the SATA SSD is connected to a RAID card and is not configured in pass-through mode, the firmware upgrade must be completed through the RAID card's Command Line Interface (CLI). In principle, the host sends the firmware to the RAID card, which then checks it and sends a Download Microcode command to the hard drive to complete the upgrade. Currently, upgrades for SATA SSDs from various manufacturers typically take 10-30 seconds, significantly longer than NVM ESSD, and online firmware upgrades for SATA SSDs usually cannot be performed while services are running.
[0020] This application embodiment provides a hardware foundation for offline upgrades of single hard drives by pre-assembling hard drive arrays at the node level, ensuring data security and business continuity. It transforms the traditional serial upgrade mode of a single hard drive on a single storage node into a parallel mode of a single hard drive on multiple storage nodes, significantly improving the upgrade efficiency of the entire resource pool. Furthermore, by associating the consistency verification task after the upgrade with the real-time load of the storage nodes and dynamically adjusting strategies, it ensures that this resource-intensive background task does not impact the performance of foreground services, guaranteeing the smoothness of the upgrade process.
[0021] Furthermore, a cyclical control process is adopted to break down the massive upgrade task into multiple manageable rounds until all nodes are upgraded, thus achieving automation and orderliness in the upgrade process.
[0022] A distributed storage system refers to a system composed of multiple storage nodes interconnected via a network. This system distributes data across these nodes, providing a unified and highly available storage service. It's important to note that this solution applies to the management node within the distributed storage system; the management node is the primary implementer of this solution. The management node acts as a logical control center, responsible for scheduling, monitoring, and making decisions regarding firmware upgrade processes across all storage nodes in the entire distributed storage system.
[0023] A storage node refers to the server that actually carries the data in a distributed storage system. Each node is configured with multiple hard drives. The primary storage node is all the storage nodes in the distributed storage system that have been configured into a hard drive array. A hard drive array is a technology that combines multiple physical hard drives into a logical unit using a hardware array card. Redundant Array of Independent Disks (RAID) technology is used to implement hard drive arrays. Through data redundancy, such as mirroring and parity checking, it ensures that data is not lost and service continues in the event of the failure of one or more hard drives.
[0024] The second storage node to be upgraded refers to the storage node that needs firmware upgrade. At the beginning of the upgrade process, the second storage node is all the nodes in the resource pool that are configured with hard drives to be upgraded. As the upgrade progresses, when all the hard drives to be upgraded on a second storage node have been upgraded, the number of second storage nodes to be upgraded will gradually decrease until there are no more second storage nodes, which means that the upgrade task of the entire resource pool is completed.
[0025] The target storage nodes refer to a group of storage nodes selected from the set of second-stage storage nodes to be upgraded in each round of the upgrade cycle, which will undergo the upgrade operation simultaneously. The number of target storage nodes is controlled by a preset parallelism to avoid putting excessive pressure on the storage cluster. The hard drives to be upgraded specifically refer to SATA solid-state drives that require firmware updates. For example, if the firmware version on a hard drive is lower than the firmware version currently provided by the manufacturer, then that hard drive is a hard drive to be upgraded. Firmware is the underlying software written into the hardware device, controlling the basic operation of the device. Upgrading the firmware can fix vulnerabilities and improve performance. Load refers to the resource pressure on the storage nodes during operation, specifically including input and output load.
[0026] Consistency Check (CC) is a background task automatically initiated by the array controller when a hard drive goes offline and then comes back online. This task synchronizes any data updates missed during the hard drive's offline period and verifies the consistency of data across all member disks. Since consistency check consumes input / output resources, its dynamic execution aims to resolve resource contention between the CC task and business operations. In other words, consistency check does not run at a fixed intensity but is intelligently adjusted based on the real-time load of the storage nodes, such as pausing or lowering its priority. When business operations are busy, it automatically yields to business operations; when business operations are idle, it accelerates synchronization to ensure that business performance is not affected during upgrades, enabling upgrades without the business's awareness.
[0027] It should be noted that, in this embodiment, two data structures can be created to store the second storage node and the target storage medium. The data structures can be lists. Specifically, the list of nodes to be upgraded records all second storage nodes to be upgraded. This list is dynamic; nodes are removed after the upgrade is completed. The target node list records the set of target storage nodes selected from the list of nodes to be upgraded in each round of the loop, based on parallelism, for the current upgrade. For example, the identifier of each storage node is stored in the list.
[0028] It's important to note that the disk array setup is completed during the business deployment phase, creating a redundant hardware environment for each storage node. Before system deployment, operations personnel or automated scripts use the hardware RAID card's management tools to configure multiple SATA SSDs on the nodes into one or more redundant logical arrays. This array redundancy ensures that offline operations on individual disks during subsequent upgrades will not result in data loss or service interruption; logically, the business can still access the array.
[0029] When there is an upgrade requirement, such as when a manufacturer releases a new firmware version or when the current firmware version fails, the management node first checks the list of nodes to be upgraded, polls and checks each node in the list, and selects the storage nodes that are currently capable of performing the upgrade task.
[0030] For each node in the target storage node list, the management node upgrades only one hard drive at a time. After the firmware upgrade on the hard drive is complete, a node-level data synchronization task is triggered automatically or manually. At this time, the management node initiates a monitoring and control process, collecting the node's input and output load metrics. Once the load exceeds the threshold, the consistency verification task is immediately paused, and the resource usage of the consistency verification task is adjusted to free up more resource space for business operations. This process continues until the consistency verification tasks of all nodes in the target node list are successfully completed in this round. It should be noted that if all hard drives to be upgraded in the current node have been upgraded, the target storage node is removed from the list of nodes to be upgraded, and it will not be polled again.
[0031] It should be noted that if a node requires manual intervention due to excessive input / output load, the administrator must manually remove it from the list of nodes to be upgraded so that it can be handled separately during off-peak hours. After removal, the management node checks the list of nodes to be upgraded. As long as the list is not empty, the process enters the next loop. When the list of nodes to be upgraded finally becomes empty, it indicates that the firmware upgrade of all storage nodes in the entire distributed storage system is complete.
[0032] In this embodiment, by grouping multiple hard drives into a single array logical disk, SATA SSD upgrades can be performed while supporting business operations without causing I / O suspension or data loss. Typically, there is only about a one-second I / O fluctuation, ensuring a seamless experience for front-end services. In contrast, existing solutions experience 10-30 seconds of I / O fluctuation per disk during upgrades, significantly reducing the difficulty of upgrades. Most current RAID card models support redundant hardware, making this invention applicable to the vast majority of RAID card models and highly versatile in ultra-large-scale cloud service provider production environments. Furthermore, the number of parallel upgrade nodes in the resource pool and the resource usage of consistency verification tasks can be customized according to the planned upgrade cycle and business load, trading time for space without interrupting business operations. This achieves both firmware upgrades and business operation, greatly enhancing the cloud service provider's massive SSD operation and maintenance capabilities and flexibility.
[0033] Reference Figure 2 The diagram illustrates a flowchart of another embodiment of the firmware upgrade method of this application, the method comprising: Step 201: For each first storage node in the distributed storage system, assemble all the hard disks on the first storage node into a hard disk array; Step 201 can be referred to step 101 above, and will not be repeated here.
[0034] Step 202: Determine the second storage node that has not performed the consistency verification task as a candidate storage node; Step 203: If the first number of candidate storage nodes is less than or equal to the preset number of nodes, the candidate storage node is determined as the target storage node; the preset number of nodes is a pre-set upper limit on the number of storage nodes for each parallel upgrade. Step 204: If the first number of candidate storage nodes is greater than the preset number of nodes, select the preset number of candidate storage nodes from the candidate storage nodes and determine them as the target storage nodes.
[0035] Regarding steps 202 to 204, it should be noted that candidate storage nodes refer to storage nodes that, during the management node's polling, do not have any running background tasks for data synchronization and verification initiated by the hardware RAID card. Since a node running consistency verification may mean it is already undergoing an upgrade or is in the midst of periodic data inspection, it is not advisable to issue new upgrade tasks in either case. Skipping these nodes avoids task management chaos and ensures that a node handles only one background job related to array data consistency at a time. This ensures that upgrade tasks are only started on nodes with relatively idle system resources, leaving a buffer for potential dynamic adjustments during subsequent upgrades.
[0036] The preset number of nodes is an integer variable pre-set by the system administrator. It represents the maximum number of storage nodes allowed to undergo firmware upgrades simultaneously across the entire resource pool. The preset number of nodes can be named `upgrade_host_parallel` or a similar name. This parameter acts as a balancing valve between upgrade efficiency and system stability. If the preset number of nodes is set too low, upgrade efficiency will be low and the total time will be too long. If it is set too high, too many nodes performing upgrades and subsequent data synchronization simultaneously may have a cumulative effect, putting enormous pressure on the network and computing resources of the entire cluster and impacting the performance of all services.
[0037] It should be noted that when candidate storage nodes are recorded in a list, the selection can be based on the list length and the preset number of nodes. From a pool of more than the preset number of candidate storage nodes, the final target storage node is selected. This selection can typically be random or based on load weights. The current input / output load metrics of the candidate nodes can be referenced, prioritizing the node with the lightest load to further reduce the risk of the upgrade impacting business operations.
[0038] It should be noted that this application can also be executed during business operation when upgrades are needed after service deployment. The management node traverses the global list of nodes to be upgraded and communicates with each node to check its RAID card status and determine if there is a running consistency verification task. All nodes that have not executed the task are added to a candidate storage node list. The management node compares the length of the candidate storage node list with the preset number of nodes. If the number of candidate nodes is less than or equal to the preset number, it means there are not many candidate nodes, and all can be upgraded. The entire candidate storage node list is directly assigned to the target storage node list for this round. If the number of candidate nodes is greater than the preset number, it means there are enough candidates, but the scale needs to be controlled. In this case, the number of nodes equal to the preset number of nodes needs to be selected from the candidate storage node list according to the selection strategy, and these nodes are used to form the target storage node list for this round.
[0039] The management node locks the target storage node list for this round and begins to issue firmware upgrade commands for the first hard drive to be upgraded to each node in the list in parallel. Simultaneously, it performs parallel upgrades on each storage node, entering the single-disk upgrade and dynamic verification phase within each node. It should be noted that a storage node performing a consistency task in this round may not be performing a consistency task in the next round, and can therefore be identified as a candidate storage node.
[0040] Step 205: For each target storage node, upgrade the firmware on the individual hard disk to be upgraded in the hard disk array.
[0041] Step 205 can be referred to step 103 above, and will not be repeated here.
[0042] Optionally, step 205 may specifically include the following steps: Step S11: For each target storage node, determine the second number of hard drives to be upgraded in the target storage node; Step S12: When the second quantity is zero, stop upgrading the target storage node and remove the target storage node from the second storage node; Step S13: If the second quantity is not zero, select a hard disk to be upgraded from the hard disk array and upgrade the firmware on the hard disk to be upgraded.
[0043] Regarding steps S11 to S13, it should be noted that when the number of physical disks to be upgraded in the target storage node is 0, the management node removes the node from the global list of nodes to be upgraded. The management node will not issue tasks to this node in the next poll and will exit the current upgrade task for that node. If the number of physical disks to be upgraded is greater than 0, an upgrade task is issued to only one physical disk of the same node at a time, and the upgrade node log is periodically retrieved to determine the overall upgrade progress.
[0044] For nodes that do not require upgrading, immediately release their occupied parallel upgrade slots, allowing resources to be allocated to other nodes that truly need upgrading, thereby accelerating the overall process. This automatic removal of nodes reduces manual intervention and enables the entire upgrade process to operate in a closed loop.
[0045] The second quantity refers to the total number of hard drives on the target storage node that need to be upgraded, obtained by querying the hardware RAID card of the target storage node. The management node issues a command to the target storage node, which instructs the RAID card command-line tool on that node to query the firmware version information of all physical hard drives. The queried information is compared with the target firmware version to determine the number of hard drives that need to be upgraded, thus obtaining the second quantity.
[0046] If the number of hard drives to be upgraded is 0, it means that the upgrade information for this node recorded by the management node is outdated, or the node has already been upgraded. In this case, immediately terminate all current and subsequent upgrade tasks for this node. Remove the node's identifier from the global list of nodes to be upgraded. This node is left unused in this upgrade round and will not be scheduled again in subsequent loops. If the number of hard drives to be upgraded is greater than 0, start the single-disk upgrade process for this node. Select one hard drive from the set of hard drives to be upgraded in the storage node's hard drive array as the current operation target. The selection strategy can be based on drive bay order, device ID order, etc. Begin executing the complete upgrade sub-process for this hard drive to be upgraded.
[0047] In this embodiment, storage nodes that have completed the upgrade are automatically removed, avoiding useless queries and waiting on node B, improving the overall upgrade efficiency, and ensuring the accuracy of state management.
[0048] Optionally, step 205 may specifically include the following steps: Step S21: Switch the status of the single hard drive to be upgraded to offline status; Step S22: Upgrade the firmware on the hard drive to be upgraded that is in the offline state; Step S23: After the firmware upgrade is completed, the status of the hard drive to be upgraded is switched to the backup disk status.
[0049] Regarding steps S21 to S23, it should be noted that the prerequisite for switching the status of the single hard drive to be upgraded to offline status during the upgrade operation is to temporarily isolate the target hard drive from the active hardware array. Before taking the hard drive offline, the RAID card command-line tool is first used to check whether the logical disk to which the hard drive belongs, i.e., the redundant array composed of multiple physical disks, is currently in a degraded state. If the logical disk has already been degraded, it indicates that the redundancy capability of the array has been compromised due to the failure of other hard drives. Taking another hard drive offline at this time may cause the entire logical disk to fail and data to be lost.
[0050] If the check result indicates that the drive has been downgraded, the offline operation will not continue. The program will print a message, such as "Logical drive X has been downgraded, and physical drive A cannot be safely upgraded," and exit the current upgrade process for that hard drive, awaiting manual intervention.
[0051] After confirming that the logical disk has not been downgraded, the target physical disk A is set to offline status using the RAID card command-line tool. This removes physical disk A from the current input / output path, and business data will no longer be read or written to this disk. Due to the redundancy of the array, business operations can still access data normally. However, the offline operation may fail due to hardware failure, system overload, or other reasons. The status of physical disk A is checked again using the RAID card command-line tool to confirm that it has been successfully made offline, ensuring that subsequent upgrade operations are performed in a deterministic and secure environment.
[0052] If the offline process fails, the program will print a message and exit to prevent the firmware from being forcibly flashed while the hard drive is still online, which could lead to unpredictable consequences.
[0053] After the hard drive goes offline, use the RAID card command-line tool or a specific tool provided by the hard drive manufacturer to issue a firmware upgrade command to the successfully offline physical disk A. After the upgrade command is completed, check the firmware version number of physical disk A using a query command to confirm whether it has been updated to the target version. If the upgrade fails, the program will print a prompt message, such as "Firmware upgrade for physical disk A failed," and record the error for subsequent manual handling.
[0054] Switching the hard drive to standby status is to restore the array's full redundancy capability by reintegrating the upgraded hard drive into the storage pool. Specifically, the RAID card command-line tool is used to change the status of physical disk A from offline to online or standby. This allows the hard drive to be re-recognized by the array and automatically starts a consistency check task to synchronize data updates missed during its offline period. After bringing the physical disk online, it is checked whether physical disk A has successfully become online or standby. If going online or becoming standby fails, the program will print a message such as "Physical disk A failed to go online," indicating that array redundancy has not been restored and immediate manual intervention is required.
[0055] Step 206: After the firmware upgrade is completed, initialize the resource usage parameters of the consistency verification task on the target storage node; Step 207: Perform a consistency check on all hard drives according to the resource usage parameters; Step 208: Periodically monitor the input and output load of the target storage node, and pause the consistency verification task if the input and output load is greater than or equal to a preset load threshold. Step 209: Reduce the resource usage parameter and re-execute the consistency verification task.
[0056] Regarding steps 206 to 209, it should be noted that the resource usage parameters of the target storage node are not immediately set to run at full speed after the hard disk comes online and the consistency verification task is triggered. Instead, an initial limit is first set for it.
[0057] Resource usage parameters include consistency check ratio and consistency check priority. These two parameters are provided directly by the RAID card and are used to control the intensity of system input / output bandwidth usage by CC tasks.
[0058] Using the RAID card command-line tool, adjust the resource usage parameters to a preset initial value. This initial value is usually an empirical value that ensures the consistency verification task has some progress without significantly impacting most business scenarios. This avoids the task running at maximum intensity from the start, thus preempting business resources and laying the foundation for subsequent fine-grained dynamic adjustment.
[0059] The RAID card hardware automatically performs data synchronization and verification on the entire hard disk array on the target storage node according to resource usage parameters, without intervention from the upper-layer system. It continuously monitors business operations and immediately decides to pause background tasks when pressure is detected.
[0060] Specifically, monitoring the input / output load of a storage node can be done by running a background monitoring script on the target storage node. This script defines a boolean variable, such as `need_collect`, which, when its value is 1, will collect data at a set frequency, for example, every 10-20 seconds. It collects the node's iostat data and focuses on the average wait time (await) metric for input / output requests.
[0061] The preset load threshold is a predefined await time threshold, such as 50ms. When the await data collected in a single instance exceeds this threshold, it is considered that the node's input / output pressure is high. Once the threshold is triggered, the monitoring script immediately uses the RAID card command-line tool to issue a command to the RAID card to pause the current consistency check task. At the same time, the script sets the need_collect variable to 0, pausing unnecessary data collection to reduce the additional pressure on the node from the script itself during periods of high system load.
[0062] Before attempting to re-execute the task, the system will first try to reduce the parameters set during initialization, namely the consistency check ratio or priority. After re-executing the CC task, the need_collect variable will be reset to 1 to resume periodic data collection in preparation for the next potential load spike.
[0063] In this embodiment, the progress and resource usage of consistency verification tasks are controlled in a timely manner based on the input and output load of the storage nodes to ensure stable system operation and reduce the impact on business operations.
[0064] Optionally, step 209 may specifically include: Step S31: If the current resource usage parameter is greater than the preset minimum value, reduce the resource usage parameter and re-execute the consistency verification task; the preset minimum value is a pre-set minimum value for the resource usage parameter. Step S32: When the current resource usage parameter is less than or equal to the preset minimum value, the consistency verification task is re-executed after the pause time of the consistency verification task reaches the preset time.
[0065] For steps S31 and S32, a minimum resource consumption limit is preset for the minimum consistency verification task as a boundary to ensure that the task can obtain the most basic resources and proceed at an extremely slow speed even when the impact on business is minimal, preventing the task from completely stalling.
[0066] The current resource usage exceeding the preset minimum means the system has room to further sacrifice background task efficiency for business performance. The current resource usage is not at its lowest and can be further reduced. Use the RAID card command-line tool to further reduce the consistency check ratio or priority from the current value. For example, reduce it from 30% to 20%, or from "high" priority to "medium" priority. Restart the previously paused consistency check tasks with the new configuration after the parameter reduction. Set the monitoring script's collection flag `need_collect` to 1 to resume periodic performance data collection, preparing for the next adjustment decision.
[0067] Use the RAID card command-line tool to query the current parameters and confirm that they are at the minimum allowed by the system. When the current resource usage parameters are less than or equal to the preset minimum, it means that the system can no longer guarantee business operations by further sacrificing background tasks. This is mainly due to the high business load itself, rather than background tasks competing for resources. Instead of immediately restarting the task, keep it paused and start timing for a preset duration, such as 5 minutes. After the pause time reaches the preset time, do not adjust the parameters; directly retry the consistency verification task with the current minimum parameters. Similarly, set need_collect to 1 to resume data collection. By waiting, it is hoped that a brief business peak can be overcome, and then the task can resume with minimal impact when the business pressure may have eased.
[0068] For example, suppose a node's minimum consistency check ratio is preset to 10%. A task's `await` value exceeds the threshold during runtime and is paused. The system checks and finds the current consistency check ratio is 30%. Because 30% > 10%, the system performs a parameter reduction operation, lowering the ratio from 30% to 15%, and then restarts the task. The task now runs with lower resource consumption, reducing pressure on the business. The task is paused again due to high load. The current ratio is checked and found to be 15%. Because 15% > 10%, the system again lowers the ratio from 15% to 10%, and then restarts the task. The task is paused a third time due to high load. The current ratio is checked and found to be 10%. Because 10% <= 10%, no parameters can be lowered further. The system does not perform any parameter reduction operations but instead pauses the task for 3 minutes. After 3 minutes, the system directly restarts the task, and the consistency check ratio is still at the minimum of 10%.
[0069] In this embodiment of the application, by actively and gradually weakening the resource competitiveness of background tasks, it is attempted to allow the consistency verification task to continue to execute slowly and stably without hindering business operations.
[0070] Step 210: Count the cumulative number of times the input / output load is greater than or equal to the preset load threshold; Step 211: If the cumulative number of times is greater than or equal to the preset number of times, stop the consistency verification task and generate a prompt message; the prompt message is used to indicate manual intervention.
[0071] Regarding steps 210 and 211, it should be noted that the cumulative count refers to the total number of times the system detects that the input / output load exceeds a preset threshold within the upgrade and synchronization cycle of a single hard drive. This counter is for the hard drive currently being processed; once the process for that hard drive ends, the counter will be reset.
[0072] The preset number of attempts is a pre-defined upper limit, such as 3 attempts. Exceeding this number means that the automatic adjustment strategy has failed. At this point, automatic retries will not be performed. The system will generate a clear message in the log file, which can be collected and alerted by management nodes, indicating that manual intervention is required.
[0073] For example, at the start of an upgrade and synchronization cycle for a specific hard drive, the system initializes a counter to 0. Each time the background monitoring script performs a check and triggers a task pause due to the await value of the iostat data exceeding a predefined threshold, the system increments this counter, for example, by 1. The system continuously checks whether the counter's value is greater than or equal to a preset number of times, for example, whether the count is greater than 3. Once the condition is met, the consistency check task for the current node is permanently paused using the RAID card command-line tool, and subsequent automatic processes will not attempt to restart it. The monitoring flag `need_collect` is set to 0 to completely stop the background monitoring script's data collection activity, thus freeing up resources.
[0074] Print explicit error messages to the log file, such as "Node [Node ID] is under excessive input / output pressure, consistency check task has been paused, manual intervention required." Then, exit the background monitoring script for that node. At this point, the automated upgrade process for the current hard drive on that node completely stops, awaiting manual intervention.
[0075] The management node will pull this log, allowing administrators to take highly flexible and adaptable actions. Manual intervention could involve manually completing the synchronization and then restarting the automated process. For example, during off-peak hours, an administrator could manually log into the node, use the RAID card tools to resume the paused consistency check task with the lowest priority, and wait for it to complete. Once the task is complete, the hard drive is considered successfully upgraded. The node remains in the global list of nodes awaiting upgrades, and the automated process will continue to check this node in the next cycle, upgrading its remaining hard drives. The manual intervention only assisted in synchronizing a single disk.
[0076] The firmware upgrade and data synchronization of all remaining hard drives on the node can also be completed manually. After confirming that all hard drives on the node have been successfully upgraded, the administrator can manually remove the node from the global list of nodes to be upgraded. The management node will no longer process the node in subsequent cycles.
[0077] Step 212: After the consistency verification task of all the target storage nodes is completed, enter the next round of loop until all the second storage nodes to be upgraded are upgraded.
[0078] Step 212 can be referred to step 104 above, and will not be repeated here.
[0079] In summary, the firmware upgrade method provided in this application allows the entire upgrade process to be completed during business operation, ensuring business continuity and high availability. By dynamically adjusting the consistency check task, the impact of background data synchronization on front-end business performance is effectively avoided, ensuring low latency and high throughput of business requests, and the end user is unaware of the upgrade process. Multi-node parallel upgrades significantly shorten the firmware upgrade cycle of the entire resource pool. The introduction of an automated cyclic process eliminates the need for repeated manual intervention, and single-disk operations based on redundant hard disk arrays fundamentally guarantee data reliability and business continuity when a single disk is offline. Furthermore, multiple security checks are embedded, forming a comprehensive security protection chain. Dynamically executing the consistency check task also prevents the risk of service instability due to resource overload, improving the overall robustness of the system. When automated processing encounters extreme situations, it can intelligently trigger manual intervention and provides flexible handling methods, demonstrating extremely high human-machine collaboration flexibility.
[0080] Reference Figure 3 The diagram illustrates a flowchart of another embodiment of the firmware upgrade method of this application, the method comprising: A1. During the business deployment phase, build a redundant hard disk array; A2. The management node initializes the list of nodes to be upgraded; A3. Number of nodes to be upgraded in parallel using a pre-configured resource pool at the management node; A4. Deploy and launch the service; A5. During business operation, the management node determines the length of the list of nodes to be upgraded; A6. If the list length is 0, it can be determined that the overall resource pool upgrade has been completed; A7. If the list length is not 0, issue an upgrade task to the upgrade node. A8, Physical disk A on an offline SATA SSD awaiting upgrade; A9. Upgrade the offline SATA SSD physical disk A; A10. After successfully upgrading the firmware on physical disk A, connect the SATA SSD physical disk A. A11. Start the logical drive CC task; A12. Adjust the CC rate according to the resource pool load; A13. Run background scripts to monitor regularly. If the CC task is not completed and the average waiting time for input and output performance data exceeds the threshold, pause the CC task on the logical disk. A14. The average waiting time for input / output performance data returns to normal. Adjust the CC rate and continue the CC task. A15. After the CC task is completed, printing is finished, and the background script exits. A16. When there are other physical disks on this node that need to be upgraded, wait for the next polling by the management node; A17. This node has completed its upgrade and has been removed from the list of nodes to be upgraded.
[0081] In this embodiment, during the business deployment phase, the basic environment is built, redundant hard disk arrays are constructed for each storage node, and the management node initializes the list of nodes to be upgraded and the number of nodes to be upgraded in parallel—key control parameters. During business operation, when an upgrade is needed, an automated upgrade cycle is initiated, with the management node continuously checking the list of nodes to be upgraded. If the list is empty, the upgrade is completed; otherwise, the parallel upgrade process begins. For each target node, a single physical disk is used as the smallest operating unit, and offline, upgrade, and online operations are strictly executed to ensure operational isolation and security. After the hard disk goes online, the data synchronization task triggered enters a dynamic control closed loop. The background script periodically monitors the node's input and output performance (e.g., average waiting time). Once business performance is impacted, such as when the average waiting time exceeds a threshold, the synchronization task is immediately paused, and its execution rate is intelligently reduced according to a preset strategy, or it is allowed to continue after the business peak has passed. After the synchronization task is completed under control, the script automatically exits. Once a node has completed all hard disk upgrades, it is automatically removed from the list of nodes to be upgraded, and the management node then starts the next cycle until all nodes are upgraded.
[0082] In this embodiment, the entire upgrade process is completed during business operation, ensuring business continuity and high availability. By dynamically adjusting the consistency check task, the impact of background data synchronization on front-end business performance is effectively avoided, ensuring low latency and high throughput of business requests, and the end user is unaware of the upgrade process. Furthermore, multiple security checks are embedded, forming a comprehensive security protection chain. Dynamically executing the consistency verification task also prevents the risk of service instability due to resource overload, improving the overall robustness of the system.
[0083] Device Examples Reference Figure 4The diagram illustrates a logical block diagram of a firmware upgrade device according to an embodiment of this application. The device may include: The module 301 is used to assemble all the hard disks on each first storage node in the distributed storage system into a hard disk array. The node determination module 302 is used to determine the target storage node for each parallel upgrade from all the second storage nodes to be upgraded in each round of the loop. The upgrade module 303 is used to upgrade the firmware on a single hard disk to be upgraded in the hard disk array for each target storage node, and after the firmware upgrade is completed, to dynamically perform a consistency check task on all hard disks according to the load of the target storage node. The loop module 304 is used to enter the next loop after all the consistency verification tasks of the target storage nodes have been completed, until all the second storage nodes to be upgraded have been upgraded.
[0084] Optionally, the node determination module includes: The candidate node determination module is used to determine the second storage node that has not performed the consistency verification task as a candidate storage node; The first target node determination module is used to determine the candidate storage node as the target storage node when the first number of candidate storage nodes is less than or equal to a preset number of nodes; the preset number of nodes is a pre-set upper limit on the number of storage nodes for each parallel upgrade. The second target node determination module is used to select the preset number of candidate storage nodes from the candidate storage nodes and determine them as the target storage nodes when the first number of candidate storage nodes is greater than the preset number of nodes.
[0085] Optionally, the upgrade module includes: The hard disk quantity determination module is used to determine a second number of hard disks to be upgraded in each target storage node. The removal module is used to stop upgrading the target storage node when the second quantity is zero, and to remove the target storage node from the second storage node; The upgrade submodule is used to select a hard drive to be upgraded from the hard drive array when the second quantity is not zero, and to upgrade the firmware on the hard drive to be upgraded.
[0086] Optionally, the upgrade module includes: The offline module is used to switch the status of the single hard drive to be upgraded to an offline state; An upgrade module is used to upgrade the firmware on the hard drive to be upgraded, which is in an offline state. The online module is used to switch the status of the hard drive to be upgraded to the backup disk status after the firmware upgrade is completed.
[0087] Optionally, the upgrade module includes: An initialization module is used to initialize the resource usage parameters of the consistency verification task on the target storage node after the firmware upgrade is completed. The task execution module is used to perform a consistency check task on all hard drives according to the resource usage parameters. The monitoring module is used to periodically monitor the input and output load of the target storage node, and to pause the consistency verification task when the input and output load is greater than or equal to a preset load threshold. The reduction module is used to reduce the resource consumption parameters and re-execute the consistency verification task.
[0088] Optionally, the reduction module includes: The reduction submodule is used to reduce the resource usage parameter and re-execute the consistency verification task when the current resource usage parameter is greater than the preset minimum value; the preset minimum value is a pre-set minimum value for the resource usage parameter. The re-execution module is used to re-execute the consistency verification task after the pause time of the consistency verification task reaches a preset time, when the current resource usage parameter is less than or equal to a preset minimum value.
[0089] Optionally, the device further includes: The statistics module is used to count the cumulative number of times the input / output load is greater than or equal to the preset load threshold; The stop module is used to stop the consistency verification task and generate a prompt message when the cumulative number of times is greater than or equal to a preset number of times; the prompt message is used to indicate manual intervention.
[0090] In summary, the firmware upgrade device provided in this application embodiment enables the entire upgrade process to be completed during business operation, ensuring business continuity and high availability. By dynamically adjusting the consistency check task, the impact of background data synchronization on front-end business performance is effectively avoided, ensuring low latency and high throughput of business requests, and the end user is unaware of the upgrade process. Multi-node parallel upgrades significantly shorten the firmware upgrade cycle of the entire resource pool. The introduction of an automated cyclic process eliminates the need for repeated manual intervention, and single-disk operations based on redundant hard disk arrays fundamentally guarantee data reliability and business continuity when a single disk is offline. Furthermore, multiple security checks are embedded, forming a comprehensive security protection chain. Dynamically executing the consistency check task also prevents the risk of service instability due to resource overload, improving the overall robustness of the system. When automated processing encounters extreme situations, it can intelligently trigger manual intervention and provides flexible handling methods, demonstrating extremely high human-machine collaboration flexibility.
[0091] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0092] Reference Figure 5 This is a structural block diagram of an electronic device for firmware upgrades provided in an embodiment of this application. Figure 5 As shown, the electronic device includes: a processor, a memory, a communication interface, and a communication bus. The processor, the memory, and the communication interface communicate with each other through the communication bus. The memory is used to store executable instructions, which cause the processor to execute the method of the aforementioned embodiment.
[0093] The processor can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable devices, transistor logic devices, hardware components, or any combination thereof. The processor can also be a combination that implements computational functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.
[0094] The communication bus may include a path for transmitting information between the memory and the communication interface. The communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.
[0095] The memory may be ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it may be EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory), magnetic tape, floppy disk, and optical data storage devices, etc.
[0096] This application also provides a non-transitory computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device (server or terminal), enables the processor to perform the methods of the foregoing embodiments.
[0097] This application also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above embodiments and achieve the same technical effect. To avoid repetition, it will not be described again here.
[0098] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.
[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0101] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, as well as combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0102] These computer program instructions may also be stored in a computer-readable storage medium capable of directing a computer or other programmable data processing terminal device to operate in a predictive manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0103] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0104] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.
[0105] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0106] The firmware upgrade method, apparatus, electronic device, and readable storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A firmware upgrade method, characterized in that, The method includes: For each first storage node in the distributed storage system, all hard disks on the first storage node are assembled into a hard disk array; In each round of the loop, the target storage node for each parallel upgrade is determined from all the second storage nodes to be upgraded; For each target storage node, the firmware on a single hard disk to be upgraded in the hard disk array is upgraded, and after the firmware upgrade is completed, a consistency check task is dynamically performed on all hard disks according to the load of the target storage node. After the consistency verification task of all the target storage nodes is completed, the next cycle begins until all the second storage nodes to be upgraded are upgraded.
2. The method according to claim 1, characterized in that, After the firmware upgrade is completed, the process of dynamically performing a consistency check on all hard drives based on the load of the target storage node includes: After the firmware upgrade is completed, the resource usage parameters of the consistency verification task on the target storage node are initialized. Perform a consistency check on all hard drives according to the resource usage parameters; The input and output load of the target storage node is periodically monitored, and the consistency verification task is paused when the input and output load is greater than or equal to a preset load threshold. Reduce the resource consumption parameter and re-execute the consistency verification task.
3. The method according to claim 2, characterized in that, The step of reducing the resource consumption parameter and re-executing the consistency verification task includes: If the current resource usage parameter is greater than the preset minimum value, the resource usage parameter will be reduced, and the consistency verification task will be re-executed; the preset minimum value is a pre-set minimum value for the resource usage parameter. If the current resource usage parameter is less than or equal to the preset minimum value, the consistency verification task will be re-executed after the pause time reaches the preset time.
4. The method according to claim 2, characterized in that, The method further includes: Count the cumulative number of times the input / output load is greater than or equal to the preset load threshold; If the cumulative number of attempts is greater than or equal to a preset number, the consistency verification task is stopped and a prompt message is generated; the prompt message is used to indicate manual intervention.
5. The method according to claim 1, characterized in that, In each round of the cycle, determining the target storage node for each parallel upgrade from all the second storage nodes to be upgraded includes: The second storage node that has not performed the consistency verification task is identified as a candidate storage node; If the first number of candidate storage nodes is less than or equal to the preset number of nodes, the candidate storage node is determined as the target storage node; the preset number of nodes is a pre-set upper limit on the number of storage nodes for each parallel upgrade. If the first number of candidate storage nodes is greater than the preset number of nodes, the preset number of candidate storage nodes is selected from the candidate storage nodes and determined as the target storage node.
6. The method according to claim 1, characterized in that, The step of upgrading the firmware on a single hard drive in the hard disk array for each target storage node includes: For each target storage node, determine a second number of hard drives to be upgraded in the target storage node; If the second quantity is zero, stop upgrading the target storage node and remove the target storage node from the second storage node; If the second quantity is not zero, select a hard drive to be upgraded from the hard drive array and upgrade the firmware on the hard drive to be upgraded.
7. The method according to claim 1, characterized in that, The step of upgrading the firmware on a single hard drive in the hard disk array for each target storage node includes: Switch the status of the single hard drive to be upgraded to offline status; Upgrade the firmware on the hard drive to be upgraded, which is in an offline state; After the firmware upgrade is completed, the status of the hard drive to be upgraded is switched to backup drive status.
8. A firmware upgrade device, characterized in that, The device includes: The module is used to assemble all the hard disks on each first storage node in a distributed storage system into a hard disk array. The node determination module is used to determine the target storage node for each parallel upgrade from all the second storage nodes to be upgraded in each round of the loop. The upgrade module is used to upgrade the firmware on a single hard disk to be upgraded in the hard disk array for each target storage node, and after the firmware upgrade is completed, to dynamically perform a consistency check task on all hard disks according to the load of the target storage node. The loop module is used to enter the next loop after all the consistency verification tasks of the target storage nodes have been completed, until all the second storage nodes to be upgraded have been upgraded.
9. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 7.
10. A readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is enabled to perform the method as described in any one of claims 1 to 7.