Data reconstruction method and device, electronic equipment and storage medium

By dynamically calculating redundancy and adjusting erasure coding type by monitoring disk status data, the problem of resource consumption and network bandwidth occupation during data reconstruction is solved, achieving a balance between efficient data recovery and low cost, and is suitable for large-scale distributed storage systems.

CN120687035BActive Publication Date: 2026-07-03JINAN INSPUR DATA TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
JINAN INSPUR DATA TECH CO LTD
Filing Date
2025-06-16
Publication Date
2026-07-03

AI Technical Summary

Technical Problem

Existing technologies cannot dynamically balance data recovery performance, resource consumption, and network bandwidth usage during data reconstruction, especially in large file scenarios, which affects the efficiency and resource consumption of real-time data transmission tasks.

Method used

By monitoring the status data of the disk to be reconstructed, its redundancy is dynamically calculated, and the erasure coding type is adjusted according to different redundancy levels. An incremental recovery mechanism is used to transmit the minimum amount of data, achieving efficient balance in the data recovery process.

Benefits of technology

In the event of a disk failure or expansion, the redundancy and erasure coding type are dynamically adjusted to reduce network bandwidth usage, improve data recovery efficiency and speed, maintain a balance between high performance and low cost, and ensure the stability of front-end services.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120687035B_ABST
    Figure CN120687035B_ABST
Patent Text Reader

Abstract

The application discloses a data reconstruction method and device, electronic equipment and storage medium, and relates to the technical field of data storage and recovery. In the case that a to-be-reconstructed disk exists in a distributed storage system, the redundancy of the to-be-reconstructed disk is determined based on state data of the to-be-reconstructed disk; a target redundancy level corresponding to the redundancy of the to-be-reconstructed disk is determined from a plurality of preset redundancy levels; different redundancy levels correspond to different erasure code types; the erasure code types corresponding to different redundancy levels represent different combinations of data recovery rates and storage resource occupation ratios; the erasure code type of the to-be-reconstructed disk is switched to a target erasure code type corresponding to the target redundancy level, and the target erasure code type is used to transmit a fault data block in the to-be-reconstructed disk to a specified disk for data reconstruction, thereby solving the problem that data recovery performance, resource consumption and network bandwidth resource occupation cannot be dynamically balanced in the data reconstruction process in the related art.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of data storage and recovery technology, and in particular to data reconstruction methods, apparatus, electronic devices and storage media. Background Technology

[0002] A distributed storage system is a system that provides distributed storage services to external cloud platforms. Based on distributed storage, other distributed storage services such as object storage services, block storage services, and file system storage services can be provided to the platform. The most fundamental element of a distributed storage system is the disk, where all data is stored. When a disk fails or storage needs to be expanded, the data on the failed disk or the disk to be expanded needs to be reconstructed, that is, the data on the failed disk or the disk to be expanded needs to be recovered on other normal hard drives.

[0003] Related data reconstruction techniques involve copying all data from a failed disk or a disk to be expanded to other healthy disks. However, this method consumes a significant amount of network bandwidth, especially in large file scenarios (such as satellite remote sensing), which can interfere with real-time data download tasks, resulting in low recovery efficiency and high resource consumption. Summary of the Invention

[0004] This application provides a data reconstruction method, apparatus, electronic device, and storage medium to at least solve the problem in the related art that the data reconstruction process cannot dynamically balance data recovery performance, resource consumption, and network bandwidth resource occupation.

[0005] This application provides a data reconstruction method, comprising: monitoring the status data of the disk to be reconstructed in a distributed storage system, and determining the redundancy of the disk to be reconstructed based on the status data; determining a target redundancy level corresponding to the redundancy of the disk to be reconstructed from a plurality of preset redundancy levels; different redundancy levels in the plurality of redundancy levels correspond to different redundancy intervals, and different redundancy levels correspond to different erasure coding types; the erasure coding types corresponding to different redundancy levels represent different combinations of data recovery rate and storage resource occupancy ratio; switching the erasure coding type of the disk to be reconstructed to the target erasure coding type corresponding to the target redundancy level; transmitting faulty data blocks in the disk to be reconstructed to a designated disk according to the target erasure coding type; and performing data reconstruction on the designated disk; the designated disk is a normal disk in the distributed storage system.

[0006] This application also provides a data reconstruction apparatus, comprising: a redundancy calculation module, used to monitor the status data of the disk to be reconstructed in a distributed storage system, and determine the redundancy of the disk to be reconstructed based on the status data; a redundancy level determination module, used to determine a target redundancy level corresponding to the redundancy of the disk to be reconstructed from a preset plurality of redundancy levels; different redundancy levels in the plurality of redundancy levels correspond to different redundancy ranges, and different redundancy levels correspond to different erasure coding types; the erasure coding types corresponding to different redundancy levels represent different combinations of data recovery rate and storage resource occupancy ratio; and a reconstruction module, used to switch the erasure coding type of the disk to be reconstructed to the target erasure coding type corresponding to the target redundancy level, transfer the faulty data blocks in the disk to be reconstructed to a designated disk according to the target erasure coding type, and perform data reconstruction on the designated disk; the designated disk is a normal disk in the distributed storage system.

[0007] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for implementing the steps of any of the above-described data reconstruction methods when executing the computer program.

[0008] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data reconstruction methods.

[0009] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data reconstruction methods.

[0010] This application pre-sets multiple redundancy levels, each corresponding to a different redundancy range and erasure coding type. The erasure coding type for each redundancy level represents a different combination of data recovery rate and storage resource utilization. Based on the real-time status data of the disk to be reconstructed, the redundancy of the disk is dynamically calculated. Based on the calculated redundancy, a target redundancy level is dynamically determined from the pre-set multiple redundancy levels. Compared to static redundancy strategies, dynamic redundancy calculation ensures rapid data recovery during failures while avoiding storage waste caused by excessive redundancy, effectively balancing data reliability and resource efficiency. Furthermore, the dynamic grading mechanism can adjust the redundancy based on the real-time status data of the disk to be reconstructed. It automatically adjusts the data redundancy level and erasure coding type, enabling flexible matching of the optimal recovery strategy within different redundancy ranges. Unlike full copying, this embodiment employs an incremental recovery mechanism using erasure coding. When the disk to be reconstructed is undergoing data reconstruction, it does not simply copy all the data. Instead, based on the redundancy information in the erasure coding, it calculates and transmits only the minimum amount of data required to recover the faulty data blocks. This reduces the amount of data transmitted during the data recovery process, thereby reducing the consumption of storage and transmission resources, improving the efficiency and speed of data recovery, ensuring the optimal balance between high performance and low cost during the data reconstruction process of the disk to be reconstructed, maintaining the stability of front-end services, and solving the problem in related technologies where data recovery performance, resource consumption, and network bandwidth resource occupation cannot be dynamically balanced during data reconstruction. Attached Figure Description

[0011] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is an application scenario diagram of a data reconstruction method provided in an embodiment of this application.

[0013] Figure 2 This is a flowchart of a data reconstruction method provided in an embodiment of this application.

[0014] Figure 3 A flowchart of another data reconstruction method provided in an embodiment of this application.

[0015] Figure 4 This is a structural diagram of another data reconstruction apparatus provided in an embodiment of this application. Detailed Implementation

[0016] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0017] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0018] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0019] According to one aspect of the embodiments of this application, a data reconstruction method is provided. Optionally, in this embodiment, the above-described data reconstruction method may be applied, but is not limited to, to applications such as... Figure 1 The diagram shows a distributed storage system comprising multiple disks. These disks can interact with each other, and each disk includes a disk to be reconstructed (102) and multiple normal disks (104). The disk to be reconstructed (102) refers to the disk among the multiple disks that requires data reconstruction, such as a failed hard drive or a disk requiring expansion. The normal disks (104) refer to the disks among the multiple disks that are currently storing data.

[0020] The aforementioned disks include, but are not limited to, hard disk drives (HDDs), solid state drives (SSDs), and hybrid hard drives (SSHDs).

[0021] The data reconstruction method in this application embodiment can be executed by a distributed storage system. Figure 2 This is a flowchart illustrating an optional data reconstruction method according to an embodiment of this application, such as... Figure 2 As shown, the process of this method may include the following steps:

[0022] Step S202: In the case of a disk to be reconstructed in the distributed storage system, monitor the status data of the disk to be reconstructed, and determine the redundancy of the disk to be reconstructed based on the status data of the disk to be reconstructed.

[0023] The data reconstruction method of this application can be applied to the field of data storage and recovery technology, specifically in scenarios such as satellite remote sensing data management, large-scale distributed storage clusters, high-performance computing environments, Internet of Things (IoT) data storage, video surveillance, and storage. For example, when a disk in a satellite storage system fails, the data reconstruction method of this application can quickly and efficiently recover satellite remote sensing data, reduce the risk of data loss, and reduce the bandwidth consumption of the satellite-to-ground link. It is particularly suitable for scenarios where satellite remote sensing platforms need to process a large number of remote sensing images in real time or near real time. For example, in a large distributed storage system composed of thousands or even tens of thousands of disks, when some disks in the large distributed storage system fail or are expanded, the data reconstruction method of this application can dynamically adjust redundancy and resource allocation according to the real-time system status and data characteristics to balance recovery speed and resource efficiency. It is suitable for large-scale data storage environments such as data centers and cloud computing platforms.

[0024] Distributed storage systems use algorithms to group disks into storage pools, providing secure and reliable storage services. However, the most significant characteristic of distributed storage is its large-scale nature; a cluster can consist of tens of thousands of disks forming a massive storage pool, with data volumes potentially reaching petabytes (PB) or even exabytes (EB). When a distributed storage system has a faulty disk or a disk requiring expansion, data reconstruction is necessary to redistribute the data from the faulty or expanded disk to new disks, ensuring data integrity and consistency. Data reconstruction is a form of data recovery, and recovery speeds vary depending on the type of data. For example, in satellite remote sensing scenarios, data is typically large, high-resolution, and requires high real-time performance, and may be distributed across different orbits and sensors. If storage devices fail, recovering this data requires efficient methods, while also considering the data's importance and timeliness. However, related data reconstruction techniques involve copying all data from the faulty or expanded disk to other healthy disks. This method consumes significant network bandwidth, especially in large file scenarios, such as satellite remote sensing where the satellite-to-ground link bandwidth utilization exceeds 80%, interfering with real-time data download tasks and resulting in low recovery efficiency and high resource consumption.

[0025] Therefore, in order to solve this problem, the embodiments of this application determine the redundancy of the disk to be reconstructed based on the state data of the disk to be reconstructed. The redundancy changes dynamically with the state data of the disk to be reconstructed, and the erasure coding parameters are automatically adjusted according to the dynamic redundancy to achieve a dynamic balance between data recovery performance and network bandwidth resource consumption.

[0026] In this context, "disks to be rebuilt" refers to disks in a distributed storage system that have failed or are about to undergo data migration for expansion. The status data of these disks describes their health condition and the current operating status of the distributed storage system, used to calculate disk redundancy and determine subsequent reconstruction strategies. For example, in a satellite remote sensing data management scenario, status data for disks to be rebuilt can be obtained from the storage manager, fault logs, and satellite-to-ground link status monitors. This status data includes, but is not limited to, the remaining storage capacity of the node (C...). free ), historical node failure rate (p fail ), satellite-to-ground link bandwidth constraints (B) limit ) and other parameters.

[0027] The redundancy of the disk to be reconstructed refers to the ratio of the amount of additional data stored by the distributed storage system to the original amount of data in order to ensure data integrity and reliability. It guides the selection of erasure coding types to achieve optimal data recovery performance and resource efficiency. In this embodiment, the redundancy of the disk to be reconstructed is calculated based on real-time status data; that is, the redundancy of the disk to be reconstructed changes dynamically based on real-time status data. For example, the health index of the disk to be reconstructed can be predicted based on its status data. Based on the health index, a target health level is determined from multiple health levels. Different health levels correspond to different redundancies, and the redundancy corresponding to the target health level is determined as the redundancy of the disk to be reconstructed. For example, the status data of the disk to be reconstructed is input into a pre-trained prediction model to obtain the redundancy of the disk to be reconstructed. The prediction model is used to determine the redundancy corresponding to the disk based on the input disk status data.

[0028] Step S204: Determine the target redundancy level corresponding to the redundancy of the disk to be reconstructed from a set of preset redundancy levels; different redundancy levels correspond to different redundancy ranges, and different redundancy levels correspond to different erasure coding types; the erasure coding types corresponding to different redundancy levels represent different combinations of data recovery rate and storage resource occupancy ratio.

[0029] Multiple redundancy levels are standards for measuring data redundancy in a distributed storage system, corresponding to different data recovery strategies. Each level is associated with a specific redundancy range and erasure coding type. Since the redundancy of the disk to be reconstructed changes dynamically, its redundancy level also changes dynamically. The target redundancy level refers to a level selected from multiple preset redundancy levels based on the current state of the distributed storage system and the redundancy calculation results of the disk to be reconstructed.

[0030] Erasure coding is a data redundancy technique that protects stored data from damage or loss by generating check blocks. Unlike full copying, this embodiment uses an incremental recovery mechanism with erasure coding. When the disk to be reconstructed is reconstructed, instead of simply copying all the data, it calculates and transmits only the minimum amount of data required to recover the faulty data blocks based on the redundancy information in the erasure code. This reduces the amount of data transmitted during the data recovery process, thereby reducing the network bandwidth usage. Different erasure code types (such as RS(6,2), RS(8,3), RS(10,4)) represent different encoding methods, and different erasure code types correspond to different redundancy data ratios. The redundancy data ratio represents the ratio of the number of check blocks after encoding to the total number of data blocks. The numbers in different erasure code types represent the total number of data blocks and the number of check blocks after encoding. Different encoding methods result in different data recovery speeds and storage resource usage ratios. For example, in RS(6,2), "6" represents a total of 6 data blocks after encoding, while "2" indicates that 2 of them are redundant check blocks. This means that, based on the original data, the distributed storage system generates two additional parity blocks, bringing the total number of data blocks to six. Under the RS(6,2) encoding method, the original data is divided into four data blocks, and two redundant parity blocks are generated simultaneously, resulting in a redundancy rate of 33% (the two parity blocks account for one-third of the total six blocks). Therefore, during data recovery, only a smaller number of data blocks are needed for computation to quickly reconstruct the complete data. For example, the classification of multiple redundancy levels and their corresponding explanations are shown in Table 1 below:

[0031] Table 1

[0032]

[0033] Table 1 categorizes redundancy levels into high, medium, and low, where γ represents redundancy. High redundancy (e.g., RS(6,2), redundancy ≥ 0.5) indicates a large amount of redundant data, suitable for scenarios with surging storage node failure rates or unstable links. Medium redundancy (e.g., RS(8,3), redundancy 0.3–0.5) balances data integrity and resource efficiency, suitable for scenarios with normal storage pressure and sufficient bandwidth. Low redundancy (e.g., RS(10,4), redundancy < 0.3) is suitable for scenarios with sufficient storage redundancy but limited link bandwidth. This redundancy grading strategy ensures that the distributed storage system can automatically select the most suitable erasure coding type under different conditions, achieving the optimal balance between data recovery speed and storage resource consumption.

[0034] To address the issue of the inability to dynamically balance data recovery performance, resource consumption, and network bandwidth usage during data reconstruction in related technologies, this implementation proposes that the higher the redundancy level, the greater the bandwidth required, and the erasure coding type will have a higher ratio of check blocks to data blocks (i.e., the ratio of redundant data).

[0035] Step S206: Switch the erasure coding type of the disk to be reconstructed to the target erasure coding type corresponding to the target redundancy level. According to the target erasure coding type, transfer the faulty data blocks in the disk to be reconstructed to the designated disk and perform data reconstruction on the designated disk. The designated disk is a normal disk in the distributed storage system.

[0036] Among them, the target erasure code type refers to the erasure code type corresponding to the target redundancy level of the disk to be reconstructed.

[0037] A designated disk specifically refers to a healthy disk selected during the data reconstruction process to receive and store faulty data blocks from the disk to be reconstructed. The number of designated disks can be one or more, depending on actual needs. Designated disks typically possess sufficient storage space and are in good health to undertake the data reconstruction task and ensure smooth data recovery. In this embodiment, the designated disk is selected from the distributed storage system through a certain filtering mechanism. For example, by comprehensively evaluating the disk's storage capacity, read / write speed, network bandwidth, and current task load, the disk with the most abundant resources and the lowest load is dynamically selected as the designated disk for the data reconstruction task.

[0038] Optionally, faulty data blocks on the disk to be reconstructed are marked and prepared for migration. Through a network transmission mechanism, the faulty data blocks are transferred from the disk to be reconstructed to a designated disk selected by a load balancing algorithm, ensuring that the designated disk has sufficient storage space and network bandwidth to receive and process the data. The data reconstruction process is then initiated on the designated disk, using the switched target erasure coding type to decode and reconstruct the data blocks, restoring data integrity.

[0039] This embodiment pre-sets multiple redundancy levels, each corresponding to a different redundancy range and erasure coding type. The erasure coding type for each redundancy level represents a different combination of data recovery rate and storage resource utilization. Based on the real-time status data of the disk to be reconstructed, the redundancy of the disk is dynamically calculated. Based on the calculated redundancy, a target redundancy level is dynamically determined from the pre-set multiple redundancy levels. Compared to static redundancy strategies, dynamic redundancy calculation ensures rapid data recovery during failures while avoiding storage waste caused by excessive redundancy, effectively balancing data reliability and resource efficiency. Furthermore, the dynamic grading mechanism can adjust the redundancy based on the real-time status data of the disk to be reconstructed. According to the invention, the system automatically adjusts the data redundancy level and erasure coding type, enabling flexible matching of the optimal recovery strategy within different redundancy ranges. Unlike full copying, this embodiment employs an incremental recovery mechanism using erasure coding. When the disk to be reconstructed is undergoing data reconstruction, it does not simply copy all the data. Instead, based on the redundancy information in the erasure coding, it calculates and transmits only the minimum amount of data required to recover the faulty data blocks. This reduces the amount of data transmitted during the data recovery process, thereby reducing the consumption of storage and transmission resources, improving the efficiency and speed of data recovery, ensuring the optimal balance between high performance and low cost during the data reconstruction process of the disk to be reconstructed, maintaining the stability of front-end services, and solving the problem in related technologies where data recovery performance, resource consumption, and network bandwidth resource occupation cannot be dynamically balanced during data reconstruction.

[0040] In one exemplary embodiment, the status data of the disk to be reconstructed is used to indicate the following information about the disk to be reconstructed: remaining storage capacity and historical failure rate;

[0041] Among them, the remaining storage capacity (C free This refers to the amount of unused storage space on the disk to be reconstructed, typically measured in TB or larger. In satellite remote sensing scenarios, the remaining storage capacity of the disk to be reconstructed can be calculated in real time using the onboard storage management module.

[0042] Historical failure rate (p fail The number represents the frequency of disk failures to be reconstructed within a past period (e.g., the past 24 hours), including but not limited to device crashes, disk failures, and network interruptions. In this embodiment, the historical failure rate is calculated by statistically analyzing failure events within a specific time window. The historical failure rate can be calculated using the following formula (1):

[0043]

[0044] In some embodiments, determining the redundancy of the disk to be reconstructed based on its status data includes: determining the percentage of used storage capacity of the disk to be reconstructed based on its total storage capacity and remaining storage capacity; and weighting and summing the percentage of used storage capacity and the historical failure rate based on a first weighting coefficient of the percentage of used storage capacity and a second weighting coefficient of the historical failure rate to obtain the redundancy of the disk to be reconstructed. The first weighting coefficient and the second weighting coefficient are dynamically adjusted based on the recovery efficiency index of the disk to be reconstructed.

[0045] The percentage of used storage capacity on the disk to be reconstructed refers to the proportion of data currently stored on the disk to its total storage capacity. This percentage can be calculated by monitoring the total and remaining storage capacity of the disk in real time.

[0046] The first weighting coefficient quantifies the importance of the percentage of used storage capacity in computational redundancy. The second weighting coefficient quantifies the importance of the historical failure rate in computational redundancy. Both weighting coefficients are dynamically adjusted based on the recovery efficiency metrics of the disk to be reconstructed. The recovery efficiency metrics refer to multiple parameters that comprehensively consider the quality, speed, and resource consumption of data recovery under the current state of the disk to be reconstructed and the actions performed. For example, the recovery efficiency metrics can be shown in Table 2 below:

[0047] Table 2

[0048] Indicator Type Key parameters sampling frequency Storage pressure <![CDATA[C free / C total ]]> 1 minute Link status Bandwidth utilization, packet loss rate 5 minutes Recovery performance Data reconstruction speed and success rate real time

[0049] This embodiment monitors recovery efficiency indicators and dynamically adjusts the first and second weighting coefficients to optimize redundancy calculation, thereby achieving a more efficient and intelligent data recovery strategy. The redundancy of the disk to be reconstructed can be expressed using the following formula (2):

[0050]

[0051] Where α represents the first weighting coefficient, β represents the second weighting coefficient, and p fail C represents the historical failure rate. free C represents the remaining storage capacity of the disk to be reconstructed. total This indicates the total storage capacity of the disk to be reconstructed; This indicates the percentage of remaining storage, reflecting storage pressure.

[0052] In this embodiment, the percentage of used storage capacity and the historical failure rate of the disk to be reconstructed are used as core parameters in the calculation of redundancy. By weighted summation of the percentage of used storage capacity and the historical failure rate, the storage pressure and failure risk are comprehensively evaluated. The redundancy can be intelligently adjusted according to the actual status of different disks, thereby determining the accurate redundancy. This ensures data security under high failure risk and avoids unnecessary resource consumption under low storage pressure.

[0053] In an exemplary embodiment, the first weight coefficient and the second weight coefficient are dynamically adjusted using reinforcement learning techniques. The data reconstruction method further includes: monitoring the recovery efficiency index of the disk to be reconstructed within a specified time period to obtain index parameters corresponding to the specified time period; inputting the index parameters into a pre-trained reinforcement learning model to obtain the first weight coefficient and the second weight coefficient; and using the reinforcement learning model to adjust the first weight coefficient and the second weight coefficient based on the input index parameters.

[0054] Among them, the indicator parameters for a specified time period specifically refer to the specific values ​​of the recovery efficiency indicators collected by the distributed storage system within the current time window.

[0055] In this embodiment, the first weight coefficient and the second weight coefficient are dynamically optimized based on the recovery efficiency index through a reinforcement learning model. The reinforcement learning model takes the index parameters of the recovery efficiency index as input, and after learning and iteration, outputs the optimized first weight coefficient and the second weight coefficient. These are used to adjust the influence ratio of the used storage capacity ratio and the historical failure rate in subsequent redundancy calculations, enabling the distributed storage system to adaptively optimize resource allocation strategies, improve data recovery efficiency, and reduce resource waste.

[0056] In this embodiment, the first and second weighting coefficients are not fixed, but are dynamically optimized using a reinforcement learning algorithm based on the recovery efficiency index of the disk to be reconstructed. This means that the distributed storage system will automatically adjust the first and second weighting coefficients according to the recovery efficiency index to find the redundancy that best suits the current environment, thereby achieving effective resource allocation and intelligent optimization of data recovery strategies.

[0057] In one exemplary embodiment, the training method for a reinforcement learning model includes the following steps:

[0058] Multiple rounds of training are performed until the reinforcement learning model meets the training termination condition, resulting in a trained reinforcement learning model. During the current round of training, a current state is selected from a pre-constructed state space, and a current action is selected from a pre-constructed action space. The state space includes multiple states, where the state parameter of one state is a sample index parameter. The action space includes multiple actions, where the action parameter of one action includes action instructions for adjusting the redundancy level. The current action is executed using the current reinforcement learning model, and the immediate reward value corresponding to the current action is determined according to the reward function. Based on the immediate reward value and the value function, the action value between the current state and the current action is determined, and the current reinforcement learning model is adjusted accordingly. The action value includes a first expected long-term return and a second expected long-term return. The value function represents the mapping relationship between the action value and the first weight coefficient, the second weight coefficient, the first expected long-term return, and the second expected long-term return.

[0059] The state space(s) contains a set of possible states of a distributed storage system. The state parameters of each state are sample index parameters, such as historical redundancy, node failure rate, and network bandwidth fluctuation coefficient.

[0060] The action space (a) defines the set of all actions that the reinforcement learning model can take, such as adjusting the redundancy level to high, medium, or low. These actions determine the resource allocation method and data recovery strategy of the distributed storage system. The reinforcement learning module explores different actions to find the optimal resource allocation scheme in the current state.

[0061] The immediate reward value is the feedback value calculated according to the reward function after the current action is executed. It reflects the direct impact of the action on system performance (such as data recovery success rate, bandwidth utilization, task time, etc.) in the current state. In this embodiment, the reward function can be calculated according to the following formula (3):

[0062]

[0063] Wherein, ω1, ω2, and ω3 are weighting coefficients used to balance reliability, efficiency, and cost; recovery success rate is the proportion of data blocks that are successfully recovered; bandwidth usage is the actual bandwidth used for redundant data transmission; and task time is the time taken to rebuild the data (in seconds).

[0064] Action value is the expected long-term return calculated using an immediate reward value and a value function, including a first expected long-term return and a second expected long-term return. The first expected long-term return specifically refers to the expected level of data integrity, consistency, and high availability that the distributed storage system can maintain over a long timescale. It measures the long-term ability of the system to effectively recover data, ensure business continuity, and safeguard user data security in the face of various failures and anomalies. The second expected long-term return refers to the long-term performance of the distributed storage system's resource utilization efficiency and data processing speed. It reflects the ability of the distributed storage system to complete data recovery with minimal resource consumption while ensuring data security, avoiding excessive bandwidth consumption and wasted computing resources, and simultaneously guaranteeing the speed and quality of data recovery. The value function characterizes the mathematical relationship between action value and the first weight coefficient, the second weight coefficient, the first expected long-term return, and the second expected long-term return, guiding the reinforcement learning model to learn and select behaviors that maximize long-term returns. In this embodiment, action value can be the sum of the first weighted result and the second weighted result, where the first weighted result refers to the weighted result of the first weight coefficient and the first expected long-term return, and the second weighted result refers to the weighted result of the second weight coefficient and the second expected long-term return. Thus, the value function can be expressed by the following formula (4):

[0065] Q(s,a;α,β)=α·Q reliability (s,a)+β·Q efficiency (s,a) (4)

[0066] Where Q(s,a;α,β) represents the action value; Q reliability (s,a) represents the first expected long-term return of the current state-current action pair; Q efficiency (s,a) represents the second expected long-term reward of the current state-current action pair. In this embodiment, the initial value of α is 0.6 and the initial value of β is 0.4. α and β are dynamically adjusted through training the reinforcement learning model.

[0067] In this embodiment, the first expected long-term benefit and the second expected long-term benefit are respectively associated with the reliability and resource efficiency of data recovery, and the first weight coefficient and the second weight coefficient are embedded in the value function. By adjusting the values ​​of the first weight coefficient and the second weight coefficient, the value function can better seek a dynamic balance between reliability (first expected long-term benefit) and efficiency (second expected long-term benefit).

[0068] In one exemplary embodiment, transferring faulty data blocks in the disk to be reconstructed to a designated disk according to the target erasure coding type includes: dividing the faulty data blocks in the disk to be reconstructed into multiple sub-fault blocks according to the target erasure coding type; constructing multiple subtasks; each subtask corresponds one-to-one with a sub-fault block; a subtask within the multiple subtasks is used to transfer the corresponding sub-fault block to the designated disk for data reconstruction; adjusting the computing resources of the multiple subtasks according to the redundancy of the disk to be reconstructed, and using the adjusted multiple subtasks to transfer the multiple sub-fault blocks to the designated disk for data reconstruction.

[0069] A faulty data block refers to a data unit stored on a faulty disk that is inaccessible or incomplete due to disk failure. A sub-fault block is a smaller data unit further divided from a faulty data block according to the target erasure coding type. By dividing a faulty data block into multiple sub-fault blocks, data recovery can be performed simultaneously on multiple specified disks, thereby shortening the overall reconstruction time. Multiple sub-fault blocks include multiple data blocks and multiple parity blocks, where the ratio of multiple parity blocks to multiple data blocks is determined according to the redundancy ratio of the target erasure coding type. For example, if the target erasure coding type is RS(8,3), then the faulty data block is divided into 8 sub-fault blocks, where the 8 sub-fault blocks include 5 data blocks and 3 parity blocks.

[0070] Subtasks are reconstruction tasks assigned separately to each sub-fault block during the data recovery process. Each subtask corresponds to the recovery process of a sub-fault block, including data transmission and reconstruction computation. By executing multiple subtasks in parallel, the overall efficiency of data recovery can be significantly improved.

[0071] Computing resources refer to the hardware resources such as CPU and memory required to execute subtasks. Based on the redundancy level of the disk to be reconstructed, the system dynamically adjusts the computing resources allocated to subtasks to balance the recovery speed of redundant data with the bandwidth usage of the satellite-to-ground link, ensuring the rational utilization of resources.

[0072] Optionally, Figure 3 A flowchart of another data reconstruction method provided in the embodiments of this application is shown below. Figure 3As shown, the distributed storage system includes a status monitoring module, a dynamic decision-making module, a resource scheduling module, and a feedback optimization module. The status monitoring module monitors the status data of the disk to be reconstructed. The feedback optimization module monitors recovery efficiency indicators during data reconstruction and uses a reinforcement learning model to perform reinforcement learning on the parameters of the monitored recovery efficiency indicators, obtaining a first weight coefficient and a second weight coefficient. These first and second weight coefficients are then fed back to the dynamic decision-making module. Based on the first and second weight coefficients and the status data of the disk to be reconstructed, the dynamic decision-making module calculates the redundancy of the disk to be reconstructed, determines the target redundancy level corresponding to the redundancy level, and switches the erasure coding type of the disk to be reconstructed to the target erasure coding type corresponding to the target redundancy level. The resource scheduling module divides the faulty data blocks in the disk to be reconstructed into multiple sub-fault blocks according to the target erasure coding type. Based on the calculated redundancy of the disk to be reconstructed, the erasure coding reconstruction task is divided into multiple sub-tasks, with each sub-task corresponding to a sub-fault block. The module also adjusts the ratio of data recovery rate and network bandwidth usage, and uses the adjusted sub-tasks to transfer the sub-fault blocks to the designated disk for data reconstruction. Dynamic threads are used to avoid resource overload.

[0073] In this embodiment, based on the target erasure coding type, the faulty data block on the disk to be reconstructed is subdivided into multiple sub-fault blocks, and an independent subtask is constructed for each sub-fault block. This strategy not only facilitates parallel processing and improves data recovery speed, but also flexibly allocates resources according to the specific needs of the data block, avoiding resource waste; dynamically adjusting the computing resources required to execute the subtask greatly saves network bandwidth resources, balancing the contradiction between data recovery performance and resource consumption.

[0074] In some embodiments, the computing resources of multiple subtasks are adjusted according to the redundancy of the disk to be reconstructed, including: collecting the status data of each specified disk in real time, and calculating the disk redundancy of each specified disk based on the status data of each specified disk; determining the risk level of each specified disk based on the disk redundancy of each specified disk; different risk levels correspond to different resource allocation strategies and priorities; and determining the priority and resource allocation strategy corresponding to each subtask based on the risk level of the specified disk to which each subtask belongs.

[0075] Optionally, the status monitoring module is activated to collect status data of each specified disk in real time, including remaining storage capacity, historical failure rate, network bandwidth status, etc. The status data of each specified disk is analyzed in real time, and the disk redundancy of each specified disk at the current moment is obtained according to the preset redundancy calculation formula (such as formula (2) above). By comparing historical data, disks with redundancy lower than the first preset threshold (which can be the average redundancy or set according to actual needs) are identified from multiple specified disks and marked as high-risk disks. Disks with redundancy greater than the first preset threshold and less than the second preset threshold (which can be set according to actual needs) are identified from multiple specified disks and marked as medium-risk disks. Disks with redundancy greater than the third preset threshold are identified from multiple specified disks and marked as low-risk disks (which can be set according to actual needs). For subtasks on high-risk disks, the highest priority and the most abundant computing resources are allocated; tasks on medium-risk disks are allocated second; tasks on low-risk disks have the lowest priority and are allocated only basic computing resources. During data reconstruction on the disk to be reconstructed, each subtask is automatically assigned a priority based on the redundancy of its associated disk, with higher-priority tasks receiving computing resources first. Based on the disk redundancy and task priority of each specified disk at the current moment, the resource scheduling module dynamically adjusts the allocation of computing resources to each subtask. High-priority tasks receive priority access to high-performance computing nodes to accelerate data recovery; low-priority tasks are allocated to nodes with moderate performance to avoid resource waste.

[0076] This embodiment determines the risk level of each specified disk based on its redundancy, and formulates specific resource allocation strategies and task priorities according to different risk levels. Subtasks corresponding to high-risk disks receive the highest priority and the most abundant computing resources, while subtasks corresponding to low-risk disks receive lower priority and more conservative resource allocation. This ensures that resources are prioritized for allocation to the disks that need them most, avoiding resource waste in high-risk situations and improving the efficiency and success rate of data recovery. Secondly, by dynamically adjusting the priority of each subtask, it can intelligently respond to real-time changes in disk status, thereby optimizing resource usage while ensuring data integrity, reducing excessive occupation of satellite-to-ground link bandwidth, and ensuring the stability and continuity of services.

[0077] In one exemplary example, the above data reconstruction method further includes: reserving a specified storage capacity in a specified disk; the specified storage capacity refers to a preset percentage of the remaining storage capacity of the specified disk; the specified storage capacity is used to cache redundant data generated by the specified disk during the data reconstruction process.

[0078] The specified storage capacity refers to the portion of the remaining storage space on a specified disk that is reserved in advance for storing redundant data. The size of the specified storage capacity is determined based on a preset percentage (e.g., 10% of the total remaining storage capacity). Its purpose is to ensure that even if new faults or anomalies are encountered during the data reconstruction process, there is enough space to store additional redundant data, thereby improving the integrity of data recovery and the robustness of the system.

[0079] Redundant data specifically refers to one or more copies of data or encoded information that are specifically generated and saved during the data reconstruction process to address potential secondary failures or future risks of a specified disk.

[0080] In this embodiment, a specified storage capacity is reserved as a redundant data cache. Even if a new disk failure occurs during or immediately after the data reconstruction process, the pre-stored redundant data can be used to quickly recover the data, avoiding secondary data loss and thus enhancing the overall stability and fault tolerance of the distributed storage system.

[0081] In one exemplary embodiment, before the disk to be reconstructed undergoes data reconstruction, the erasure coding type of the disk to be reconstructed is used to indicate the allocation of a first redundancy level for data in the first data layer and a second redundancy level for data in the second data layer; the first data layer is used to store data with an access frequency greater than a first frequency value; the second data layer is used to store data with an access frequency less than a second frequency value; and the first redundancy level is greater than the second redundancy level.

[0082] Before the erasure coding type of the disk to be reconstructed is switched to the target erasure coding type corresponding to the target redundancy level, the erasure coding type of the disk to be reconstructed is to divide the disk data into a first data layer and a second data layer according to the data access frequency, and different layers correspond to different redundancy levels, that is, different layers correspond to different data recovery rates and storage resource usage ratios. The first data layer is a storage layer specifically designed for storing hot data with an access frequency higher than a first frequency threshold. Hot data typically refers to data that is frequently accessed or updated; a higher redundancy level is used for hot data to ensure its rapid recovery and high availability in the event of storage anomalies. The second data layer is a storage layer used for storing cold data with an access frequency lower than a second frequency threshold. Cold data is data that is accessed less frequently than hot data; a lower redundancy level can reduce storage space usage and resource consumption while ensuring data security. In this embodiment, the first redundancy level of the first data layer is greater than the second redundancy level of the second data layer, meaning that the redundancy of hot data is higher and can better cope with storage failures, while cold data optimizes storage space usage while meeting basic security requirements.

[0083] When a disk to be reconstructed is undergoing data reconstruction, the erasure coding type of the disk to be reconstructed needs to be switched to the target erasure coding type corresponding to the target redundancy level.

[0084] In some embodiments, not only the disk to be reconstructed can adopt the erasure coding type of this embodiment, but any disk in the distributed storage system can adopt the scheme of setting different erasure coding types according to the access frequency in this embodiment.

[0085] In this embodiment, before data reconstruction, each disk in the distributed storage system divides the disk data into a first data layer and a second data layer according to different data access frequencies. The first redundancy level of the first data layer is greater than the second redundancy level of the second data layer. This ensures the rapid recovery of high-frequency access data (first data layer) while reducing network bandwidth consumption. It enables rapid location and recovery during data reconstruction without the need for full data transmission, significantly improving the speed of data recovery and reducing bandwidth consumption. For the second data layer with lower access frequency, a lower redundancy level is adopted, which saves storage space and avoids unnecessary computation and bandwidth overhead, achieving fine-grained resource management.

[0086] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0087] Embodiments of this application also provide a data reconstruction apparatus, such as... Figure 4 As shown, it includes:

[0088] The redundancy calculation module 402 is used to monitor the status data of the disk to be reconstructed in the case of a disk to be reconstructed in the distributed storage system, and to determine the redundancy of the disk to be reconstructed based on the status data of the disk to be reconstructed.

[0089] The redundancy level determination module 404 is used to determine the target redundancy level corresponding to the redundancy of the disk to be reconstructed from a plurality of preset redundancy levels; different redundancy levels in the plurality of redundancy levels correspond to different redundancy ranges, and different redundancy levels correspond to different erasure coding types; the erasure coding types corresponding to different redundancy levels represent different combinations of data recovery rate and storage resource utilization ratio.

[0090] The reconstruction module 406 is used to switch the erasure coding type of the disk to be reconstructed to the target erasure coding type corresponding to the target redundancy level, transfer the faulty data blocks in the disk to be reconstructed to the designated disk according to the target erasure coding type, and perform data reconstruction on the designated disk; the designated disk is a normal disk in the distributed storage system.

[0091] In an exemplary embodiment, the status data of the disk to be reconstructed is used to indicate the following information about the disk to be reconstructed: remaining storage capacity and historical failure rate; the redundancy calculation module 402 is further used to determine the proportion of used storage capacity of the disk to be reconstructed based on the total storage capacity and remaining storage capacity of the disk to be reconstructed; and to obtain the redundancy of the disk to be reconstructed by weighted summing of the proportion of used storage capacity and the historical failure rate based on a first weighting coefficient and a second weighting coefficient of the historical failure rate; the first weighting coefficient and the second weighting coefficient are dynamically adjusted based on the recovery efficiency index of the disk to be reconstructed.

[0092] In an exemplary embodiment, the redundancy calculation module 402 is further configured to monitor the recovery efficiency index of the disk to be reconstructed within a specified time period, and obtain the index parameters corresponding to the specified time period; input the index parameters into a pre-trained reinforcement learning model to obtain a first weight coefficient and a second weight coefficient; and the reinforcement learning model is configured to adjust the first weight coefficient and the second weight coefficient according to the input index parameters.

[0093] In an exemplary embodiment, the redundancy calculation module 402 is further configured to perform multiple rounds of training operations until the reinforcement learning model meets the training termination condition, thereby obtaining a trained reinforcement learning model. During the execution of the current round of training operations, a current state is selected from a pre-constructed state space, and a current action is selected from a pre-constructed action space. The state space includes multiple states, and the state parameter of one of the states is a sample index parameter. The action space includes multiple actions. The action parameter of one of the actions includes an action instruction for adjusting the redundancy level. The current action is executed through the current reinforcement learning model, and the immediate reward value corresponding to the current action is determined according to the reward function. Based on the immediate reward value corresponding to the current action and the value function, the action value between the current state and the current action is determined, and the current reinforcement learning model is adjusted based on the action value between the current state and the current action. The action value includes a first expected long-term return and a second expected long-term return. The value function characterizes the mapping relationship between the action value and the first weight coefficient, the second weight coefficient, the first expected long-term return, and the second expected long-term return.

[0094] In an exemplary embodiment, the reconstruction module 406 is further configured to divide the faulty data block in the disk to be reconstructed into multiple sub-fault blocks according to the target erasure coding type; construct multiple subtasks; each subtask corresponds one-to-one with a multiple sub-fault block; a subtask in the multiple subtasks is used to transfer the corresponding sub-fault block to a designated disk for data reconstruction; adjust the computing resources of the multiple subtasks according to the redundancy of the disk to be reconstructed, and use the adjusted multiple subtasks to transfer the multiple sub-fault blocks to the designated disk for data reconstruction.

[0095] In an exemplary embodiment, the reconstruction module 406 is further configured to reserve a specified storage capacity in a specified disk; the specified storage capacity refers to a preset percentage of the remaining storage capacity of the specified disk; the specified storage capacity is used to cache redundant data generated by the specified disk during the data reconstruction process.

[0096] In one exemplary embodiment, the erasure coding type of the disk to be reconstructed is used to indicate the allocation of a first redundancy level to data in the first data layer and a second redundancy level to data in the second data layer; the first data layer is used to store data with an access frequency greater than a first frequency value; the second data layer is used to store data with an access frequency less than a second frequency value; and the first redundancy level is greater than the second redundancy level.

[0097] For a description of the features in the embodiment corresponding to the data reconstruction device, please refer to the relevant description of the embodiment corresponding to the data reconstruction method, which will not be repeated here.

[0098] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data reconstruction method embodiments.

[0099] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data reconstruction method embodiments at runtime.

[0100] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0101] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data reconstruction method embodiments.

[0102] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data reconstruction method embodiments.

[0103] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The data reconstruction method, apparatus, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data reconstruction method, characterized in that, include: In a distributed storage system where a disk needs to be reconstructed, monitor the status data of the disk to be reconstructed. The status data of the disk to be reconstructed is used to indicate the following information about the disk to be reconstructed: remaining storage capacity and historical failure rate; Monitor the recovery efficiency index of the disk to be reconstructed within a specified time period to obtain the index parameters corresponding to the specified time period; the recovery efficiency index refers to multiple parameters that comprehensively consider the quality, speed and resource consumption of data recovery under the current state of the disk to be reconstructed and the actions performed. The index parameters are input into a pre-trained reinforcement learning model to obtain a first weight coefficient and a second weight coefficient; the reinforcement learning model is used to adjust the first weight coefficient and the second weight coefficient according to the input index parameters. Based on the total storage capacity of the disk to be reconstructed and the remaining storage capacity, determine the percentage of used storage capacity of the disk to be reconstructed; The redundancy of the disk to be reconstructed is obtained by weighting and summing the percentage of used storage capacity and the historical failure rate based on the first weighting coefficient of the percentage of used storage capacity and the second weighting coefficient of the historical failure rate. The first weighting coefficient and the second weighting coefficient are dynamically adjusted based on the recovery efficiency index of the disk to be reconstructed; The target redundancy level corresponding to the redundancy of the disk to be reconstructed is determined from a plurality of preset redundancy levels; different redundancy levels in the plurality of redundancy levels correspond to different redundancy ranges, and different redundancy levels correspond to different erasure coding types; the erasure coding types corresponding to different redundancy levels represent different combinations of data recovery rate and storage resource utilization ratio; The erasure coding type of the disk to be reconstructed is switched to the target erasure coding type corresponding to the target redundancy level. According to the target erasure coding type, the faulty data blocks in the disk to be reconstructed are transferred to the designated disk, and data reconstruction is performed on the designated disk. The designated disk is a normal disk in the distributed storage system.

2. The method according to claim 1, characterized in that, The method further includes: Perform multiple rounds of training until the reinforcement learning model meets the training termination condition, and obtain the trained reinforcement learning model. During the execution of the current round of training operations, the current state is selected from the pre-constructed state space, and the current action is selected from the pre-constructed action space. The state space includes multiple states, and the state parameter of one of the multiple states is a sample index parameter. The action space includes multiple actions, and the action parameter of one of the multiple actions includes an action instruction for adjusting the redundancy level. The current action is executed through the current reinforcement learning model, and the immediate reward value corresponding to the current action is determined according to the reward function; Based on the immediate reward value and value function corresponding to the current action, the action value between the current state and the current action is determined, and the current reinforcement learning model is adjusted according to the action value between the current state and the current action; the action value includes a first expected long-term return and a second expected long-term return; the value function represents the mapping relationship between the action value and the first weight coefficient, the second weight coefficient, the first expected long-term return, and the second expected long-term return.

3. The method according to claim 1, characterized in that, The step of transferring the faulty data blocks in the disk to be reconstructed to the designated disk according to the target erasure coding type includes: According to the target erasure coding type, the faulty data block in the disk to be reconstructed is divided into multiple sub-faulty blocks; Multiple subtasks are constructed; each subtask corresponds one-to-one with a sub-fault block; a subtask within the multiple subtasks is used to transfer the corresponding sub-fault block to the designated disk for data reconstruction. Based on the redundancy of the disk to be reconstructed, the computing resources of the multiple subtasks are adjusted, and the adjusted multiple subtasks are used to transfer the multiple sub-fault blocks to the designated disk for data reconstruction.

4. The method according to claim 1, characterized in that, The method further includes: A specified storage capacity is reserved in the specified disk; the specified storage capacity refers to a preset percentage of the remaining storage capacity of the specified disk; the specified storage capacity is used to cache redundant data generated by the specified disk during the data reconstruction process.

5. The method according to any one of claims 1 to 4, characterized in that, The erasure coding type of the disk to be reconstructed is used to indicate the allocation of a first redundancy level to the data in the first data layer and a second redundancy level to the data in the second data layer; the first data layer is used to store data with an access frequency greater than a first frequency value; the second data layer is used to store data with an access frequency less than a second frequency value; the first redundancy level is greater than the second redundancy level.

6. A data reconstruction apparatus, characterized in that, include: The redundancy calculation module is used to monitor the status data of the disks to be reconstructed in a distributed storage system. The status data of the disk to be reconstructed is used to indicate the following information about the disk to be reconstructed: remaining storage capacity and historical failure rate; monitoring the recovery efficiency index of the disk to be reconstructed within a specified time period to obtain index parameters corresponding to the specified time period; the recovery efficiency index refers to multiple parameters that comprehensively consider the quality, speed, and resource consumption of data recovery under the current state of the disk to be reconstructed and the actions performed; the index parameters are input into a pre-trained reinforcement learning model to obtain a first weight coefficient and a second weight coefficient; the reinforcement learning model is used to adjust the first weight coefficient and the second weight coefficient according to the input index parameters; and the percentage of used storage capacity of the disk to be reconstructed is determined based on the total storage capacity of the disk to be reconstructed and the remaining storage capacity. The redundancy of the disk to be reconstructed is obtained by weighting and summing the percentage of used storage capacity and the historical failure rate based on a first weighting coefficient and a second weighting coefficient. The first weighting coefficient and the second weighting coefficient are dynamically adjusted based on the recovery efficiency index of the disk to be reconstructed. The redundancy level determination module is used to determine the target redundancy level corresponding to the redundancy of the disk to be reconstructed from a plurality of preset redundancy levels; different redundancy levels in the plurality of redundancy levels correspond to different redundancy ranges, and different redundancy levels correspond to different erasure coding types; the erasure coding types corresponding to different redundancy levels represent different combinations of data recovery rate and storage resource occupancy ratio. The reconstruction module is used to switch the erasure coding type of the disk to be reconstructed to the target erasure coding type corresponding to the target redundancy level, transfer the faulty data blocks in the disk to be reconstructed to a designated disk according to the target erasure coding type, and perform data reconstruction on the designated disk; the designated disk is a normal disk in the distributed storage system.

7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data reconstruction method as described in any one of claims 1 to 5 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data reconstruction method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Data reconstruction method and device, storage medium and program product

    CN119336536A

  • Medical image management method and system based on photo-electromagnetic hybrid hierarchical storage

    CN120066420A