An erasure coding strategy conversion method, apparatus, device, medium, and product

Abstract

This invention provides an erasure coding policy conversion method, apparatus, device, medium, and product, applied in the field of storage technology. The method includes: determining the access frequency of a data block based on read / write requests; obtaining load information and network bandwidth information of the storage node where the data block resides; determining a target erasure coding policy corresponding to the data block based on the access frequency, load information, and network bandwidth information; if the target erasure coding policy is inconsistent with the current erasure coding policy of the data block, identifying the changed sub-blocks from the data block and encoding the sub-blocks using the target erasure coding policy to obtain target policy encoded data. This enables dynamic conversion of erasure coding policies and reduces resource overhead during the conversion process, thereby improving storage performance.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to an erasure coding strategy conversion method, apparatus, device, medium, and product. Background Technology

[0002] In distributed storage systems, erasure coding technology is widely used due to its high storage efficiency. However, currently, a static strategy is usually adopted, meaning the system uses a fixed erasure coding strategy. This strategy cannot be flexibly switched, cannot adapt to dynamic changes in data access, resulting in insufficient redundancy of hot data, high read latency, and excessive redundancy of cold data, wasting storage space. Furthermore, when switching erasure coding strategies, all data needs to be changed, which incurs significant resource overhead.

[0003] Therefore, how to achieve dynamic conversion of erasure coding strategies and reduce the resource overhead of the conversion process to improve storage performance is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide an erasure coding policy conversion method, apparatus, device, medium, and product that can realize dynamic conversion of erasure coding policies and reduce resource overhead in the conversion process, thereby improving storage performance.

[0005] In a first aspect, the present invention provides an erasure coding policy conversion method, comprising: determining the access popularity of a data block based on read and write requests; obtaining load information and network bandwidth information of the storage node where the data block is located; determining a target erasure coding policy corresponding to the data block based on the access popularity, the load information, and the network bandwidth information; if the target erasure coding policy is inconsistent with the current erasure coding policy of the data block, determining the changed data sub-block from the data block, and encoding the data sub-block using the target erasure coding policy to obtain target policy encoded data.

[0006] Secondly, the present invention provides an erasure coding policy conversion device, comprising: an access popularity determination module, configured to determine the access popularity of a data block based on read / write requests of the data block; a node information acquisition module, configured to acquire load information and network bandwidth information of the storage node where the data block is located; an erasure coding policy determination module, configured to determine a target erasure coding policy corresponding to the data block based on the access popularity, the load information, and the network bandwidth information; and an erasure coding policy conversion module, configured to, if the target erasure coding policy is inconsistent with the current erasure coding policy of the data block, determine the changed data sub-block from the data block, and encode the data sub-block using the target erasure coding policy to obtain target policy encoded data.

[0007] Thirdly, the present invention provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of the aforementioned erasure coding strategy conversion method.

[0008] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned erasure coding strategy conversion method.

[0009] Fifthly, the present invention provides a computer program product, including a computer program / instruction that, when executed by a processor, implements the steps of the aforementioned erasure coding strategy conversion method.

[0010] As can be seen from the above scheme, the present invention provides an erasure coding policy conversion method, including: determining the access popularity of the data block based on the read and write requests of the data block; obtaining the load information and network bandwidth information of the storage node where the data block is located; determining the target erasure coding policy corresponding to the data block based on the access popularity, the load information, and the network bandwidth information; if the target erasure coding policy is inconsistent with the current erasure coding policy of the data block, then determining the changed data sub-block from the data block, and encoding the data sub-block using the target erasure coding policy to obtain target policy encoded data.

[0011] As can be seen, the beneficial effects of the present invention are as follows: the access popularity of a data block is determined by the read and write requests of the data block; when determining the target erasure coding strategy corresponding to the data block, the access popularity of the data block, the load information of the storage node, and the network bandwidth information are comprehensively considered to dynamically and intelligently determine the erasure coding strategy adapted to the data block; and, in the case of inconsistency with the current erasure coding strategy, the changed data sub-blocks are encoded and converted, that is, the incremental data is encoded and converted. In this way, the dynamic conversion of erasure coding strategy can be realized and the resource overhead of the conversion process can be reduced, thereby improving storage performance.

[0012] Correspondingly, the erasure coding strategy conversion device, equipment, medium and product provided by the present invention also have the above-mentioned technical effects. Attached Figure Description

[0013] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1A flowchart of an erasure coding strategy conversion method provided in an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram of an erasure coding strategy conversion provided in an embodiment of the present invention;

[0016] Figure 3 A specific flowchart of erasure coding strategy conversion is provided for an embodiment of the present invention;

[0017] Figure 4 This is a schematic diagram of an erasure coding strategy conversion device provided in an embodiment of the present invention;

[0018] Figure 5 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0020] The terms "comprising" and "having," and any variations thereof, in the specification and accompanying drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the steps or units listed, but may include steps or units not listed.

[0021] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0022] First, the terms involved in this invention are explained: ICFS: Distributed storage system, adopting an object storage architecture; RS code: Reed-Solomon code, an erasure coding technique; LRC code: Local Repair Erasure Code, optimizing recovery efficiency; Access heatmap: A three-dimensional data model (time / space / access frequency) reflecting the distribution of data block access frequency. Hot and cold data: Hot data: High-frequency access data (e.g., QPS (Query Per Second) > 100); Cold data: Low-frequency access data (e.g., QPS < 1).

[0023] In distributed storage systems, erasure coding technology is widely used due to its high storage efficiency (saving 50% of space compared to three replicas). However, existing technologies have significant drawbacks: Static policy issues: The system uses a fixed EC (Erasure Coding) policy (such as ICFS's default RS(10,4)), which cannot adapt to dynamic changes in data access; Resource mismatch: Insufficient redundancy of hot data leads to high read latency (measured >80ms), while excessive redundancy of cold data wastes storage space. Since the system uses erasure or replica storage for all data, low redundancy policies are configured for high-access hot data, resulting in performance bottlenecks, while high redundancy policies are configured for low-access cold data, causing storage waste; High conversion overhead: Policy changes require full data migration, consuming up to 1.5 times the network bandwidth of the original data; Lack of intelligent decision-making: Microsoft's tiered solution is based on simple threshold switching and does not consider network topology and node load. The most mature technical solution currently available is hot-cold tiering, but its limitations include: supporting only binary switching between replicas and ECs severely restricts the flexibility of storage strategies; the long conversion decision cycle of up to 6 hours significantly lags behind the dynamic changes in business data; and the unresolved migration bandwidth bottleneck seriously hinders the data migration process. Therefore, these limitations not only reduce the flexibility and response speed of the storage system but also waste resources, impact business performance, and to some extent restrict its application effectiveness in highly dynamic and demanding business scenarios. To address this, this invention provides an erasure coding strategy conversion scheme that enables dynamic conversion of erasure coding strategies and reduces resource overhead during the conversion process, thereby improving storage performance.

[0024] Next, we will describe in detail an erasure coding strategy conversion method provided by an embodiment of the present invention. Figure 1 The flowchart of an erasure coding policy conversion method provided in this embodiment of the invention includes the following steps.

[0025] Step S11: Determine the access frequency of the data block based on the read and write requests of the data block.

[0026] The data block can be a data block divided according to preset rules, such as a preset size. In this embodiment of the invention, the access frequency can be calculated according to the granularity of the data block to perform erasure coding policy conversion. Read and write requests include read requests and write requests. The access frequency can reflect the access frequency of the data block. In this invention, the access frequency is real-time, that is, the access frequency is updated periodically.

[0027] In an optional implementation, data access logs can be read from a message queue; wherein the data access logs are collected periodically at a preset frequency and placed in the message queue; read and write requests for data blocks are parsed from the data access logs, and the access frequency of the data blocks is determined.

[0028] The message queue can be a Kafka queue. A preset log collection component can periodically collect data access logs at a preset frequency and place them in the message queue. In this embodiment, a preset hot update component can subscribe to the message queue, consume the data access logs in the message queue, parse the read and write requests of data blocks, and count the number of read requests and write requests of data blocks according to a preset time period. The time period is a certain time interval.

[0029] In this embodiment of the invention, the access popularity of a data block can be determined based on the number of read requests, the number of write requests, and the weighting coefficients corresponding to the read and write requests in the current time period.

[0030] Read and write requests can be assigned to different weighting systems or have the same weighting coefficient. The access frequency of a data block can be determined through weighted calculation. For example, if read operations are more important to the business, the weighting coefficient for read requests can be set to a higher value.

[0031] Furthermore, determining the access popularity of the data block based on the number of read requests, the number of write requests, and the weight coefficients corresponding to the read and write requests in the current time period can include: determining the access popularity of the data block based on the number of read requests, the number of write requests, and the weight coefficients corresponding to the read and write requests in the current time period and the most recent preset number of historical time periods.

[0032] In other words, when calculating access popularity in this embodiment of the invention, in addition to the current time period, access information from a preset number of historical time periods can also be considered.

[0033] In addition, embodiments of the present invention can also control the contribution of historical time periods to access popularity by setting a decay factor and the distance between historical time periods and the current time period, wherein the farther away from the current time period, the lower the contribution.

[0034] In an optional implementation, the present invention may use the following access popularity calculation formula to calculate access popularity.

[0035] .

[0036] in, λ represents the access popularity, and λ represents the preset decay factor. The preset decay factor controls the decay rate of historical access contributions. The larger the value, the faster the decay and the higher the weight of recent accesses; the smaller the value, the slower the decay and the more persistent the contribution of historical accesses. i represents the time period identifier, which can characterize the distance between the historical time period and the current time period. i=0 is the current time period, i=1 is the previous historical time period, i=2 is the previous historical time period, and so on. The farther away from the current time period, the larger i is. The weight coefficient corresponding to the read request. The weighting coefficient corresponding to the write request. The number of read requests within the corresponding time period. This represents the number of write requests in the corresponding time period, where n represents the total number of time periods involved in the calculation.

[0037] Step S12: Obtain the load information and network bandwidth information of the storage node where the data block is located.

[0038] This invention provides an embodiment of the invention that obtains one or more of the following: the current CPU (Central Processing Unit), memory, and disk I / O (Input / Output) utilization rates of a storage node, thereby obtaining the storage node's load information. It can also obtain the current network bandwidth utilization rate or remaining bandwidth, thereby obtaining network bandwidth information.

[0039] Step S13: Determine the target erasure coding strategy corresponding to the data block based on the access popularity, the load information, and the network bandwidth information.

[0040] In one optional implementation, a decision score can be obtained by weighting the access popularity, the load information, and the network bandwidth information; and the target erasure coding strategy corresponding to the data block can be determined based on the decision score.

[0041] In this embodiment, if there are multiple different erasure coding strategies, a scoring range can be set for each erasure coding strategy, and the erasure coding strategy corresponding to the range to which the decision score belongs can be used as the target erasure coding strategy.

[0042] Furthermore, determining the target erasure coding strategy corresponding to the data block based on the decision score may include: comparing the decision score with a preset threshold; if the decision score is greater than the preset threshold, the data block is determined to be hot data and the target erasure coding strategy is a locally repairable erasure coding strategy; otherwise, it is determined to be cold data and the target erasure coding strategy is a Reed-Solomon code strategy.

[0043] In other words, data with higher scores is considered hot data, while data with lower scores is considered cold data, and different erasure coding strategies are used accordingly.

[0044] In another optional implementation, the target erasure coding strategy corresponding to the data block is determined based on the access popularity, the load information, and the network bandwidth information, including: generating a state vector based on the access popularity, the load information, and the network bandwidth information; and using a target reinforcement learning model to determine the action corresponding to the state vector to obtain the target erasure coding strategy corresponding to the data block.

[0045] That is, in this invention, the state can be access popularity, load information, and network bandwidth information, and the action can be erasure coding strategy. In an optional implementation, access popularity can be divided into corresponding levels, with different levels corresponding to different values. For example, the levels can include high temperature, low temperature, and medium temperature to construct the state vector.

[0046] Furthermore, during the training process, the target reinforcement learning model employs a preset reward function, which can be determined based on performance rewards, cost penalties, and stability penalties. The cost penalties may include storage cost penalties and / or migration cost penalties.

[0047] Among these, performance rewards are negatively correlated with data access latency; the lower the latency, the higher the reward. Storage cost penalties are positively correlated with storage space usage; the more storage space used, the higher the penalty, i.e., a negative reward. Migration cost penalties: When policies change, data migration consumes network bandwidth and computing resources; the more resources consumed, the higher the penalty. Stability penalties: If the load exceeds a preset threshold after a policy change, a corresponding penalty is imposed; that is, if the storage system load is too high (e.g., high CPU utilization, high IO utilization), a penalty is imposed. IO utilization can represent the percentage of time the storage device is in a busy state during the sampling period.

[0048] Step S14: If the target erasure coding policy is inconsistent with the current erasure coding policy of the data block, then the changed data sub-block is determined from the data block, and the data sub-block is encoded using the target erasure coding policy to obtain target policy encoded data.

[0049] In an optional implementation, determining the changed data sub-blocks from the data block may include: dividing the data block into data sub-blocks of a preset size; comparing the data sub-blocks with snapshot data corresponding to the data sub-blocks at a preset historical time point; if they are inconsistent, then determining that the data sub-block is a changed data sub-block.

[0050] In this embodiment of the invention, snapshots of data can be generated at preset time points, and when comparing, they can be compared with the data of the most recently generated snapshot.

[0051] Further, comparing the data sub-block with the snapshot data corresponding to the data sub-block at a preset historical time point includes: calculating the check value of the data sub-block to obtain a first check value; calculating the check value of the snapshot data corresponding to the data sub-block at the preset historical time point to obtain a second check value; comparing the first check value and the second check value, and if the first check value and the second check value are consistent, then the data sub-block is consistent with the snapshot data corresponding to the data sub-block at the preset historical time point; otherwise, they are inconsistent.

[0052] In this embodiment of the invention, the method used to calculate the check value is not limited, such as calculating the CRC (Cyclic Redundancy Check) 64 checksum.

[0053] Furthermore, embodiments of the present invention can employ a parallel strategy to acquire data blocks and then determine the changed data sub-blocks. One parallel strategy involves directly copying data blocks from storage nodes, while another involves calculating data blocks using parity data from multiple storage nodes. Once a data block is obtained using one strategy, the other strategy is stopped. This ensures rapid acquisition of data blocks, and even if a data block is corrupted, it can be restored using parity data. Moreover, multiple nodes can be used to calculate new parity blocks in parallel, accelerating the reconstruction process.

[0054] Furthermore, after encoding the data sub-block using the target erasure coding strategy to obtain target strategy encoded data, this embodiment of the invention further includes: determining the migration target position of the target strategy encoded data; and migrating the target strategy encoded data to the migration target position.

[0055] The process of determining the migration target location for the target policy-encoded data includes: if the target erasure coding policy is a locally repairable erasure coding policy, then the migration target location for the target policy-encoded data is determined to be a storage location within a local node, wherein the local node is the storage node where the data block is located or a storage node in the same rack; if the target erasure coding policy is a Reed-Solomon coding policy, then the migration target location for the target policy-encoded data is determined to be a storage location within a remote node, wherein the remote node is a storage node in a different rack than the storage node where the data block is located.

[0056] In this way, hot data can be stored on local nodes, which have a high access frequency and low communication cost, while cold data, which has a relatively low access frequency, can be stored on remote nodes.

[0057] Furthermore, determining the migration target location of the target policy encoded data as the storage location within a local node includes: selecting a target local node from all local nodes based on a preset performance filtering standard, and determining the migration target location of the target policy encoded data as the storage location within the target local node. Determining the migration target location of the target policy encoded data as the storage location within a remote node may include: selecting a target remote node from all remote nodes based on a preset performance filtering standard, and determining the migration target location of the target policy encoded data as the storage location within the target remote node.

[0058] The preset performance screening criteria may include load and / or bandwidth. For example, the CPU remaining space is greater than a preset space threshold, or the bandwidth utilization is lower than a preset utilization threshold. In this embodiment, the storage system may include multiple storage nodes, and each data block has tagging information that marks its corresponding local node and remote node. The tagging information can be determined based on the topology of the storage system.

[0059] Furthermore, after encoding the data sub-blocks using the target erasure coding strategy to obtain target strategy encoded data, the embodiments of the present invention may further include: detecting performance indicators; adjusting the weight coefficient and / or the preset decay factor if the performance indicators do not meet the preset standard; and adjusting the number of time periods involved in the calculation.

[0060] In this embodiment of the invention, the performance metrics may include one or more of read latency, the number of read / write operations per second, and storage utilization, including but not limited to read latency, the number of read / write operations per second, and storage utilization. That is, this embodiment of the invention can monitor the performance metrics of the storage system, and can obtain performance metrics after data migration to optimize the heat calculation formula.

[0061] Furthermore, embodiments of the present invention can update metadata information after data migration is complete. Updating metadata is an atomic operation, ensuring data consistency.

[0062] As can be seen, the embodiments of the present invention determine the access popularity of data blocks through read and write requests. When determining the target erasure coding strategy corresponding to a data block, the access popularity of the data block, the load information of the storage node, and the network bandwidth information are comprehensively considered to dynamically and intelligently determine the erasure coding strategy that suits the data block. Furthermore, in the case of inconsistency with the current erasure coding strategy, the changed data sub-blocks are encoded and converted, that is, the incremental data is encoded and converted. In this way, dynamic conversion of erasure coding strategy can be achieved, and the resource overhead of the conversion process can be reduced, thereby improving storage performance.

[0063] Further, see Figure 2As shown, this invention provides a schematic diagram of erasure coding strategy conversion. This invention drives dynamic EC strategy conversion through access heatmaps, achieving real-time optimization of storage efficiency and performance. The system generates an access heatmap by monitoring the access frequency distribution of data blocks in real time. Combined with the load status of storage nodes and network bandwidth changes, it dynamically selects the optimal erasure coding strategy (RS code, LRC code, etc.) to achieve adaptive encoding conversion of hot and cold data. Hot data adopts a high-redundancy, low-recovery-overhead strategy, while cold data adopts a high-storage-efficiency strategy. Employing a two-stage data migration and parallel reconstruction mechanism, the conversion process reduces bandwidth consumption by more than 37%, improves system IOPS (Input / Output Operations Per Second) performance by 20-50%, and increases storage space utilization by 35%. It may include the following components.

[0064] 1. Client I / O Requests: The input source for the entire system is the client's read and write operation requests to the storage system. All client read and write requests enter the system, and this invention generates an access heatmap based on these requests. Simultaneously, these requests are also actually executed (reading or writing data). This invention can record access information for each data block, including access time, access type (read / write), access frequency, etc.

[0065] 2. Heatmap Generation Module: Based on client IO requests, this module generates and updates access heatmaps for data blocks in real time. It receives client IO requests, calculates the access frequency of each data block, and constructs a heatmap based on three dimensions: time, space (data block location), and access frequency. In the heatmap, each data block is labeled with a different temperature (e.g., high, medium, low) to further characterize its access frequency. In this invention, "real-time" refers to the storage updating its access heatmap simultaneously with the client sending IO requests (read or write requests). In this embodiment, the heatmap can be viewed as a three-dimensional coordinate system, where each point represents a data block, and each data block has time, space, and access frequency labels. The x, y, and z axes in the heatmap correspond to the time, space, and access frequency labels of the data blocks.

[0066] 3. Strategy Decision Engine: Based on heatmaps, storage node load, and network bandwidth, this module dynamically determines the optimal erasure coding strategy. It receives heatmap generation information from the heatmap generation module and monitors storage node load (e.g., CPU, memory, disk I / O utilization) and network bandwidth. Based on this information, it selects the most suitable erasure coding strategy (e.g., RS code, LRC code) using built-in decision algorithms (e.g., reinforcement learning models, weighted scoring). For hot data, it selects a strategy with high redundancy and low recovery overhead (e.g., LRC); for cold data, it selects a strategy with high storage efficiency (e.g., RS code).

[0067] 4. Recoding Executor: This module executes the policy decision engine's decisions, converting data blocks from the old erasure coding policy to the new policy. Based on the decision engine's instructions, it performs encoding conversion on specified data blocks. To reduce bandwidth consumption and performance impact during the conversion process, a two-stage migration (transmitting only the changed portion of the data block (incremental data), rather than the entire data block) and a parallel reconstruction mechanism (utilizing multiple nodes to compute the new parity block in parallel, accelerating reconstruction) are employed. In other words, based on the policy decision engine's decisions, the recoding executor converts the changed data in the data block into new RS or LRC codes, and then performs the migration. Changing the erasure coding policy of a data block requires encoding conversion, simultaneously triggering changes to the data block and data parity in the storage system, and also changing the data's storage location on disk. For example, if the erasure coding strategy changes from RS(4,2) to RS(10,4), migration refers to the change in the disk storage location of data blocks caused by the strategy change. This requires migrating data from one sector to another (not necessarily all data needs to be migrated; only the changed data is migrated, depending on sector usage). The check block is distributed across different nodes, and the complete data block can be obtained using the EC algorithm based on the existing data block and the check block. The parallel reconstruction mechanism utilizes multiple nodes, reducing the time to obtain the complete data block and improving efficiency. Furthermore, in this embodiment, copying the complete data block and calculating the complete data block using the check block are performed in parallel; once the complete data block is obtained, the other method stops. After the erasure coding change, the intended purpose of this invention is achieved.

[0068] 5. Distributed Metadatabase: Stores the system's metadata, including data block locations, erasure coding policies, version information, etc. After completing data transformation, the recoding executor updates the corresponding data block information in the metadata database (such as the new erasure coding policy, data block location, version number, etc.). Metadatabase updates are atomic operations, ensuring data consistency. Simultaneously, the metadata database also provides query services for other modules (such as the policy decision engine). Before a data block is written to disk, an address mapping is stored in the metadata pool. This is because metadata is typically deployed as a replica pool on SSD (Solid State Disk) or NVMe (Non-Volatile Memory Express) media, offering significantly higher read and write performance than erasure pools on ordinary HDD (Hard Disk Drive) media. When a user accesses a data block again, the address mapping is retrieved from the metadata pool, allowing for more efficient data block location lookup and faster data access. Consistency is ensured through atomic operations, where an atomic operation (e.g., updating all information of a data block) is indivisible. It either succeeds completely (all changes take effect) or fails completely (all changes take effect), without remaining in an intermediate state. Atomic operations can be achieved using transaction mechanisms. Consistency means that the metadata recorded in the metadata database must completely match the physical state of the data blocks in the underlying storage system.

[0069] 6. Storage Node Cluster: The physical node cluster that actually stores the data blocks. Storage nodes are responsible for storing data blocks and parity blocks. After the recoding executor completes the transformation, the new data blocks and parity blocks are written to the storage nodes, and the old data blocks are reclaimed. Specifically, the reclaimed data blocks are the migrated, changed sub-blocks within the old data blocks. The target policy-coded data corresponding to these migrated, changed sub-blocks is used as new data blocks, or data with the same erasure coding policy as other data blocks is aggregated into new data blocks. When clients read data, the storage nodes provide data services.

[0070] 7. Feedback Control (Policy Decision Engine -> Heatmap Generation Module): Based on the effects of the policy conversion, the parameters of the heatmap generation module are dynamically adjusted to form a closed-loop control. The policy decision engine monitors the performance metrics after the conversion (such as storage utilization, IOPS, latency, etc.). If the actual effect does not match the expectations (the heatmap prediction may not be accurate enough), it feeds back to the heatmap generation module to adjust the heatmap parameters (such as the decay factor λ, the weight coefficients corresponding to read requests and write requests, etc.). Expectations refer to the policy decision engine's anticipated improvement in system performance metrics after the conversion when making EC policy conversion decisions. This includes performance metrics (such as reduced read latency, increased IOPS, etc.), cost metrics (such as increased storage utilization, etc.), and system load (the network and computational overhead of the conversion operation itself should be within an acceptable range and should not negatively impact system stability). The entire system forms a closed-loop automatic control process. Client I / O requests drive heatmap generation. The decision engine makes decisions based on the heatmap, load, and bandwidth. The recoding executor performs policy transformations and updates metadata and storage nodes. Finally, the heatmap model is continuously optimized through a feedback mechanism, thereby achieving dynamic optimization of storage efficiency and performance.

[0071] For heatmap modeling, in the time dimension, a sliding time window (such as 24 hours) is used to track the access trajectory of data blocks, and the timeliness of historical access is quantified by an exponential decay model. The access popularity is calculated using the following access popularity calculation formula.

[0072] .

[0073] in, This represents the data popularity value at time t, i.e., access popularity. A higher value indicates more popular data, meaning more frequent or recent access. λ is a decay factor, controlling the decay rate of historical access contributions. A larger λ value results in faster decay and a higher weight for recent accesses; a smaller λ value results in slower decay and a more lasting contribution from historical accesses. i: Time interval (i.e., the aforementioned time period) index, representing the time offset from the current time. i=0 represents the current time interval, i=1 represents the previous time interval, up to i=n (the furthest time interval considered). n is the number of historical time intervals considered (window size). For example, if the time interval is 1 hour, n=23 means considering accesses from the past 24 hours. This is a weighting coefficient for read operations, used to adjust the contribution of read accesses to popularity. Typically... If read operations are more important to the business, a higher value can be set. Let be the number of read operations that occur within the i-th time interval. This is the weighting factor for write operations, used to adjust the contribution of write access to popularity. Write operations may be more time-consuming or more critical than read operations, so their weights may differ. This represents the number of write operations that occur within the i-th time interval. For example, λ=0.05 controls the historical access decay rate. (Weighting coefficients for read operations) (Weight coefficient for write operations).

[0074] The heat index calculation is a weighted sum of read and write operations within each time interval, determined by a weighting coefficient and an exponential decay factor. The decay factor ensures that recent accesses (smaller i) contribute more, while older accesses (larger i) contribute less. This aligns with the principle of locality, where recently accessed data is more likely to be accessed again. The weighting coefficient allows the system to differentiate the importance of read and write operations. For example, if the system reads more than writes, and read performance is more critical, a weighting coefficient can be set... The summation from i=0 to n covers a sliding time window, considering only the most recent n+1 intervals to avoid processing infinite historical data. In the policy decision engine, this heat value... The engine uses access patterns to evaluate data blocks. Higher access patterns may indicate hot data, potentially leading to the selection of high-redundancy, low-recovery-overhead erasure coding strategies (such as LRC). Lower access patterns may indicate cold data, potentially leading to the selection of high-storage-efficiency strategies (such as RS coding). Furthermore, the engine monitors actual performance metrics after conversion (such as latency and storage utilization). If the actual access patterns differ from the predictions, feedback is sent to the heatmap generation module to adjust parameters such as λ (attenuation factor). , Or n (window size) to optimize prediction accuracy.

[0075] Furthermore, in this embodiment of the invention, the dynamic decision engine provides a multi-factor constrained decision model, see [link to relevant documentation]. Figure 3 As shown, Figure 3 This invention provides a specific flowchart of erasure coding strategy conversion, which illustrates the decision-making process of dynamic coding strategy in a distributed storage system. The flowchart adaptively selects Local Repair Coding (LRC) or Reed-Solomon Coding (RS) based on data popularity, system load, and network conditions.

[0076] In this invention, the real-time system monitoring input adopts a three-tiered monitoring system, corresponding to three dimensions: data attributes (access popularity), computing resources (node ​​load information), and network environment (node ​​network bandwidth information).

[0077] Strategy decision engine processing: In one optional implementation, weighted calculation: Where CPU / 100 represents CPU utilization, and bandwidth represents the currently used bandwidth. Decision threshold: A score > 0.75 indicates hot data; otherwise, it's considered cold data. Dual-path strategy execution: Hot data path (LRC selector), cold data path (RS selector). Hot data is migrated to local nodes, while cold data is migrated to remote nodes. That is, hot data is stored on the same node or a node within the same rack whenever possible, while cold data is stored on nodes across different racks. Furthermore, thresholds can be used to filter nodes participating in data block migration, minimizing impact on upper-layer services by selecting nodes with CPU utilization below a preset CPU utilization threshold and / or bandwidth utilization below a preset bandwidth utilization threshold.

[0078] Unified Resource Allocation: Based on the Raft (Replicated And Fault Tolerant) protocol, allocation consistency is guaranteed, and a resource reservation mechanism (to prevent deadlock) is implemented. When the decision engine decides to change the erasure coding strategy of the data, data migration and re-encoding need to be performed. This process involves cooperation between multiple nodes, including reading data from the original nodes, re-encoding, writing the newly encoded data blocks to the new location, and updating metadata. During this process, resources (such as network bandwidth, storage space, and computing resources) need to be allocated and managed. Without a coordination mechanism, the following problems may occur: Inconsistent resource allocation: Multiple nodes may compete for the same resource simultaneously, leading to inconsistent views on resource allocation status across different nodes, resulting in data inconsistency or operation failure. Deadlock: Multiple data migration tasks may wait for resources held by each other, leading to deadlock. Therefore, this invention provides a unified resource allocation mechanism, whose functions are as follows: Guaranteeing allocation consistency based on the Raft protocol: Raft is a distributed consensus algorithm used to maintain state consistency among multiple nodes in a distributed system. Here, the Raft protocol is used to implement a resource allocation coordinator (which can be a distributed lock or a resource allocation service) to ensure that all nodes reach a consensus on the allocation status of resources during resource allocation. For example, when a node needs to occupy certain bandwidth and storage space to perform a migration task, the node requests resources from the Raft cluster. The Raft cluster ensures that only one node can be authorized to obtain the resource, thereby avoiding conflicts. Resource reservation mechanism (preventing deadlock): The resource reservation mechanism refers to pre-reserving and locking all necessary resources during resource allocation before executing the task. If all resources cannot be obtained at once, the task waits or is abandoned, thus avoiding blocking due to unavailable resources during task execution and avoiding circular wait conditions. Deadlock is avoided by pre-reserving resources. The role of the unified resource allocation mechanism is to ensure that resource allocation is consistent throughout the distributed system and that deadlocks do not occur when implementing erasure coding policy changes, thereby guaranteeing system reliability and consistency.

[0079] Additionally, the policy decision engine processes the data. In one optional implementation, it can employ dynamic parameter tuning based on reinforcement learning. That is, a reinforcement learning model is used for decision-making. The state space includes: heatmap value, network bandwidth, and node load. The reward function is as follows: Reinforcement learning uses dynamic parameter tuning, primarily relying on factors such as heatmap values, network bandwidth, and node load to make decisions. α, β, and γ are weight parameters used to balance the importance of different factors and are configured according to requirements or actual application scenarios. IOPS improvement is a positive reward, representing an increase in input / output operations per second; bandwidth consumption is a negative cost, representing the use of network bandwidth; and CPU overload penalty is a negative reward, representing the penalty when the CPU is overloaded.

[0080] In this embodiment, a multi-time-window monitoring strategy can be adopted to achieve a balance between short-term response and long-term optimization, while considering the overhead that may be caused by frequent strategy switching. Multi-time windows can be roughly divided into: real-time windows (seconds to minutes), short-term windows (hours), and long-term windows (days to weeks) based on actual application scenarios and requirements. Data such as the load of monitoring storage nodes (e.g., CPU, memory, disk I / O utilization) and network bandwidth status can be processed through aggregation and standardization to transform into a fixed-dimensional, numerical mathematical representation that can be directly processed by machine learning models—that is, a state vector.

[0081] For reinforcement learning models, the state space (S) includes data access frequency, node load (CPU, memory, disk I / O), network bandwidth, etc. The action space (A) is discrete, representing the selection of different EC (Extended Equalizer) policies, such as: Action 0: RS(4,2), Action 1: RS(6,3), Action 2: RS(10,4), Action 3: LRC(6,2,2). The reward function (R) is mainly used to encourage the system to achieve a balance between performance, cost, and stability. It includes: Performance reward: negatively correlated with data access latency (lower latency, higher reward). Cost penalty: positively correlated with storage space usage (more storage space used, higher penalty, i.e., negative reward). Migration cost penalty: when the policy changes, data migration consumes network bandwidth and computing resources, thus requiring a negative reward. Stability penalty: a penalty is given if the system load is too high (e.g., high CPU, high I / O) after the policy change. An example of a reward function design is shown in the following formula.

[0082] R = w1 * (1 / average latency) - w2 * storage overhead - w3 * migration cost - w4 * load metric, where w1, w2, w3, and w4 represent weights and can be customized. The reinforcement learning model training process (taking a deep Q-network (DQN) as an example) can include the following steps.

[0083] 2.1 Initialization: Initialize the Q-network (action value function) and the target Q-network, and the experience playback buffer.

[0084] 2.2 Interactive loop: for each time step t (e.g., every certain period of time, such as 5 minutes).

[0085] a. Observe the current state s_t.

[0086] b. Select an action a_t (i.e., select an EC policy) based on the current Q network and the exploration-exploitation policy (such as ε-greedy).

[0087] c. Execute action a_t: migrate the data to the new EC policy (Note: migration takes time, but it can be simulated to be completed instantly, or you can wait for the migration to be completed and observe the new status).

[0088] d. Observe the new state s_{t+1} and the reward r_t.

[0089] e. Store the experience (s_t, a_t, r_t, s_{t+1}) into the experience replay buffer.

[0090] f. Sample a batch of experiences from the experience replay buffer.

[0091] g. Calculate the target Q value: For each sample, the target Q value = r + γ * max{a} Q_target(s{t+1}, a) (where γ is the discount factor).

[0092] h. Calculate the predicted Q value using the current Q-network, and calculate the loss (e.g., mean squared error).

[0093] i. Update the current Q network.

[0094] j. Update the target Q network periodically (e.g., copy the parameters of the current Q network to the target Q network every C steps).

[0095] 2.3 Repeat the above steps until convergence.

[0096] A sample can be a record randomly drawn from the experience replay buffer, containing the complete experience gained by the model after making a decision at time step t. This includes s_t (current state), a_t (action taken), r_t (reward obtained), and s_{t+1} (next state). The objective Q-value is calculated as: Total value of a decision = Immediate reward + Discounted future best value. r (r_t): Immediate reward. This is the reward immediately obtained after performing action a_t (e.g., changing the policy of data block A to LRC) in state s_t, reflecting the short-term benefits or drawbacks of this policy change (e.g., reduced access latency immediately after migration). γ (discount factor): A constant between 0 and 1 (e.g., 0.9), representing the importance placed on future rewards. γ close to 0: The model only cares about the immediate reward r and not the long-term impact of the decision; γ close to 1: The model fully considers the long-term impact of the current decision on the future system state. In this invention, the impact of policy changes is long-term (e.g., an efficient policy may continue to save storage space for hours or even days). Therefore, γ should be set to a relatively high value (e.g., 0.99) to encourage the model to consider long-term gains. `max{a} Q_target(s_{t+1}, a)`: The present value of future rewards. Where `s_{t+1}`: The new state the system enters after executing the action. `Q_target(s_{t+1}, a)`: This is the target Q-network's estimate of the total future reward obtainable by taking all possible actions `a` (i.e., all optional EC policies) in the new state `s_{t+1}`. `max{a}`: The Q-value corresponding to the action `a` that yields the maximum future reward in state `s_{t+1}`. This represents the best future prospect achievable by following the current optimal policy. The target Q-network is a copy of the current Q-network and is not directly used for decision-making; it is only used to calculate the training target Q-value (i.e., `Q_target(s_{t+1}, a)` in the formula above), and its parameter updates are lagging. Typically, the parameters of the current Q-network are copied to the target Q-network only every certain period of time (e.g., every few hundred or thousands of training steps). The target Q network is a concept of the DQN algorithm.

[0097] Furthermore, embodiments of the present invention employ low-overhead conversion techniques: a two-stage incremental migration protocol with atomic commit and rollback mechanisms.

[0098] The real-time update mechanism in this embodiment of the invention is as follows: node access logs are collected every second, and the heatmap matrix is ​​updated through Kafka stream processing. The logs refer to the logs of data blocks being read or written, and the logs correspond to time points. Access logs are collected every second. Although the sliding time window is 24 hours, it is not necessary to recalculate the complete 24-hour data every second. Instead, an incremental update method is used.

[0099] Furthermore, in optional implementations, embodiments of the present invention can perform spatial-dimensional topology association. For example, data blocks are divided into 256MB granularities to construct a cross-rack-aware topology matrix, allowing the system to know the rack and server where each data block resides (i.e., topology information). Then, a topology matrix can be constructed, recording the communication costs at different locations (e.g., between racks, between servers). The communication cost weight is 0 within a unified node, 0.3 within a rack, and 0.7 across racks. To optimize data migration paths, the system can consider the network overhead incurred during data migration. For example, the communication cost weight is 0.3 (lower) for data transmission within the same rack, while it is 0.7 (higher) for cross-rack data transmission. Thus, when planning data migration paths, the system will prioritize paths with lower communication costs, such as completing migration within the rack as much as possible to avoid cross-rack migration.

[0100] The data flow architecture is as follows: Node access logs (per second) (raw data) → Kafka stream processing (streaming aggregation) → Heatmap matrix update (heat calculation) → Topology optimization decision (migration path planning). The overall idea of ​​this architecture is to collect node access logs every second and update the heatmap matrix through Kafka stream processing (real-time update mechanism). When data migration is required, the heatmap matrix is ​​used to determine the hotness or coldness of data blocks, and the optimal migration path is selected based on the topology matrix (intra-rack and cross-rack costs), thereby minimizing migration costs. In this way, the system can quickly respond to changes in data access patterns and intelligently select the path with lower network costs during data migration.

[0101] This invention provides a dynamic coupling mechanism for three-dimensional heatmaps, fusing time decay factors and operation type intensity into a single heat value. It solves the problem of a 30% misjudgment rate for hot and cold data in traditional solutions (actually reduced to 5%) by using multi-dimensional data for strategy decision-making, selecting RS code or LRC code strategies based on heat values, node load, and real-time bandwidth. It provides a local reconstruction technique for incremental migration, based on a difference comparison algorithm (CRC64 checksum comparison) of data block version snapshots, compressing the migration time of 10TB of data from 18 hours to 2 hours. The recoding executor employs a two-stage migration protocol: the first stage transmits the differing data blocks, and the second stage reconstructs the checksum blocks in parallel. Changes in erasure coding strategies cause changes in the storage location of data blocks, including metadata and data. This invention only migrates the changed data portions. In the storage system, a data block may only be partially modified, not entirely. Therefore, differences can be identified by comparing the current version and the previous version of the data block, and only the differing portions are transmitted. Version snapshots of data blocks can be created: at a certain point in time, a snapshot (recording the state of the data block) is created for the data block. When migration is required, compare the differences between the current data block and the snapshot. For example, use CRC64 checksums for comparison: divide the data block into fixed-size blocks (e.g., 4KB), calculate a CRC64 checksum for each block. Compare the checksums of the two snapshot versions; only blocks with different checksums need to be migrated. Only re-encode and transmit changed blocks: for changed blocks, decode the original data from the old EC policy (only needing to read k fragments, i.e., k fragments from the old EC policy), then encode using the new EC policy, transmitting only the changed fragments from the new EC policy.

[0102] This invention demonstrates significant value across several key dimensions. In terms of performance improvement, hot data read latency is reduced by a substantial 63.2%, from 87ms to 32ms, while system IOPS can increase by 20-50% depending on workload, effectively improving data processing efficiency. Regarding cost optimization, storage space utilization is improved by 35%, averaging 82%, and bandwidth consumption during the conversion process is reduced by 37.5%, achieving efficient resource utilization and cost savings. In terms of reliability assurance, atomic commits ensure zero data loss during conversion, and local fault recovery time can be controlled within 30 seconds, providing solid support for data security and stability. In terms of intelligent operation and maintenance, manual configuration requirements are reduced by 85%, and the anomaly self-healing rate exceeds 95%, greatly alleviating operational pressure and improving the system's automated management level.

[0103] Further, see Figure 4 The diagram shown is a structural schematic of an erasure coding policy conversion device provided in an embodiment of the present invention. The erasure coding policy conversion device includes the following modules.

[0104] Access popularity determination module 41 is used to determine the access popularity of the data block based on the read and write requests of the data block.

[0105] The node information acquisition module 42 is used to acquire the load information and network bandwidth information of the storage node where the data block is located.

[0106] The erasure coding strategy determination module 43 is used to determine the target erasure coding strategy corresponding to the data block based on the access popularity, the load information and the network bandwidth information.

[0107] The erasure coding policy conversion module 44 is used to determine the changed data sub-block from the data block if the target erasure coding policy is inconsistent with the current erasure coding policy of the data block, and to encode the data sub-block using the target erasure coding policy to obtain target policy encoded data.

[0108] As can be seen, the embodiments of the present invention determine the access popularity of data blocks through read and write requests. When determining the target erasure coding strategy corresponding to a data block, the access popularity of the data block, the load information of the storage node, and the network bandwidth information are comprehensively considered to dynamically and intelligently determine the erasure coding strategy that suits the data block. Furthermore, in the case of inconsistency with the current erasure coding strategy, the changed data sub-blocks are encoded and converted, that is, the incremental data is encoded and converted. In this way, dynamic conversion of erasure coding strategy can be achieved, and the resource overhead of the conversion process can be reduced, thereby improving storage performance.

[0109] In an optional implementation, the access popularity determination module 41 is specifically used to: determine the access popularity of the data block based on the number of read requests, the number of write requests, and the weight coefficients corresponding to the read and write requests in the current time period.

[0110] In an optional implementation, the access popularity determination module 41 is specifically used to: determine the access popularity of the data block based on the number of read requests, the number of write requests, and the weight coefficients corresponding to the read requests and write requests in the current time period and the most recent preset number of historical time periods.

[0111] In an optional implementation, the access popularity determination module 41 is further configured to: control the contribution of the historical time period to the access popularity by using a preset attenuation factor and the distance between the historical time period and the current time period, wherein the farther away from the current time period, the lower the contribution.

[0112] In an optional embodiment, the device further includes a parameter adjustment module for detecting performance indicators; and adjusting the weighting coefficient and / or the preset attenuation factor if the performance indicators do not meet the preset standards.

[0113] In an optional implementation, the performance metrics include read latency, number of read / write operations per second, and storage utilization.

[0114] In an optional implementation, the erasure coding strategy determination module 43 may specifically include: a decision score calculation submodule, used to perform weighted calculation based on the access popularity, the load information and the network bandwidth information to obtain a decision score; and an erasure coding strategy determination submodule, used to determine the target erasure coding strategy corresponding to the data block based on the decision score.

[0115] The erasure coding strategy determination submodule can be specifically used to: compare the decision score with a preset threshold; if the decision score is greater than the preset threshold, then the data block is determined to be hot data and the target erasure coding strategy is a locally repairable erasure coding strategy; otherwise, it is determined to be cold data and the target erasure coding strategy is a Reed-Solomon code strategy.

[0116] In an optional implementation, the erasure coding strategy determination module 43 can be used to: generate a state vector based on the access popularity, the load information, and the network bandwidth information; determine the action corresponding to the state vector using a target reinforcement learning model, and obtain the target erasure coding strategy corresponding to the data block.

[0117] The target reinforcement learning model employs a preset reward function during training. This preset reward function is determined based on performance rewards, cost penalties, and stability penalties. The cost penalties include storage cost penalties and / or migration cost penalties.

[0118] The erasure coding strategy conversion module 44 may include a changed data determination submodule, which is used to determine the changed data sub-blocks from the data block.

[0119] The changed data determination submodule may specifically include: a sub-block division unit, used to divide the data block into data sub-blocks of a preset size; and a changed sub-block determination unit, used to compare the data sub-block with the snapshot data corresponding to the data sub-block at a preset historical time point, and if they are inconsistent, to determine that the data sub-block is a changed data sub-block.

[0120] Specifically, the variable sub-block determination unit can be used to calculate the verification value of the data sub-block to obtain a first verification value; calculate the verification value of the snapshot data corresponding to the data sub-block at a preset historical time point to obtain a second verification value; compare the first verification value and the second verification value, and if the first verification value and the second verification value are consistent, then the data sub-block is consistent with the snapshot data corresponding to the data sub-block at the preset historical time point; otherwise, they are inconsistent.

[0121] Furthermore, the apparatus may further include: a migration location determination module, used to determine the migration target location of the target policy encoded data; and a data migration module, used to migrate the target policy encoded data to the migration target location.

[0122] The migration location determination module is specifically used to determine the migration target location of the target policy encoded data as a storage location in a local node if the target erasure coding policy is a locally repairable erasure coding policy, wherein the local node is the storage node where the data block is located or a storage node in the same rack; and to determine the migration target location of the target policy encoded data as a storage location in a remote node if the target erasure coding policy is a Reed-Solomon code policy, wherein the remote node is a storage node in a different rack than the storage node where the data block is located.

[0123] The migration location determination module is specifically used to: filter target local nodes among all local nodes based on preset performance filtering criteria, and determine the migration target location of the target policy encoded data as the storage location in the target local node.

[0124] In an optional implementation, the access popularity determination module 41 is specifically used for: reading data access logs from a message queue; wherein the data access logs are collected periodically at a preset frequency and placed in the message queue; parsing read and write requests for data blocks from the data access logs, and determining the access popularity of the data blocks.

[0125] Figure 4 For a description of the features in the corresponding embodiments, please refer to Figure 3 The relevant descriptions of the corresponding embodiments will not be repeated here.

[0126] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 5 As shown, the electronic device includes: a memory 50 for storing a computer program; and a processor 51 for executing the computer program to implement the steps of the erasure coding strategy conversion method as described in the above embodiments.

[0127] The electronic device provided in this embodiment may include, but is not limited to, servers. The processor 51 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 51 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 51 may also include a main processor and a coprocessor. The main processor, also known as a central processing unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 51 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 51 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0128] The memory 50 may include one or more computer-readable storage media, which may be non-transitory. The memory 50 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 50 is used to store at least the following computer program 501, which, after being loaded and executed by the processor 51, is capable of implementing the relevant steps of the erasure coding policy conversion method disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 50 may also include an operating system 502 and data 503, and the storage method may be temporary or permanent storage. The operating system 502 may include Windows, Unix, Linux, etc. The data 503 may include, but is not limited to, log data, decision data, etc.

[0129] In some embodiments, the electronic device may further include a display screen 52, an input / output interface 53, a communication interface 54, a power supply 55, and a communication bus 56.

[0130] Those skilled in the art will understand that Figure 5 The structures shown do not constitute a limitation on electronic devices and may include more or fewer components than those shown.

[0131] It is understood that if the erasure coding strategy conversion method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0132] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the erasure coding strategy conversion method described above.

[0133] Furthermore, embodiments of the present invention provide a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the aforementioned erasure coding strategy conversion method.

[0134] The foregoing provides a detailed description of an erasure coding strategy conversion method, apparatus, device, medium, and product provided by embodiments of the present invention. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0135] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0136] The foregoing provides a detailed description of the erasure coding strategy conversion method, apparatus, device, medium, and product provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this invention. It should be noted that those skilled in the art can make various improvements and modifications to this invention without departing from its principles, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A method for converting erasure coding strategies, characterized in that, The method comprises the following steps: determining the access heat of the data block based on the read and write requests of the data block; obtaining the load information and network bandwidth information of the storage node where the data block is located; determining the target erasure code strategy corresponding to the data block based on the access heat, the load information and the network bandwidth information; if the target erasure code strategy is inconsistent with the current erasure code strategy of the data block, determining the changed data sub-block from the data block, and encoding the data sub-block by using the target erasure code strategy to obtain target strategy encoded data; wherein, the access heat of the data block is determined based on the number of read requests, the number of write requests and the weight coefficients corresponding to the read requests and the write requests of the data block in the current time period and the last preset number of historical time periods; the read requests and the write requests correspond to different weight coefficients or the same weight coefficients; the contribution degree of the historical time period to the access heat is controlled by a preset decay factor and the distance between the historical time period and the current time period, wherein the farther the distance is, the lower the contribution degree is; after the target strategy encoded data is obtained by encoding the data sub-block by using the target erasure code strategy, the method further comprises detecting a performance index; and adjusting the weight coefficients and / or the preset decay factor if the performance index does not reach a preset standard.

2. The method of claim 1, wherein, The performance index includes read delay, number of read and write operations per second, and storage utilization.

3. The method of claim 1, wherein, The target erasure code strategy corresponding to the data block is determined based on the access heat, the load information and the network bandwidth information, which comprises: performing weighted calculation based on the access heat, the load information and the network bandwidth information to obtain a decision score; determining the target erasure code strategy corresponding to the data block based on the decision score.

4. The method of claim 3, wherein, The target erasure code strategy corresponding to the data block is determined based on the decision score, which comprises: comparing the decision score with a preset threshold value; if the decision score is greater than the preset threshold value, determining that the data block is hot data and the target erasure code strategy is a local repairable erasure code strategy, otherwise determining that the data block is cold data and the target erasure code strategy is a Reed-Solomon code strategy.

5. The method of claim 1, wherein, The target erasure code strategy corresponding to the data block is determined based on the access heat, the load information and the network bandwidth information, which comprises: generating a state vector based on the access heat, the load information and the network bandwidth information; determining the action corresponding to the state vector by using a target reinforcement learning model to obtain the target erasure code strategy corresponding to the data block.

6. The method of claim 5, wherein, In the training process of the target reinforcement learning model, a preset reward function is used, which is determined based on a performance reward, a cost penalty and a stability penalty; the cost penalty includes a storage cost penalty and / or a migration cost penalty.

7. The method of claim 1, wherein, The changed data sub-block is determined from the data block, which comprises: dividing the data block into data sub-blocks of a preset size; The data sub-block is compared with snapshot data corresponding to the data sub-block at the preset historical time point, and if the data sub-block is inconsistent with the snapshot data, the data sub-block is determined as a changed data sub-block.

8. The method of claim 7, wherein, The data sub-block is compared with snapshot data corresponding to the data sub-block at the preset historical time point, and if the data sub-block is inconsistent with the snapshot data, the data sub-block is determined as a changed data sub-block. The check value of the data sub-block is calculated to obtain a first check value. The check value of the snapshot data corresponding to the data sub-block at the preset historical time point is calculated to obtain a second check value. The first check value and the second check value are compared, and if the first check value and the second check value are consistent, the data sub-block is consistent with the snapshot data corresponding to the data sub-block at the preset historical time point, otherwise, the data sub-block is inconsistent with the snapshot data.

9. The method of any one of claims 1 to 8, wherein, After the target policy encoded data is obtained by encoding the data sub-block by using the target erasure code strategy, the method further includes: determining a migration target position of the target policy encoded data; migrating the target policy encoded data to the migration target position.

10. The method of claim 9, wherein, The migration target position of the target policy encoded data is determined, including: if the target erasure code strategy is a local repairable erasure code strategy, the migration target position of the target policy encoded data is determined as a storage position in a local node, wherein the local node is a storage node where the data block is located or a storage node in the same cabinet; if the target erasure code strategy is a Reed-Solomon code strategy, the migration target position of the target policy encoded data is determined as a storage position in a remote node, and the remote node is a storage node that is not in the same cabinet as the storage node where the data block is located.

11. The method of claim 10, wherein, The migration target position of the target policy encoded data is determined as a storage position in a local node, including: filtering a target local node from all local nodes based on a preset performance filtering standard, and determining the migration target position of the target policy encoded data as a storage position in the target local node.

12. The method of claim 1, wherein, The access heat of the data block is determined based on read and write requests of the data block, including: reading data access logs from a message queue; wherein the data access logs are collected at a preset frequency and placed into the message queue; parsing read and write requests of the data block from the data access logs, and determining the access heat of the data block.

13. An erasure coding strategy conversion device, characterized in that, including: an access heat determination module configured to determine the access heat of the data block based on read and write requests of the data block; a node information acquisition module configured to acquire load information and network bandwidth information of a storage node where the data block is located; an erasure code strategy determination module configured to determine a target erasure code strategy corresponding to the data block based on the access heat, the load information, and the network bandwidth information; an erasure code strategy conversion module configured to, if the target erasure code strategy is inconsistent with a current erasure code strategy of the data block, determine changed data sub-blocks from the data block, and encode the data sub-blocks by using the target erasure code strategy to obtain target policy encoded data; The access hotness of the data block is determined based on the number of read requests, the number of write requests, and weight coefficients corresponding to the read requests and the write requests of the data block in a current time period and a preset number of historical time periods; the read requests and the write requests correspond to different weight coefficients or the same weight coefficients; a contribution degree of the historical time periods to the access hotness is controlled through a preset decay factor and a distance between the historical time periods and the current time period, wherein the farther from the current time period, the lower the contribution degree; After the data sub-blocks are encoded by using the target erasure code strategy to obtain target strategy encoded data, the method further includes detecting a performance index; and adjusting the weight coefficients and / or the preset decay factor in a case where the performance index does not reach a preset standard.

14. An electronic device, comprising: The method comprises: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the erasure code strategy conversion method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium and is executed by the processor to implement the steps of the erasure code strategy conversion method according to any one of claims 1 to 12.

16. A computer program product comprising computer programs / instructions, characterized in that, The computer program / instructions are executed by the processor to implement the steps of the erasure code strategy conversion method according to any one of claims 1 to 12.

Images