Data protection method and device based on popularity, equipment, medium and product

Abstract

The invention discloses a heat-based data protection method and device, equipment, a medium and a product. Relates to the technical field of data processing. The heat-based data protection method comprises the following steps: monitoring the access intensity of each data object in a storage pool; determining a popularity level and a target data protection strategy according to the access strength of each data object; and converting the data objects of which the current data protection strategy is inconsistent with the corresponding target data protection strategy into the corresponding target data protection strategy from the current data protection strategy. According to the technical scheme, the data protection strategy of the data object is dynamically adjusted according to the access strength, fine protection of the data is realized by taking the data object as granularity, cross-pool migration is not needed in the process of adjusting the data protection strategy, the overhead is low, the access strength is taken as the adjustment basis of the data protection strategy, and the data protection efficiency is improved. Storage and performance of data with different access intensities can be effectively balanced, and automatic and efficient dynamic data protection is achieved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data protection method, apparatus, device, medium and product based on heat. Background Technology

[0002] Ceph is a widely used open-source distributed storage system known for its high scalability, reliability, and performance. In Ceph, data is stored as objects in logical units called "storage pools." Data protection policies can be set for data objects within a single storage pool using CRUSH rules and storage policies. A Ceph pool is a logical data container, and each pool can be associated with one CRUSH rule. This rule defines how data is distributed across the cluster. Traditionally, only one CRUSH rule can be used per pool, meaning all data within the pool follows the same data protection policy. However, in some scenarios, data within the same pool can be protected using different policies. This can be achieved through storage policies, which logically divide a pool into multiple subsets (called "namespaces" or "data pools"), each of which can apply different CRUSH rules. For example, multiple namespaces can be created within the same pool, and each namespace can be associated with different CRUSH rules. Data in different namespaces within the same pool can then use different data protection strategies. Alternatively, Ceph can specify the CRUSH rules used by each object through extended attributes or special tags, allowing different objects to use different data protection strategies within the same pool or even the same namespace.

[0003] Currently, the data protection strategy for storage pools is mainly static. This means that a storage pool selects a single data protection strategy during its creation and keeps it fixed throughout its lifecycle. The pool's capacity remains unchanged, and data access performance remains constant. This results in stagnant performance for frequently accessed data while infrequently accessed data consumes excessive storage space. In some scenarios, the data protection strategy can be adjusted. For example, multiple pools with different data protection strategies can be created, and caching tiers or manual data migration strategies can be configured to migrate data between different pools, thus adjusting the data packet strategy.

[0004] In general, existing data protection strategies have several shortcomings: they operate on a per-pool or per-object-set basis, failing to provide granular management of the lifecycle of individual objects; they rely on manual administrator intervention, making it difficult to respond in real-time to changes in business access patterns, resulting in low automation; and data migration between pools generates additional network I / O and disk I / O, incurring significant migration overhead, impacting overall cluster performance, and the process is opaque, making it impossible to achieve smooth, gradual changes to individual object protection strategies. Therefore, a fine-grained, low-overhead, automated, and efficient dynamic data protection mechanism is urgently needed. Summary of the Invention

[0005] This application provides a data protection method, apparatus, device, medium, and product based on heat intensity to achieve automatic and efficient dynamic data protection and effectively balance the storage and performance of data with different access intensities.

[0006] In a first aspect, embodiments of this application provide a data protection method based on heat intensity, including: Monitor the access intensity of each data object within the storage pool; Based on the access intensity of each data object, determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level; The data object to be protected is converted from the current data protection policy to the corresponding target data protection policy, wherein the data object to be protected is a data object whose current data protection policy and the corresponding target data protection policy are inconsistent.

[0007] Secondly, embodiments of this application also provide a data protection device based on heat intensity, comprising: The monitoring module is used to monitor the access intensity of each data object in the storage pool; The decision module is used to determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level based on the access intensity of each data object. The protection module is used to convert the data object to be protected from the current data protection policy to the corresponding target data protection policy, wherein the data object to be protected is a data object whose current data protection policy and the corresponding target data protection policy are inconsistent.

[0008] Thirdly, embodiments of this application provide an electronic device, including: One or more processors; Storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the heat-based data protection method as described in the first aspect.

[0009] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the heat-based data protection method as described in the first aspect.

[0010] Fifthly, embodiments of this application also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the heat-based data protection method as described in any of the above embodiments.

[0011] This application provides a data protection method, apparatus, device, medium, and product based on access intensity. The method includes: monitoring the access intensity of each data object in a storage pool; determining the access intensity of each data object and a corresponding target data protection strategy based on that access intensity; and converting the data object to be protected from the current data protection strategy to the corresponding target data protection strategy, wherein the data object to be protected is a data object whose current data protection strategy and the corresponding target data protection strategy are inconsistent. This technical solution dynamically adjusts the data protection strategy of data objects based on access intensity, achieving fine-grained data protection at the data object level. The adjustment of the data protection strategy does not require cross-pool migration, resulting in low overhead. Furthermore, using access intensity as the basis for adjusting the data protection strategy effectively balances the storage and performance of data with different access intensities, achieving automatic and efficient dynamic data protection. Attached Figure Description

[0012] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0013] Figure 1 A flowchart illustrating a heat-based data protection method provided in this application embodiment; Figure 2 This is a schematic diagram illustrating the hot status migration of a data object and the corresponding data protection strategy conversion, provided as an embodiment. Figure 3 This is a schematic diagram illustrating the conversion from a multiple-replica strategy to an erasure coding strategy, as provided in one embodiment. Figure 4 This is a schematic diagram illustrating the conversion from an erasure coding strategy to a multiple-replica strategy, as provided in one embodiment. Figure 5 A schematic diagram of a heat-based data protection process provided in one embodiment; Figure 6A schematic diagram illustrating the implementation of a heat-based data protection system is provided as an example. Figure 7 A schematic diagram of a heat-based data protection device provided in this application embodiment; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. Furthermore, it should be noted that, for ease of description, the accompanying drawings show only the parts relevant to the present application, not the entire structure.

[0015] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0016] It should be noted that the concepts of "first" and "second" mentioned in the embodiments of this application are only used to distinguish different devices, modules, units or other objects, and are not used to limit the order of functions performed by these devices, modules, units or other objects or their interdependencies.

[0017] Furthermore, the embodiments and features described in this application may be combined with each other, unless otherwise specified.

[0018] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0019] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the relevant content of the solution.

[0020] Figure 1This flowchart illustrates a heat-based data protection method provided in this application embodiment. This embodiment is applicable to situations where data objects in a storage pool need to be protected. Specifically, this heat-based data protection method can be executed by a heat-based data protection device, which can be implemented through software and / or hardware and integrated into an electronic device. The electronic device includes, but is not limited to, devices with data processing capabilities such as computers, smartphones, or servers.

[0021] like Figure 1 As shown, the method specifically includes the following steps: S110, Monitor the access intensity of each data object in the storage pool; For example, a storage pool contains multiple data objects. At the Ceph Object Storage Daemon (OSD) level, the intensity of access operations (including read and write operations) for each object in the storage pool can be counted and recorded. Access intensity can refer to the frequency of access within a recent period, or it can be related to the time of the most recent access, reflecting whether the data object is frequently accessed, whether it has been accessed recently, or the probability of it being accessed in the future. Generally, the higher the access frequency or the more recent the time of the most recent access, the higher the access intensity, the greater the probability of it being accessed in the future, and the higher the requirements for its read performance. Different data protection strategies can be adopted for data objects with different access intensities.

[0022] In one example, this statistical information can be collected periodically (e.g., every minute) by a separate popularity analysis module, which calculates the access frequency of each object within a past time window (e.g., 10 minutes) and adds corresponding labels, such as popularity tags, based on the access frequency.

[0023] S120. Based on the access intensity of each data object, determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level; For example, popularity level, also known as popularity status, reflects the frequency or probability of a data object being accessed over a past period and in the future. Access intensity is positively correlated with popularity level; the higher the access intensity, the higher the popularity level. Access intensity can be divided into different intervals, each corresponding to a specific popularity level. Popularity levels include at least two levels, such as high popularity and low popularity, or high popularity, medium popularity, and low popularity, or hot level, warm level, and cold level.

[0024] A data object in a storage pool can employ at least two data protection strategies, with different "heat" levels corresponding to different target data protection strategies. For example, high heat corresponds to the first data protection strategy, and low heat corresponds to the second data protection strategy; or, a hot level corresponds to the first data protection strategy, a moderate level corresponds to the second data protection strategy, and a cold level corresponds to the third data protection strategy, and so on. It should be noted that this application embodiment does not specifically limit the types and number of data protection strategies employed for data objects in the storage pool. For ease of description, this application embodiment mainly uses two data protection strategies as examples to illustrate the dynamic data protection process.

[0025] One data protection strategy is the multi-replica strategy, which copies the same data N times and stores them on different physical devices to provide high read and write performance. N is a positive integer, for example, N defaults to 3, and can be as small as 1 or as large as the number of nodes in the Ceph cluster. This data protection strategy has relatively low storage utilization, but data reading is efficient and reliable, and read performance is relatively high, making it suitable for data objects with high frequency of use.

[0026] Another data protection strategy is erasure coding (EC), which divides the data into K blocks and calculates M parity blocks, storing a total of K+M coded blocks. Data can be recovered as long as any K coded blocks survive. Erasure coding strategies can be further subdivided, with different K and / or M values. This data protection strategy has relatively high storage utilization (e.g., K=4, M=2, utilization 4÷6≈67%), but the overhead of write operations and data recovery is relatively large, and read performance is relatively low. It is suitable for data objects with low accessibility. Understandably, different values ​​for K and M will affect utilization; data objects with different accessibility levels can use different erasure coding strategies with different K and / or M values.

[0027] S130. Convert the data object to be protected from the current data protection policy to the corresponding target data protection policy, wherein the data object to be protected is a data object whose current data protection policy is inconsistent with the corresponding target data protection policy. For example, the data object to be protected can be understood as a data object whose current data protection policy is not optimal or suitable, or as an object whose data protection policy needs to be converted or adjusted. Specifically, if the current data protection policy of a data object is inconsistent with the target data protection policy corresponding to its popularity level, then the data object is a data object to be protected, and its data protection policy needs to be converted to the target data protection policy. During the conversion process, the data object to be protected can still be accessed.

[0028] For example, if a data object's current data protection strategy is a multi-replica strategy, but its access intensity over a past period determines its "hotness" level to be low, and the corresponding target data protection strategy is erasure coding, then its data protection strategy will be changed from multi-replica to erasure coding. It's important to note that this process requires appropriate methods to clean up redundant data in the multi-replica strategy, and also involves data block partitioning, checksum calculation, and coded block storage. Similarly, if a data object's data protection strategy is changed from erasure coding to multi-replica, the conversion process requires appropriate methods to clean up redundant coded blocks in the erasure coding strategy, and also involves copying and storing the data object. Furthermore, another scenario involves converting a data object from one erasure coding strategy to another, which requires adjusting the K and / or M values ​​in the erasure coding strategy. During the conversion process, the old coded blocks need to be decoded, and the new coded blocks need to be re-partitioned and calculated before being written to the storage location.

[0029] In one example, a central decision engine can periodically check the popularity level (tag) of data objects and determine whether the current data protection policy of the data object is inconsistent with the target data protection policy based on the mapping rules between popularity level and target data protection policy. If they are inconsistent, a policy conversion task can be generated for that data object and placed in a task queue. The task information of the policy conversion task can include the data object identifier (ID), the current data protection policy, and the target data protection policy. The decision engine described above can be integrated into CephMonitor or used as a standalone service.

[0030] In one example, one or more transformation executors can retrieve policy transformation tasks from the task queue and perform data protection policy transformations on the corresponding data objects, a process transparent to the client. For instance, in the transformation from a multi-replica policy to an erasure coding policy, all replicas of the data object need to be read, erasure coding is re-performed to generate K+M coded blocks, and the K+M coded blocks are written to a new OSD location according to the CRUSH rule. After successful writing, the old replica data is deleted. As another example, in the transformation from an erasure coding policy to a multi-replica policy, any K coded blocks of the data object need to be read, the complete object data decoded, and N replicas of the complete object data written to a new OSD location according to the CRUSH rule of the replica pool. After successful writing, the old coded blocks are deleted. For example, in the process of converting from the old erasure coding strategy (with erasure coding parameters K1 and M1) to the new erasure coding strategy (with erasure coding parameters K2 and M2), it is necessary to read any K1 coded blocks of the data object, decode the complete object data, re-encode the erasure coding, generate K2+M2 coded blocks, write the K2+M2 coded blocks to the new OSD location according to the CRUSH rule, and delete the old copy data after successful writing.

[0031] This application provides a data protection method based on access intensity, which dynamically adjusts the data protection strategy of data objects according to access intensity. It achieves fine-grained protection of data at the data object level, and transparently converts the object's data between different data protection strategies in the background without cross-pool migration, resulting in low overhead. Furthermore, by using access intensity as the basis for adjusting the data protection strategy, it can improve the access performance of frequently accessed data and reduce the protection level of infrequently accessed data, thereby effectively balancing the storage and performance of data with different access intensities and achieving automatic and efficient dynamic data protection.

[0032] In one embodiment, determining the popularity level of each data object and the target data protection strategy corresponding to the popularity level based on the access intensity of each data object includes: determining the popularity level of each data object and the target data protection strategy corresponding to the popularity level based on the access intensity of each data object and the access intensity threshold; the popularity level of each data object includes at least one of the following: hot level, warm level, and cold level; the target data protection strategy corresponding to the hot level is a multi-copy strategy, and the target data protection strategies corresponding to the warm level and cold level are erasure coding strategies with different parameters.

[0033] In this embodiment, the access intensity of data objects and access intensity thresholds can be used to determine the heat level, and a dynamic protection strategy can be configured for the storage pool accordingly. Taking access frequency as an example, access frequency thresholds T1 and T2 can be defined. Access frequencies below T1 are classified as cold levels, access frequencies between T1 and T2 are classified as warm levels, and access frequencies above T2 are classified as hot levels. Furthermore, a target data protection strategy is specified for each heat level. For example, the target data protection strategy for hot levels is a multi-replica strategy; the target data protection strategy for warm levels is an erasure coding strategy (K=4, M=2); and the target data protection strategy for cold levels is an erasure coding strategy (K=8, M=3). This erasure coding strategy has higher storage utilization than the previous one. Based on this, flexible and adaptive fine-grained protection can be implemented for data objects with different access intensities.

[0034] In one embodiment, after converting the data object to be protected from the current data protection policy to the corresponding target data protection policy, the method further includes: S140, updating the metadata of the data object to be protected.

[0035] For example, after a successful data protection policy transition, the metadata of the corresponding data object is updated, such as updating the data object's extended attributes and / or specific OMAP (Object Map) key-value pairs to record its new data protection policy. During this process, entries for the corresponding policy transition task in the task queue can be cleaned up. Subsequent access to the data object will be based on the new data protection policy.

[0036] Figure 2 This is a schematic diagram illustrating the hot status migration of a data object and the corresponding data protection strategy conversion, provided as an embodiment. For example... Figure 2As shown, the popularity status refers to the popularity level, and the access intensity is represented by the access frequency. Popularity status includes hot, warm, and cold states. The data object is in its initial state when it is created. The initial state can be the status of the popularity tag, or it can be empty, or it can be a specified default popularity status, such as cold. If the access frequency exceeds the first threshold (5 ops / min), the data object's hot status changes to hot, and the corresponding target data protection strategy changes to a 3-replica (N=3 multiple replicas) strategy. If the access frequency decreases to or below the first threshold, the data object's hot status changes to warm, and the corresponding target data protection strategy changes to erasure coding (K=4, m=2). If the access frequency recovers to or exceeds the first threshold, the data object's hot status changes to hot, and the corresponding target data protection strategy changes to a 3-replica strategy. If the access frequency further decreases to or below the second threshold (1 op / day), the data object's hot status changes to cold, and the corresponding target data protection strategy changes to erasure coding (K=8, m=3). If the access frequency recovers to or exceeds the second threshold but does not exceed the first threshold, the data object's hot status changes to warm, and the corresponding target data protection strategy changes to erasure coding (K=4, m=2). Based on this, a data protection strategy change driven by data popularity was realized. That is, data objects can migrate between different popularity states according to changes in access intensity. The most appropriate data protection strategy can be adopted in each popularity state, forming a complete closed-loop management mechanism.

[0037] In one example, the analysis of the mapping between popularity status and data protection policy is as follows: 1) For the thermal state: Status indicators (or labels): can be marked in red or warm colors to represent the highest level of activity; Access characteristics include: Access frequency: More than 5 ops / min (configurable); Recent visits: Frequent visits in the past hour; Access mode: typically random read / write, low latency required; Data protection strategy: 3-replica strategy, mainly considering providing the lowest read latency (1-5ms); supporting high-concurrency random read and write; excellent write performance, no coding calculation required; fast data reading and recovery; Cost-effectiveness: Storage efficiency 33%, highest cost; In-state behavior: Data objects enjoy optimal performance in this state, and the system will try its best to maintain the data layout in this state.

[0038] 2) For the temperature state: Status indicators can be marked with yellow or neutral colors to represent moderate activity. Access characteristics include: Access frequency: 1-5 ops / min; Recent visits: Regular visits within the past day; Access mode: Primarily sequential read / write, with acceptable moderate latency; Data protection strategy: erasure coding strategy (k=4, m=2), mainly considering high storage efficiency: 4 / (4+2)=66.7%; read latency is 8-15ms (acceptable range); fault tolerance: can tolerate the loss of any 2 coded blocks; Cost-effectiveness: A good balance between performance and cost.

[0039] Parameter meaning: 4 data blocks, 2 parity blocks.

[0040] State-based behavior: The system monitors the access mode and is ready to migrate to a hot or cold state at any time.

[0041] 3) For the cold state: Status indicator: Blue or cool colors indicate the lowest activity level.

[0042] Access characteristics include: Access frequency: less than or equal to 1 op / day; Recent visits: Very few visits in the past week; Access mode: Primarily sequential reading, latency-insensitive; Data protection strategy: erasure coding strategy (K=8, M=3), mainly considering storage efficiency: 8 / (8+3)=72.7%; read latency is 12-25ms (acceptable range), and fault tolerance is tolerating the loss of any 3 coded blocks; Cost-effectiveness: Maximize storage utilization; In-state behavior: The system performs periodic checks, and may trigger archiving or deletion when the retention period is reached.

[0043] 4) For the initial state: As the starting state when an object is created, the newly created object can be labeled as "hot state"; the system begins to monitor its access patterns; the state is adjusted according to actual access behavior. This process can provide flexible initialization strategies and can preset the initial state according to the business type.

[0044] 5) For the final state: As the end of an object's lifecycle, the conditions that trigger the final state include at least one of the following: the data object reaches a predefined retention period; the data object is explicitly or permanently deleted, requiring the release of storage space; the data object is deeply archived, i.e. migrated to a lower-cost storage medium (such as tape or cloud archiving).

[0045] 6) For state transitions, including: 6.1) Warming migration (upward migration): Path 1: From warm state to hot state: Migration criteria: Access frequency exceeds 5 ops / min.

[0046] Triggering scenarios include periodic business peaks (such as e-commerce promotions), the initiation of data analysis tasks, and batch data processing.

[0047] System response: The heat monitoring module detected an increase in frequency; the decision engine generated a policy conversion task from erasure coding to multi-replica policy; the conversion executor completed the data protection policy conversion in the background; Performance impact: Read latency reduced by 60-80%, write performance improved by 2-3 times.

[0048] Path 2: From cold state to warm state Migration criteria: Access frequency exceeds 1 op / day.

[0049] Triggering scenarios include historical data queries, compliance audit requirements, and annual report generation.

[0050] System response: The status label is updated from "cold" to "warm"; the conversion from erasure coding policy (k=8, m=3) to erasure coding policy (k=4, m=2) is triggered; Performance impact: Storage efficiency decreases slightly, but performance improves.

[0051] 6.2) Cooling migration (downward migration): Path 3: From thermal state to warm state Migration conditions: Access frequency not exceeding 5 ops / min.

[0052] Triggering scenarios include the end of promotional activities, the transformation of hot data into ordinary data, and the passing of peak business periods.

[0053] System response: The popularity score drops below the threshold; the decision engine generates a strategy conversion task from a multi-replica strategy to an erasure coding strategy (4+2). Performance impact: Storage efficiency improved from 33% to 66.7%, and storage costs decreased by approximately 50%.

[0054] Path 4: From warm state to cold state Migration conditions: Access frequency does not exceed 1 op / day.

[0055] Triggering scenarios include data exceeding its active period, changes in business needs, and entering the archiving preparation stage.

[0056] System response: The status label is updated to "cold"; the transition from erasure coding policy (K=4, M=2) to erasure coding policy (K=8, M=3) is triggered; Performance impact: Storage efficiency improved from 66.7% to 72.7%, and storage costs decreased by approximately 9%.

[0057] In one embodiment, the current data protection policy is a multi-copy policy, and the corresponding target data protection policy is an erasure coding policy; converting the data object to be protected from the current data protection policy to the corresponding target data protection policy includes: S1310: Read all copies of the data object to be protected; S1320. Encode all copy data using erasure coding to obtain the first coded block; S1330: Write the first encoded block to a new storage location and delete all copy data.

[0058] Figure 3 This diagram illustrates a conversion from a multi-replica strategy to an erasure coding strategy, as provided in one embodiment. The diagram uses a linear time sequence to depict the complete technical process of achieving a background, transparent, and atomic strategy conversion while ensuring data consistency and service availability. The triggering conditions for the conversion from a multi-replica strategy to an erasure coding strategy include: the decision engine generating a specific strategy conversion task, with the corresponding instruction being to convert a specific data object from a multi-replica strategy to an erasure coding strategy with specified parameters. Figure 3 As shown, the process of converting from a multi-replica strategy to an erasure coding strategy includes the following stages: During the preparation phase, the conversion executor can retrieve policy conversion tasks from the task queue, obtain and parse the task information, which includes the data object ID (used to uniquely identify the data object to be protected) and the erasure coding parameters (K and M) of the target data protection policy. It also locks the metadata of the data object to prevent other processes (such as another conversion task, deletion operations, or the metadata manager) from concurrently modifying the object's metadata or data during the conversion, thus avoiding inconsistencies in the data object's state. Optionally, a distributed lock (based on the object's key) can be used, with lock granularity down to the individual object level. This fine-grained locking minimizes performance impact and allows the conversion and access of other objects within the cluster to proceed in parallel. During the data reading and encoding phase, the OSD nodes storing all N replicas of the data object are located. Complete object data is read from these OSDs in parallel or sequentially. Data integrity verification is then performed on the read object data, such as verifying the validity and consistency of the replica data read from each OSD, and confirming that the read replica data has the same epoch. If verification fails, such as due to a corrupted or unreadable replica, the object data can be retrieved from other healthy replicas or through Ceph's self-healing mechanism to ensure fault tolerance in the conversion process. This means that even if some storage nodes fail, the conversion process can continue as long as the data itself is recoverable (meeting the replica policy's survival requirements). If verification passes, the selected erasure coding algorithm, such as Reed-Solomon or locally repairable coding, can be used. The erasure coding algorithm encodes the complete object data into K+M independent coding blocks (i.e., the first coding block, which includes K data blocks and M check blocks). The size of each coding block is approximately 1 / K of the original data, and then the data writing stage begins. It should be noted that erasure coding is a CPU-intensive step, which can be avoided from impacting the front-end business by executing it asynchronously in the background and by adjusting the concurrency control. During the data writing phase, according to the CRUSH rules, an independent OSD (selecting K+M OSDs in total) is chosen as the new storage location for each newly generated coded block (K+M in total) to ensure that each coded block is distributed in different fault domains (such as different hosts or racks) and to meet the fault tolerance requirements of the EC policy. Then, K+M coded blocks are written to the K+M target OSDs in parallel. The parallel writing method makes full use of the network and disk I / O bandwidth of the distributed storage and shortens the conversion time window. Then, the success of the write is verified, that is, whether each coded block has been successfully persisted to the target OSD, which also verifies the integrity of the written EC data. If the write is successful, the cleanup and commit phase begins; otherwise, the data can be rewritten or marked as failed and reported to the decision engine for further decision-making. During the cleanup and commit phase, the metadata of the data object is updated using atomic operations. The updates include: Protection policy field: changed from "replicated size=N" to "erasure k=K m=M"; Data distribution mapping: pointing to the location information of the newly written K+M coded blocks; Version number: Incrementing the object version to identify the change in data protection policy; Based on this, all subsequent new client read requests will read and decode data from the EC-encoded block according to the new metadata.

[0059] Furthermore, after the metadata is successfully updated and the new data layout takes effect, the old copy data is deleted. This can be achieved by sending asynchronous deletion commands to the N original OSDs storing the old copy, thus cleaning up the old data. Deleting the old copy data only after confirming that the data update is successful and correct ensures that a complete and usable copy of the data is always available. Asynchronous deletion does not block the main conversion process or affect the access performance of the new data layout. It also has a certain degree of fault tolerance; even if the deletion operation temporarily fails, a background garbage collection mechanism will eventually handle it, without affecting the system's correctness, thereby effectively improving the reliability of the conversion process.

[0060] Furthermore, the object lock is released, allowing the data object to accept other management operations or new transformation tasks.

[0061] Based on this, the data protection strategy transformation is complete. The data object has been completely converted from a multi-replica format to an erasure coding format, and the storage overhead has been reduced from N times to (K+M) / K times (e.g., from 3 times to 1.5 times). Subsequent read and write operations on this data object will follow the semantics of the EC pool.

[0062] Furthermore, error handling and rollback mechanisms are provided. For example, if the writing of some encoded blocks fails due to target OSD failure or network timeout, the result of the measurement policy conversion task is marked as a partial failure. It is then determined whether this failure is a critical failure. If the number of successfully written blocks is insufficient to form a complete EC stripe, it is a critical failure, which will prevent new data from forming a valid EC set. In this case, the triggering condition for a full rollback is met: an unrecoverable error occurs during the data writing phase or before commit, and a rollback operation is performed. The rollback operation includes: releasing the object lock, i.e., restoring the data object to an accessible state; cleaning up intermediate data, i.e., deleting any partially written encoded blocks; notifying the decision engine, such as reporting the reason for the conversion failure (e.g., "target OSD unavailable"); and task handling, such as the policy conversion task being discarded or re-added to the queue after a certain delay for retry. Based on this, it can be ensured that when the conversion fails, the object is completely restored to its original multi-replica state, without affecting business continuity.

[0063] The above-described strategy conversion process has the following advantages: it follows a two-phase model of preparation and commit, ensuring that the conversion is atomic; before the metadata is updated, the client always accesses the old copy, and after the update, it immediately accesses the new EC data, with no service interruption throughout the process; data reading and writing are both done in parallel, maximizing the use of cluster I / O capabilities; EC encoding calculations are initiated by the conversion executor and can be scheduled to dedicated or idle computing nodes for execution, avoiding impact on the OSD master data path; old data deletion is asynchronous, without blocking critical paths; it includes integrity verification after data reading and write completion verification; it defines a multi-level error handling path from retryable errors to complete rollback; and each step has a clear state, facilitating monitoring and debugging.

[0064] In one embodiment, the current data protection policy is an erasure coding policy, and the corresponding target data protection policy is a multi-copy policy; converting the data object to be protected from the current data protection policy to the corresponding target data protection policy includes: S1340: Read a specified number of second encoded blocks of the data object to be protected; S1350. Decode the second encoded block to obtain the decoded data; S1360: Write the decoded data to a new storage location in the form of multiple copies and delete the second encoded block.

[0065] Figure 4 This is a schematic diagram illustrating the conversion from erasure coding policy to multi-replica policy in one embodiment. In this embodiment, EC data with high storage efficiency but low access performance can be converted back to multi-replica data with high-performance access, thereby achieving bidirectional and symmetrical policy conversion and further ensuring the integrity of the dynamic data protection policy system. In this embodiment, bidirectional intelligent scheduling can be performed between cost and performance based on changes in data popularity. The triggering conditions for converting from erasure coding policy to multi-replica policy include: the decision engine determines that a certain EC data object needs to be converted to a multi-replica policy (due to increased access frequency), generates a clear policy conversion task, and the corresponding instruction is to convert the specific data object from erasure coding policy to multi-replica policy with specified parameters. like Figure 4 As shown, the process of converting from erasure coding to a multi-replica strategy includes the following stages: During the preparation phase, the conversion executor can retrieve policy conversion tasks from the task queue, obtain the task information of the policy conversion tasks, and parse them. The task information includes the data object ID (used to uniquely identify the data object to be protected), the parameters of the target data protection policy (the number of replicas N, for example, N=3), and the current EC parameters (K and M, used to guide the reading and decoding of data, determining how many encoded blocks need to be read for successful decoding). It also locks the metadata of the data object, which can use a distributed lock (based on the object's key) to ensure that the access and state changes of the data object are carried out in an orderly manner during the conversion. During the data reading and encoding phase, based on the data object ID and the CRUSH rules of the EC pool, the OSD nodes storing all K+M encoded blocks of the data object are located, connections are established to these OSDs, and parallel reading is prepared. EC encoded blocks are typically distributed across different fault domains and need to be read from multiple OSDs. Then, an attempt is made to read at least K EC blocks. If any K usable encoded blocks (data blocks or parity blocks) can be read, the original data can be recovered. The reading strategy can be: prioritize reading from the OSDs containing the K data blocks; if the data blocks are unavailable, then read the parity blocks instead. Parallel reading can be used, i.e., simultaneously initiating read requests to multiple OSDs to improve efficiency. Then, it is checked whether at least K encoded blocks have been successfully read, i.e., whether K encoded blocks have been successfully read and the data is complete. If the reading is successful, EC decoding is performed; otherwise, the extended reading process begins. The extended read process includes: when it is impossible to read enough blocks from the preferred OSD, it can try to read from other available OSDs, up to a maximum of all K+M coded blocks; if a sufficient number of blocks are successfully read, EC decoding is performed, which fully utilizes the redundancy of EC, as long as no more than M coded blocks fail, the data can still be recovered; if it is impossible to read enough valid coded blocks (more than M failed blocks), Ceph's data repair process can be triggered to attempt to repair the damaged coded blocks; if the repair fails, the transformation task is marked as failed, thus ensuring that policy transformation is only performed when the data itself is recoverable; The EC decoding process includes: using the same erasure coding algorithm as during encoding, decoding the K read coded blocks to recover the complete original object data. The principle is to solve a system of linear equations to obtain K unknowns (original data blocks) from the K equations (coded blocks). The decoding computation is usually less than that of encoding, but still requires certain CPU resources. During the data writing phase, storage locations are selected for N replicas according to the CRUSH rules configured for the multi-replica strategy, i.e., N target OSDs are selected. Optionally, the N replicas are distributed across different fault domains; optionally, OSDs with lower loads are selected first. Then, the N replicas of the decoded complete object data are written to the N target OSDs to speed up the conversion. Each write operation contains complete object data and metadata. During this process, it must be ensured that all replicas are written with the same data content. It is confirmed whether all N replicas have been successfully written. If all replicas have been successfully written, the commit phase is initiated; otherwise, the error handling process is initiated. During the commit phase, after confirming that all new copies have been successfully written, the metadata of the data objects is updated atomically. The updates include: Protection policy field: changed from "erasure k=K m=M" to "replicated size=N"; Data distribution mapping: pointing to the location information of the N newly written replicas; Access flag: The marked object can now be quickly accessed via copies; Based on this, all subsequent read requests will read directly from the nearest replica without requiring decoding calculations.

[0066] Furthermore, update the extended attributes to record additional information, such as: conversion timestamp; EC parameters before conversion; and engine identifier used to perform the conversion. This information can be used for monitoring, auditing, and troubleshooting.

[0067] Furthermore, asynchronous deletion of old EC blocks can be achieved by sending asynchronous deletion commands to the K+M OSDs storing the old EC blocks. Based on this, it can be ensured that the old data is cleaned up only after the new replicas are fully available. Even if the deletion fails, it will not affect the availability of data, as there is background garbage collection as a fallback. After the EC block is deleted, a certain amount of storage space can be freed up, but the increase in the number of replicas will occupy new space.

[0068] Furthermore, releasing the object lock allows the object to accept new access and management operations.

[0069] Furthermore, the accessibility of the new copy can be verified: actively read the new copy to verify successful conversion.

[0070] Based on this, the strategy conversion is completed, and the data objects are completely converted from EC form to multi-replica form, resulting in a significant performance improvement.

[0071] Furthermore, error handling and rollback mechanisms are provided. If some writes fail among the N replicas, a limited number of retries can be performed on the failed replicas. For example, it can be determined whether a write failure is a critical failure. If the number of successful writes does not meet the minimum availability requirements, it is considered a critical failure and a rollback operation can be triggered. The rollback operation includes: marking the conversion as failed and entering the rollback process; releasing the object lock and restoring the object's normal access state; notifying the decision engine of the conversion failure and its cause; and the task entering a waiting-for-retry or manual intervention state. During this process, it is necessary to ensure that the object is restored to its original EC state and that all intermediate state data is cleaned up.

[0072] The above strategy conversion process has the following advantages: redundant reading and fault-tolerant design, meaning recovery can be achieved with any K encoded blocks; fully utilizing the mathematical characteristics of EC, not relying on specific blocks; progressive reading strategy, prioritizing data blocks and replacing them with check blocks in case of failure; maximum redundancy utilization, attempting to read at most K+M encoded blocks to maximize the success probability; decoded data can be written to all replicas in parallel, resulting in high efficiency; decoding calculations can be performed on dedicated computing nodes, avoiding impact on storage nodes; EC block reading may involve multiple nodes, but writing to replicas after decoding can be optimized to prioritize the nearest node; in addition, the above strategy conversion process follows a two-phase mode of preparation and commit phases, ensuring that the conversion is atomic and the entire process is uninterrupted; only a single object is locked, without affecting access to other objects; metadata updates take effect immediately, without the client's awareness; storage space usage changes from (K+M) / K times to N times; read latency is reduced from 10-50ms for EC decoding to 1-5ms for direct replica reading; one-time decoding overhead is exchanged for continuous read performance improvement.

[0073] In one embodiment, the process of converting the data object to be protected from the current data protection policy to the corresponding target data protection policy further includes: If a read request for a protected data object is received, the protected data object will be read according to the current data protection policy. If a write request is received for a data object to be protected, the data object to be protected is temporarily locked, and the data to be written to the data object to be protected is merged with the data obtained by converting it into the corresponding target data protection policy and written to a new storage location.

[0074] In this embodiment, the data object to be protected can still be accessed during the data protection policy conversion process. For read requests, the old data (current data protection policy) layout can provide services. For example, if the current data protection policy is a multi-replica policy, a copy of the data object can be read; if the current data protection policy is an erasure coding policy, K coded blocks can be read and the data object can be restored. For write requests, the data object can be temporarily locked, its write log recorded, and after the data protection policy conversion is completed, it can be merged with the data of the target data protection policy and written, thereby ensuring data consistency and achieving a smooth data conversion.

[0075] Figure 5 This is a schematic diagram illustrating a heat-based data protection process as provided in one embodiment. For example... Figure 5 As shown, the data protection process based on heat includes five stages, intuitively demonstrating the full lifecycle management of dynamic data protection strategies from strategy configuration, data monitoring, intelligent decision-making to strategy execution and feedback optimization.

[0076] I. Policy Configuration Phase: During system startup or re-initialization, dynamic protection policies are configured for the storage pool. For example, administrators define dynamic protection policies for the storage pool through the management interface or application programming interface (API). Key inputs include: storage pool identifier, business characteristics, performance requirements, and cost constraints; the output is a preliminary policy rule framework. Define a popularity threshold, which is to clearly define the quantitative standard for the popularity of data objects. It is usually divided into: hot status: data accessed frequently; warm status: data accessed at a medium frequency; cold status: data accessed infrequently or archived. For example, popularity levels can be divided by setting access frequency thresholds (such as 5 ops / min) and / or time windows (such as the past 24 hours). Define a protection policy mapping, that is, establish a correspondence between hot data states and target data protection policies. For example: the target data protection policy for hot states is a multi-replica policy (high performance); the target data protection policy for warm states is an erasure coding policy (EC) (k=4, m=2) (balancing performance and cost); and the target data protection policy for cold states is an erasure coding policy (K=8, M=3) (high storage efficiency). Flexible and customizable mapping relationships can be supported based on this.

[0077] II. Monitoring and Computation Phase: OSD records data object access operations, that is, intercepts and records the read and write operations of each data object at the OSD level, such as recording the following information: operation type (read / write); operation timestamp; operator identifier; data size; this can be achieved through Ceph OSD plugins or hook mechanisms.

[0078] For example, the popularity analysis module can periodically (e.g., every minute) collect statistical access operations, such as pulling from or receiving access statistics pushed by various OSDs; it can aggregate distributed statistical information into a global view; and it can use sliding windows and incremental calculations to reduce resource consumption. The access intensity (e.g., access frequency) of data objects is calculated based on the statistical information, which can be done using a popularity calculation model; predefined time windows (e.g., the past 1 hour, 24 hours) can be set; the calculation content can include: basic access frequency, and the final output is a comprehensive popularity score for each data object; the popularity level can be determined by comparing the calculated popularity score with a predefined threshold, and popularity tags can be added. Tag types include: HOT; WARM; COLD, and the popularity tags are written to the data object metadata or a dedicated metadata database.

[0079] III. Decision-Making and Task Generation Phase: The decision engine can periodically (e.g., every 5 minutes) check the popularity tags, such as scanning the latest popularity tags of all data objects in the storage pool, choosing between a full scan or incremental scan. It compares the current data protection policy of the data object with the configured target data protection policy for the corresponding popularity level to determine if a protection policy conversion is needed: if the current data protection policy matches the target data protection policy, the policy conversion process can be skipped, returning to the monitoring phase; if the current data protection policy does not match the target data protection policy, a policy conversion task is generated. The trigger condition for generating a policy conversion task is: the current policy and the target policy are inconsistent. Task information includes: data object identifier; current data protection policy; target data protection policy; conversion priority; execution constraints. The generated policy conversion task is placed in the task queue, awaiting execution.

[0080] IV. Transformation Execution Phase: One or more transformation executors retrieve policy transformation tasks from the task queue and can schedule tasks based on factors such as priority and resource availability. Furthermore, the number of concurrently executed transformation tasks can be controlled to avoid resource conflicts. Policy transformation tasks are executed, and transformation types include: transformation from a multi-replica policy to an erasure coding policy, transformation from an erasure coding policy to a multi-replica policy, and parameter adjustment of the erasure coding policy. Optimized transformation algorithms can be used to reduce data movement and computational overhead. It should be noted that data objects remain accessible during policy transformation; that is, the transformation process is transparent to the client. Corresponding access handling includes: for read requests, services are provided by the old data layout; for write requests, copy-on-write or logging mechanisms are used to ensure data consistency. Fine-grained locking or concurrency control mechanisms can be used during this process.

[0081] V. Metadata Update Phase: Update object metadata, including: protection policy field (updating from the old policy to the new policy); conversion timestamp; conversion version number. Based on this, ensure the atomicity of metadata updates and data conversion.

[0082] Clean the task queue, i.e., delete completed policy conversion tasks from the task queue; record the success or failure status of the conversion for monitoring and analysis. Subsequent access will use the new policy, which takes effect after the metadata update is complete. After it takes effect, access will be redirected, meaning that subsequent access to this object will be based on the new data protection policy. The performance immediately reflects the characteristics of the new policy after the conversion is complete.

[0083] Furthermore, the design incorporates a closed-loop process and continuous optimization, including: after each complete transformation, the process automatically returns to the monitoring phase, forming a continuous optimization loop of "monitoring-decision-execution-feedback". For failed strategy transformation tasks, retries and backoff strategies can be configured; for critical failures, alert messages can be generated to notify administrators.

[0084] For example, key decision points in the data protection policy transition process include: The main decision point is the strategy consistency judgment, which is to determine whether the current strategy is consistent with the target strategy. This is the core judgment that determines whether to trigger a switch. Implicit decision points include: The decision on the cycle of popularity calculation, i.e. when to trigger a new round of popularity calculation; Task prioritization decision, i.e. how to sort multiple strategy transformation tasks; Resource allocation decisions, i.e., how to allocate cluster resources when executing strategy conversion tasks; Exception handling decisions refer to the recovery strategy to be adopted when the conversion fails.

[0085] For example, the periodic characteristics of the above data protection policy conversion process are shown in Table 1. For example, the real-time requirements for the above policy conversion process are as follows: High real-time performance: Access statistics collection needs to be near real-time.

[0086] Medium real-time performance: Heat calculation and decision checking allow for a certain delay.

[0087] Low real-time performance: The conversion can be performed asynchronously in the background.

[0088] For example, the resource consumption distribution of the above strategy conversion process is as follows: Monitoring phase: Low CPU, medium memory, low I / O; Calculation phase: High CPU (computation-intensive), medium memory, low I / O; Decision-making phase: Low CPU, medium memory, low I / O; Execution phase: High CPU (encoding / decoding), high network, high I / O.

[0089] For example, the key configuration parameters for the above policy conversion process include: Popularity calculation parameters: time window, decay factor (a coefficient between 0 and 1 (e.g., 0.9), after each visit, the popularity will be multiplied by this factor), weighting coefficient (e.g., using the directly obtained number of visits as the basic weight; giving higher weight to recent visits, such as using an exponential decay function; combining user dwell time or click-through rate for weighting).

[0090] Decision parameters: switching hysteresis, capacity threshold, performance threshold.

[0091] Execution parameters: concurrency (the number of strategy conversion tasks executed in parallel), retry policy, and resource limits.

[0092] For example, the performance and capacity planning for the above strategy conversion process is as follows: Based on the resource consumption characteristics of each stage of the flowchart: Monitoring nodes require sufficient network bandwidth and storage I / O capabilities.

[0093] Compute nodes: require strong CPU computing power.

[0094] Decision node: Requires sufficient memory to store metadata.

[0095] Execution node: requires a balance of CPU, network, and I / O capabilities.

[0096] For example, the above strategy conversion process involves the following exception handling flow: Data collection failed: OSD node crashed or network partition; Calculation anomaly: The heat calculation algorithm is faulty or the data is faulty; Decision anomalies: rule conflicts or inconsistent metadata; Execution failed: Node failure or insufficient resources during the conversion process.

[0097] For example, the exception handling strategy is as follows: Retry strategy: Automatic retry for temporary failures.

[0098] Skip strategy: Exceptions that do not affect core functionality can be temporarily skipped.

[0099] Alarm strategy: Critical anomalies trigger administrator alerts.

[0100] Rollback strategy: If the conversion fails, it can be rolled back to the original state.

[0101] The above strategy conversion process has the following characteristics: Intelligent closed loop: From data collection to intelligent analysis to automatic decision-making to precise execution to feedback optimization, the system can continuously adapt to changes in business.

[0102] Fine-grained management: Each step operates on a single data object, offering the advantage of fine-grained management compared to traditional pool-level management.

[0103] Uninterrupted service: Data objects remain accessible during policy transitions, enabling seamless business transitions.

[0104] Flexible and configurable: The cycle and parameters of each stage can be configured to adapt to different business scenarios: such as financial transaction scenarios, which shorten the monitoring and decision-making cycle; and backup and archiving scenarios, which extend the monitoring cycle and focus on cost optimization.

[0105] The following is a specific example of a data protection policy conversion. Assume a Ceph storage pool called backup-pool used for backup archiving, where all data is initially stored with an erasure coding policy of (K=8, M=3).

[0106] Policy Configuration: Administrators configure the dynamic policies for this pool using command-line tools. ceph osd pool set backup-pool dynamic_tiering_rule \ '{"hot": {"type": "replicated", "size": 3}, “warm”: {“type”: “erasure”, “profile”: “k=4 m=2”}, "cold": {"type": "erasure", "profile": "k=8 m=3"}, "thresholds": {"hot_to_warm": "5 ops / min", "warm_to_cold": "1 op / day"}}' Monitoring and Hotness Calculation: The hotness analysis module collects the object access counts reported by the OSD every 30 seconds. Object A is marked as "hot" because it is frequently accessed, with an average access frequency of 10 ops / min over the past 5 minutes.

[0107] Decision: The decision engine finds that object-A is labeled as hot, but the current data protection policy is K=8, M=3 erasure coding policy, which does not match the 3-replica policy corresponding to hot in the rule, and generates a replica conversion task.

[0108] Execution Transformation: The transformation executor acquires the task. From the current 11 EC blocks of object-A, it randomly selects 8 blocks and decodes them to recover the complete object data. The complete data is then used as 3 replicas and written to 3 different OSDs according to the pool's CRUSH rules. After successful writing, the metadata of object-A is updated, marking its protection policy as replicated size=3. The old encoded blocks originally distributed across the 11 OSDs are asynchronously deleted. Afterward, applications reading object-A will directly read from one of the 3 replicas quickly, achieving lower latency. After a period of time, if its access frequency drops to the corresponding warm or cold range, the system will automatically convert it back to the corresponding erasure coding policy, freeing up storage space.

[0109] Figure 6This is a schematic diagram illustrating the implementation of a heat-based data protection system as an example. Figure 6 As shown, the system includes: Policy Management Module: Provides an interface for administrators to configure dynamic protection policy rules for storage pools.

[0110] Popularity Monitoring and Analysis Module: Deployed on each OSD and / or centralized node, responsible for collecting object access counts, calculating and updating object popularity tags. The Popularity Monitoring and Analysis Module is deployed on Ceph OSD nodes to collect object access counts; the decision engine and policy conversion executor can be integrated into Ceph Monitor or run as a standalone service.

[0111] Decision engine: Based on popularity tags and policy rules, it determines whether conversion is needed and generates conversion tasks.

[0112] Transformation executor: Retrieves tasks from the task queue and performs data recoding / rewriting operations securely and efficiently in the background, completing the switching of protection strategies.

[0113] Metadata service: Used to store extended information such as the current protection policy and popularity tags of objects.

[0114] For example, in a heat-based data protection system, the application layer components can be applications or clients, representing upper-layer business applications that use storage services. They perform data read and write operations through standard Ceph client interfaces (such as RADOS, RBD, RGW); they can directly interact with the Ceph core layer Monitor and OSD to initiate normal storage requests, and are completely unaware of the dynamic policy conversion process of this invention.

[0115] The policy management module, located on the left side of the architecture, provides an administrator configuration interface or API for defining dynamic protection policy rules, including: thresholds for hot status division (such as access frequency standards for hot / warm / cold); target data protection policies corresponding to each hot status (such as 3 replicas for hot status, erasure coding (K=4, M=2) for warm status, and erasure coding (K=8, M=3) for cold status); advanced settings such as conversion hysteresis parameters and execution time windows; the data flow is: writing the configured policy rules to persistent storage in the metadata service.

[0116] The popularity monitoring and analysis module, located in the middle of the architecture, connects to the OSD cluster and the metadata service. It is used to collect access counts (read / write frequency and timestamps) of objects from each Ceph OSD in real time; analyze object access patterns using popularity calculation formulas; calculate a comprehensive popularity score for each object and label it with popularity tags (hot / warm / cold); its input is the raw access statistics collected from the OSD cluster, and its output is to write the calculated popularity tags to the metadata service.

[0117] The decision engine, located at the core of the architecture, connects the metadata service and the transformation executor. It can periodically scan the metadata service to read the current popularity tags and stored data protection policies of data objects; compare the current data protection policies of data objects with the configured target data protection policies to determine whether transformation is needed; if so, it generates specific policy transformation tasks, the input of which is the policy rules and popularity tags read from the metadata service; the output is to issue transformation tasks to the policy transformation executor.

[0118] The transformation executor connects the decision engine and the OSD cluster. It receives transformation tasks from the decision engine and asynchronously executes the actual transformation operations of the protection policy in the background. Examples include: transformation from a multi-replica to an erasure coding policy, which involves reading replica data, performing erasure coding, and writing it to a new location; transformation from an erasure coding to a multi-replica policy, which involves decoding erasure coding blocks and writing them to a new location as replicas; adjustment of erasure coding parameters, including changing the K / M parameters of the erasure coding policy; and ensuring the atomicity and data consistency of the transformation process. Its input is the policy transformation task received from the decision engine; its output is the data read / write and re-encoding operations performed on the OSD cluster, and the update of the new protection policy for the object to the metadata service.

[0119] The metadata service, located in the middle of the architecture, serves as the data exchange center for various modules. It can be used to store the core metadata of the system, including: dynamic data protection policy configuration rules; real-time heat tags and historical trends of each data object; current and historical data protection policies of each data object; transformation task status and result records; provide a highly available and consistent metadata access interface; and enable the data to be persisted to the Metadata Server (MDS) of the Ceph core layer.

[0120] The Ceph core layer (basic platform) includes Ceph Monitor, Metadata Server, and Ceph OSD cluster. CephMonitor is the management node for the Ceph cluster, maintaining the cluster state map (including OSD Map, Monitor Map, PG Map, etc.). It receives routine management requests from the application layer, monitors the overall health of the cluster, and provides cluster-level context information (such as load and capacity utilization) for intelligent decision-making. The Metadata Server is a dedicated service for storing file system metadata or object extended attributes, providing underlying persistent storage capabilities for metadata services and ensuring highly reliable storage of metadata such as popularity tags and policy rules. The Ceph OSD cluster includes multiple OSD (Object Storage Daemon) nodes, which are used to actually store user data objects; handle core I / O operations such as data read / write / copy / recovery; and perform erasure coding and decoding calculations. As a data source, it provides object access statistics to the popularity monitoring and analysis module; its execution goal is to receive instructions from the policy transformation executor and perform data rearrangement operations; as a service provider, it continues to provide standard data storage services to the application layer.

[0121] In this system, the policy configuration flow is as follows: from the administrator to the policy management module to the metadata service. That is, the administrator defines rules through the policy management module, and these rules are persisted to the metadata service for use by the decision engine.

[0122] The monitoring and calculation flow is as follows: OSD cluster to heat monitoring and analysis module to metadata service. That is to say, OSD records object access in real time, the monitoring module collects and calculates the heat, and the results are written to the metadata service.

[0123] The intelligent decision-making process is as follows: from metadata service to decision engine to strategy conversion executor. That is, the decision engine reads metadata and popularity information, makes intelligent judgments, generates conversion tasks, and sends them to the conversion executor.

[0124] The policy execution flow is as follows: policy transformation executor to OSD cluster to metadata service, that is, the transformation executor performs the actual data transformation on the OSD cluster, and updates the policy information in the metadata after success.

[0125] The application access flow is as follows: the application requests normal read and write requests to the Ceph Monitor / OSD cluster, i.e., the upper-layer application, fully comply with the Ceph standard protocol and are not affected by the dynamic strategy of this invention.

[0126] The underlying storage flow includes: metadata service to metadata server, Ceph Monitor to OSD cluster, demonstrating the standard data management flow within Ceph while maintaining compatibility with the native architecture.

[0127] For example, the following deployment methods can be adopted: Integrated Deployment: Integrate the new module into the Ceph Monitor or MDS process; Standalone service deployment: Runs as a standalone service, interacting with Ceph through a plugin mechanism; Hybrid deployment: Some modules are integrated (such as heat collection integrated into OSD), while others are independent (such as decision engine).

[0128] For example, multiple instances of the heat monitoring module and the conversion executor can be deployed to handle larger-scale clusters; each module can flexibly replace algorithms or strategies through interface definitions.

[0129] For example, the metadata service requires a highly available design, which can employ master-slave replication or a distributed consensus protocol; the decision engine can be deployed in an active standby mode to avoid single points of failure.

[0130] The heat-based data protection method in this application extends Ceph. The metadata for RADOS objects is updated to include data protection scheme (protection_scheme) and access heat (access_heat) fields; the ceph-osd process is modified to add access interception and counting hooks, and the counts are periodically reported to the centralized analysis module; a lightweight, distributed transformation executor is implemented, capable of utilizing idle cluster resources for data recoding and migration; atomicity and consistency during transformation are ensured, for example, by using two-phase commit or rollback mechanisms; fine-grained automation: intelligent management is performed at the data object level, automatically adapting to changes in access patterns, greatly reducing manual management costs; adaptive performance and cost optimization: high-performance replica strategies are maintained for popular data, while cold data is automatically converted to high storage utilization EC strategies, maximizing storage cost reduction while ensuring performance; low-overhead transformation: strategy transformation is performed in the background within the pool, avoiding large-scale data flow and complex cache eviction logic across pools, with less impact on front-end business; flexible strategies: bidirectional, progressive transformation between different data protection strategies is supported, allowing for more flexible strategy combinations; transparent compatibility: completely transparent to upper-layer applications and clients, requiring no modification to application code, and compatible with existing Ceph client protocols.

[0131] Figure 7 A schematic diagram of a heat-based data protection device provided in this application embodiment. The heat-based data protection device provided in this embodiment includes: Monitoring module 210 is used to monitor the access intensity of each data object in the storage pool; The decision module 220 is used to determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level based on the access intensity of each data object. The protection module 230 is used to convert the data object to be protected from the current data protection policy to the corresponding target data protection policy, wherein the data object to be protected is a data object whose current data protection policy and the corresponding target data protection policy are inconsistent.

[0132] This device dynamically adjusts the data protection strategy for data objects based on access intensity, achieving fine-grained protection of data at the data object level. The adjustment of the data protection strategy does not require cross-pool migration, resulting in low overhead. Furthermore, by using access intensity as the basis for adjusting the data protection strategy, it can effectively balance the storage and performance of data with different access intensities, achieving automatic and efficient dynamic data protection.

[0133] Based on any of the above embodiments, the decision module 220 is specifically used to: determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level according to the access intensity and access intensity threshold of each data object.

[0134] The heat level of each data object includes at least one of the following: hot level, warm level, and cold level.

[0135] The target data protection strategy corresponding to the hot level is a multi-copy strategy, while the target data protection strategies corresponding to the warm level and the cold level are erasure coding strategies with different parameters.

[0136] Based on any of the above embodiments, the current data protection strategy is a multi-copy strategy, and the corresponding target data protection strategy is an erasure coding strategy; the protection module 230 is specifically used for: reading all copy data of the data object to be protected; performing erasure coding on all copy data to obtain a first coding block; writing the first coding block to a new storage location and deleting all copy data.

[0137] Based on any of the above embodiments, the current data protection strategy is an erasure coding strategy, and the corresponding target data protection strategy is a multi-copy strategy; the protection module 230 is specifically used to: read a specified number of second coding blocks of the data object to be protected; decode the second coding blocks to obtain decoded data; write the decoded data into a new storage location in the form of multiple copies and delete the second coding blocks.

[0138] Based on any of the above embodiments, the device further includes: a read / write module, configured to, during the process of converting the data object to be protected from the current data protection policy to the corresponding target data protection policy, if a read request for the data object to be protected is received, read the data object to be protected according to the current data protection policy; if a write request for the data object to be protected is received, temporarily lock the data object to be protected, and merge the data to be written from the data object to be protected with the data obtained after converting to the corresponding target data protection policy and write it to the new storage location.

[0139] Based on any of the above embodiments, the device further includes: an update module, used to update the metadata of the data object to be protected after converting the data object to be protected from the current data protection policy to the corresponding target data protection policy.

[0140] The heat-based data protection device provided in this application embodiment can be used to execute the heat-based data protection method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0141] Figure 8 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 10 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, user equipment, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0142] like Figure 8 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0143] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks and wireless networks.

[0144] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above.

[0145] In some embodiments, the methods described above can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the methods described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the methods of any of the embodiments described above by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device 10, which includes: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device 10. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0151] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0152] This application also provides a computer program product, including a computer program and / or instructions, which, when executed by a processor, implement the heat-based data protection method as described in any of the above embodiments.

[0153] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0154] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A data protection method for storage pools based on heat intensity, characterized in that, include: Monitor the access intensity of each data object within the storage pool; Based on the access intensity of each data object, determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level; The data object to be protected is converted from the current data protection policy to the corresponding target data protection policy, wherein the data object to be protected is a data object whose current data protection policy and the corresponding target data protection policy are inconsistent.

2. The method according to claim 1, characterized in that, Based on the access intensity of each data object, determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level, including: Based on the access intensity and access intensity threshold of each data object, determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level; The popularity level of each data object includes at least one of the following: hot level, warm level, and cold level; The target data protection strategy corresponding to the hot level is a multi-copy strategy, while the target data protection strategies corresponding to the warm level and the cold level are erasure coding strategies with different parameters.

3. The method according to claim 1, characterized in that, The current data protection strategy is a multi-replica strategy, and the corresponding target data protection strategy is an erasure coding strategy. The step of converting the data object to be protected from the current data protection policy to the corresponding target data protection policy includes: Read all copies of the data object to be protected; The entire copy of the data is erasure encoded to obtain the first encoded block; Write the first encoded block to a new storage location and delete all the copy data.

4. The method according to claim 1, characterized in that, The current data protection strategy is erasure coding, and the corresponding target data protection strategy is a multi-copy strategy. The step of converting the data object to be protected from the current data protection policy to the corresponding target data protection policy includes: Read a specified number of second encoded blocks from the data object to be protected; Decoding the second encoded block yields the decoded data; The decoded data is written to a new storage location in multiple copies and the second encoded block is deleted.

5. The method according to claim 3 or 4, characterized in that, The process of converting the data object to be protected from the current data protection policy to the corresponding target data protection policy also includes: If a read request for the data object to be protected is received, the data object to be protected is read according to the current data protection policy; If a write request for the data object to be protected is received, the data object to be protected is temporarily locked, and the data to be written to the data object to be protected is merged with the data obtained by converting it into the corresponding target data protection policy and written to the new storage location.

6. The method according to any one of claims 1-4, characterized in that, After converting the data object to be protected from the current data protection policy to the corresponding target data protection policy, the following steps are also included: Update the metadata of the data object to be protected.

7. A data protection device based on heat, characterized in that, include: The monitoring module is used to monitor the access intensity of each data object in the storage pool; The decision module is used to determine the popularity level of each data object and the target data protection strategy corresponding to the popularity level based on the access intensity of each data object. The protection module is used to convert the data object to be protected from the current data protection policy to the corresponding target data protection policy, wherein the data object to be protected is a data object whose current data protection policy and the corresponding target data protection policy are inconsistent.

8. An electronic device, characterized in that, include: At least one processor; A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the heat-based data protection method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the heat-based data protection method as described in any one of claims 1-6.

10. A computer program product comprising a computer program and / or instructions, characterized in that, When the computer program and / or instructions are executed by the processor, they implement the heat-based data protection method as described in any one of claims 1-6.

Images