Intelligent EC processing method and device

Through intelligent EC processing methods and devices, the EC ratio is dynamically adjusted, solving the problem of traditional EC storage efficiency not being able to improve after capacity expansion, and achieving optimization of storage efficiency and cost.

CN117827097BActive Publication Date: 2025-09-05CHINA TELECOM CLOUD TECH CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202311708891.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-12-13
Publication Date
2025-09-05
Estimated Expiration
2043-12-13

AI Technical Summary

Technical Problem

In existing technologies, as the ratio of data disks to redundant disks increases, storage efficiency cannot be improved. The storage efficiency of traditional EC cannot be improved after capacity expansion, which affects the storage cost per TB.

Method used

Adopting intelligent EC processing methods and devices, through the Chunk allocation module, PG view management module, Pool storage resource pool, Disk list and Chunk Saver thread, it realizes the conversion from single-node EC to multi-node EC, dynamically adjusts the EC ratio, and improves storage reliability and availability.

Benefits of technology

It realizes online EC mode conversion, improves storage data efficiency, reduces storage cost per GB, and can flexibly adjust protection level after capacity expansion to adapt to different storage needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117827097B_ABST
    Figure CN117827097B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of computer storage technology, and specifically relates to an intelligent EC processing method and device, including a chunk allocation module, a PG view management module, a pool storage resource pool, a disk list, and a chunk saver thread. The present invention effectively avoids write operations on a database undergoing EC conversion through chunk space allocation management, thereby eliminating the need to worry about write data mutual exclusion issues and achieving online EC mode conversion. Through database-based protection mode management and control of the database read-write state machine, the database protection mode can be easily adjusted, and intelligent EC can be implemented to dynamically adjust the EC ratio. When a single node is expanded to multiple nodes, the protection level can be changed online from disk-level failures to node-level failures, or even rack-level and data center-level failures. When a small-scale node is expanded to multiple nodes, the protection level can be changed online, improving storage data efficiency and reducing the storage cost per GB.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer storage technology, and in particular relates to an intelligent EC processing method and device. Background Art

[0002] EC (Erasure Code) is a coding technology that can add m copies of data to n original copies. The original data can be restored from any n copies of the n+m copies. This means that if any of the m or fewer copies of data fail, the data can still be restored from the remaining data. EC is the mainstream data redundancy technology currently used in distributed storage, typically using a 4+2, 8+3, or even a larger ratio of 22+3.

[0003] In existing technologies, storage efficiency improves as the ratio of data disks to redundant disks increases. However, in distributed storage, the number of storage nodes is typically increased on demand. Traditional EC (Encrypted Content Cache) remains unchanged after creation or replication, preventing increased storage efficiency after expansion, impacting the cost per TB of storage. Therefore, this solution provides an intelligent EC processing method that can upgrade from single-node EC to multi-node EC, improving storage reliability and availability. Summary of the Invention

[0004] The purpose of the present invention is to provide an intelligent EC processing method and device, which can upgrade from single-node EC to multi-node EC and improve storage reliability and availability.

[0005] The technical solutions adopted by the present invention are as follows:

[0006] An intelligent EC processing device includes a Chunk allocation module, a PG view management module, a Pool storage resource pool, a Disk list, and a Chunk Saver thread;

[0007] The Chunk allocation module is used to allocate new data storage space to the storage system for additional writes and to write new user data;

[0008] The PG view management module is used to obtain the PG state, wherein the PG state includes a read-only state, a read-write state, and an uncertain state;

[0009] The Pool storage resource pool includes multiple PGs, and when the Pool storage resource pool is created, the total number of the PGs is synchronously determined;

[0010] The Disk list is used to list all PGs, where each PG corresponds to a unique number;

[0011] The Chunk Saver thread is used to process Disk information, and each Chunk Saver thread is responsible for processing one or more Disk information.

[0012] In a preferred solution, before the chunk allocation module is executed, it obtains the requirements of the upper-layer application, and when the upper-layer application needs to write data, it generates a chunk allocation request and sends it to the chunk allocation module, which allocates the newly added chunk space to the upper-layer application.

[0013] In a preferred solution, the specific process of the chunk allocation module allocating the newly added chunk space to the upper-layer application is as follows:

[0014] Obtain the PG status corresponding to the newly added Chunk space and the PG space occupancy;

[0015] Get the allowed allocation threshold and compare it with the PG space occupancy. When the PG state is read-write state, if the PG space occupancy is greater than the allowed allocation threshold, allocate Chunks from the PG to the upper-level application. Otherwise, continue to match the next PG or add PG.

[0016] In a preferred solution, when the Pool storage resource pool executes node expansion, it performs EC conversion, adds a new PG, and sets the protection mode of the new PG to the EC mode that needs to be converted. The Chunk allocation module applies for Chunks from the PG in the new EC mode to the upper-level application, and then organizes the PG data of the original PG. After the original PG is organized into the new EC mode, the Chunk allocation module continues to apply for Chunks from the original PG to the upper-level application.

[0017] In a preferred solution, the EC conversion includes node expansion and node contraction. The node expansion is from within the node to between nodes, and the EC ratio between nodes is gradually increased. The node contraction is to adjust the PG EC ratio.

[0018] In a preferred embodiment, the steps for arranging the PG data of the original PG are as follows:

[0019] Set the original PG to read-only mode, then adjust it to the new EC mode, and generate a PG reconstruction list;

[0020] Get a list of all chunks containing data in the original PG, read the complete chunk data, and then re-encode the read chunk data using the new PG EC protection mode. Write the re-encoded data to the reconstructed disk by EC sharding until all chunk data has been re-EC-encoded.

[0021] Update the disk list of the new PG to the reconstructed disk list, change the PG protection mode, and update the PG version information at the same time, and then change the status of the PG after the sorting is completed to read-write status.

[0022] In a preferred solution, after the original PG is set to read-only state, if there is a read request, EC decoding is performed from the original PG disk list according to the original PG EC protection mode, and the corresponding data in the Disk segment is read; otherwise, the original PG still performs EC mode conversion in read-only state.

[0023] In a preferred solution, after the PG data of the original PG is sorted out, the disks in the original Disk list are notified through the corresponding Chunk saver thread to reclaim the space occupied by the Chunk and delete the Chunk information in the Disk;

[0024] Among them, the client does not change the PG status information, and sends a read request to the disk in the original PG disk list to read data. After the disk in the original PG disk list receives the request;

[0025] If the Chunk data has been deleted, the Chunk saver thread corresponding to the Disk will return an error message, and the client will refresh the PG view to read the Chunk shard data from the new Disk.

[0026] If Disk does not receive the chunk deletion message, it indicates that the chunk exists and the corresponding data is returned.

[0027] In a preferred solution, the disks in the PG reconstruction list and the disks in the original PG disk list do not overlap with each other.

[0028] The present invention further provides an intelligent EC processing method, which is applied to the above-mentioned intelligent EC processing device, comprising:

[0029] Obtain the needs of the upper-layer application, and when the upper-layer application needs to write data, generate a Chunk allocation request, and then allocate new Chunk space from the PG in the read-write state according to the Chunk allocation request;

[0030] Get the expanded node, perform EC conversion, update the PG protection mode, change the original PG state to read-only, and continue to receive read requests;

[0031] After the original PG is organized into the new EC mode, the chunk allocation module continues to request chunks from the original PG for upper-layer applications.

[0032] Notify the disks in the original Disk list to reclaim the space occupied by the Chunk and delete the Chunk information in the Disk.

[0033] The technical effects achieved by the present invention are:

[0034] The present invention can effectively avoid writing operations to the database undergoing EC conversion through Chunk space allocation management, so there is no need to worry about the mutual exclusion problem of write data, and realize online EC mode conversion. Through database-based protection mode management and control of the database read and write state machine, the database protection mode can be easily adjusted, and intelligent EC can be implemented to dynamically adjust the EC ratio. When a single node is expanded to multiple nodes, the protection level can be changed online from disk-level failures to node-level failures, and even rack-level and data center-level failures. After expanding from small-scale nodes to multiple nodes, the protection level can be changed online, thereby improving storage data efficiency and reducing storage costs per GB. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a module diagram of the device provided by the present invention;

[0036] Figure 2 This is an example diagram of the original PG data sorting process provided by the present invention;

[0037] Figure 3 It is a flow chart of the method provided by the present invention. DETAILED DESCRIPTION

[0038] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0039] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0040] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in a preferred embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it constitute a separate or selective embodiment that is mutually exclusive of other embodiments.

[0041] See also Figure 1 As shown, the present invention provides an intelligent EC processing device, including a Chunk allocation module, a PG view management module, a Pool storage resource pool, a Disk list, and a Chunk Saver thread;

[0042] The Chunk allocation module is used to allocate new data storage space to the storage system for additional writes and to write new user data;

[0043] The PG view management module is used to obtain the PG state, wherein the PG state includes a read-only state, a read-write state, and an uncertain state;

[0044] The Pool storage resource pool includes multiple PGs, and when the Pool storage resource pool is created, the total number of the PGs is synchronously determined;

[0045] The Disk list is used to list all PGs, where each PG corresponds to a unique number;

[0046] The Chunk Saver thread is used to process Disk information, and each Chunk Saver thread is responsible for processing one or more Disk information.

[0047] In the present invention, EC technology is the data redundancy technology currently used in the mainstream distributed storage. Generally, 4+2, 8+3 or a larger ratio of 22+3 is used. In EC technology, as the ratio of data disks to redundant disks increases, storage efficiency is further improved. In distributed storage, the number of storage nodes is usually increased on demand, but in traditional EC, EC or replicas do not change after creation, so that storage efficiency cannot be improved after expansion, affecting the storage cost per TB. In this embodiment, a Chunk allocation module, a PG view management module, a Pool storage resource pool, a Disk list and a Chunk list are set. Saver thread, Chunk allocation module is highly intelligent and automated, and can intelligently allocate new data storage space according to the load and demand of the storage system, thereby ensuring that the additional user data can be properly stored and managed. This module effectively improves the utilization of storage space through precise calculations and algorithms, and can flexibly adjust the storage space allocation strategy to adapt to different user needs and data types. The PG view management module is a powerful and efficient management tool that can obtain and monitor the status of PG in real time. These states include read-only state, read-write state and uncertain state. Each state represents the different functions and uses of PG. For example, the read-only state indicates that PG can only read data, while the read-write state means that PG can perform data manipulation. For read and write operations, an uncertain state means that the status of the PG may be questionable or unstable and requires further inspection and confirmation. The Pool storage resource pool is a comprehensive storage solution that includes multiple PGs. These PGs are determined when the Pool storage resource pool is created, and each PG has its own unique number. This design makes the Pool storage resource pool highly scalable and flexible, and PGs can be easily added or deleted to meet changing storage needs. The Disk list is an important component that lists all PGs in the Pool storage resource pool. This list not only provides quick access to each PG, but also allows users to easily understand and manage every element in the Pool storage resource pool. The Chunk Saver thread is the key component responsible for processing Disk information. Each Chunk Saver thread is responsible for processing one or more Disk information to ensure data integrity and consistency. This design makes data storage and management more efficient and reliable, greatly improving overall work efficiency.

[0048] In a preferred embodiment, the Chunk allocation module obtains the requirements of the upper-layer application before execution, and when the upper-layer application needs to write data, generates a Chunk allocation request and sends it to the Chunk allocation module, which allocates the newly added Chunk space to the upper-layer application.

[0049] In this embodiment, before executing the Chunk allocation module, it is first necessary to obtain the needs of the upper-level application. Specifically, when the upper-level application needs to write data, a Chunk allocation request will be generated and sent to the Chunk allocation module. The module will then allocate new Chunk space for the upper-level application. This process has high efficiency and response speed because the Chunk allocation module has advanced design and algorithms and can accurately calculate and allocate the required Chunk space. This allocation mechanism not only meets the upper-level application's demand for data storage space, but also optimizes the utilization of storage space and realizes efficient data storage and management. In addition, the Chunk allocation module also has a high degree of reliability and stability, ensuring that the upper-level application can run stably and complete data storage tasks when it needs to write data.

[0050] Secondly, the specific process of the chunk allocation module allocating the newly added chunk space to the upper-layer application is as follows:

[0051] Obtain the PG status corresponding to the newly added Chunk space and the PG space occupancy;

[0052] Get the allowed allocation threshold and compare it with the PG space occupancy. When the PG state is read-write state, if the PG space occupancy is greater than the allowed allocation threshold, allocate Chunks from the PG to the upper-level application. Otherwise, continue to match the next PG or add PG.

[0053] In this embodiment, before allocating newly added Chunk space to the upper-level application, it is necessary to first obtain the PG status of each PG. Among them, only PGs that are in the read-write state and whose PG space occupancy is greater than the allowable allocation threshold will be deemed allocable. Otherwise, Chunk space can be allocated to the upper-level application by adding PGs.

[0054] Secondly, when the Pool storage resource pool executes node expansion, it performs EC conversion, adds a new PG, and sets the protection mode of the new PG to the EC mode that needs to be converted. The Chunk allocation module applies for Chunks from the PG in the new EC mode to the upper-level application, and then organizes the PG data of the original PG. After the original PG is organized into the new EC mode, the Chunk allocation module continues to apply for Chunks from the original PG to the upper-level application.

[0055] In this implementation, when EC conversion is performed, it supports single-node, inter-disk EC within the node to achieve disk-level fault processing and recovery. When there are 2 or 3 nodes, inter-node data reliability management is achieved through inter-node replicas or folded EC to achieve node and fault processing and recovery. When the number of nodes is greater than 3 nodes, a more efficient EC mode is adopted to achieve inter-node data reliability management and improve storage space utilization. EC can gradually increase the number of EC disks from 4+2 to 20+3 according to the number of nodes to improve data redundancy efficiency. By configuring the default number of redundant data copies, when the number of nodes is greater than the number of redundant copies + 1, the system will automatically convert the unreachable The data with reliability requirements is sorted to meet the corresponding reliability requirements, such as from single-node EC to inter-node EC (3 copies or 4+2:1 EC can also be sorted into EC with higher storage efficiency, such as 16+3). You can also set availability priority or reliability priority. When it is set to availability priority, the service can still be provided after the number of nodes is less than the set number of redundant data copies, but it will degenerate to 2 copies or intra-node EC. In addition, data can be read during the original PG data sorting process, which has no impact on system availability. When all PGs are sorted completely, the Chunk allocation unit will allocate storage space from each PG according to the balance strategy.

[0056] In a preferred embodiment, the EC conversion includes node expansion and node contraction, wherein the node expansion is from within the node to between nodes, and the EC ratio between nodes is gradually increased, and the node contraction is to adjust the PG EC ratio.

[0057] In this embodiment, when performing node expansion, taking the total number of PGs as 2048 as an example, when a single node is used, PGs 1 to 16 are used, and the PG redundancy mode is set to intra-node EC: 4+2. When there are three nodes, PGs 17 to 64 are added, and the PG redundancy mode is set to 3 replicas (4+2:1). Newly applied data uses PGs 17 to 64, thereby ensuring that new data can meet reliability requirements. At the same time, PGs 1 to 16 are set to read-only status and are pending for sorting. When upgrading to N nodes, PGs 65 to 1024 are added, and the PG redundancy mode is set to inter-node EC mode. Newly applied data uses PGs 65 to 1024, thereby ensuring that new data can meet reliability requirements and improving data storage efficiency.

[0058] When scaling down a node, take the example of adjusting 10+2 to 8+2. First, adjust the original 10+2 PG status to read-only, add PGs 1025 to 1224, set the PG redundancy mode to (8+2), use PGs 1025 to 1224 for new data, organize the original 10+2 PG data, and convert it to 8+2 EC. After the organization is completed, set the PG status to read-write.

[0059] Secondly, if Figure 2 As shown, the steps for arranging PG data of the original PG are as follows:

[0060] Set the original PG to read-only mode, then adjust it to the new EC mode, and generate a PG reconstruction list;

[0061] Get a list of all chunks containing data in the original PG, read the complete chunk data, and then re-encode the read chunk data using the new PG EC protection mode. Write the re-encoded data to the reconstructed disk by EC sharding until all chunk data has been re-EC-encoded.

[0062] Update the disk list of the new PG to the reconstructed disk list, change the PG protection mode, and update the PG version information at the same time, and then change the status of the PG after the sorting is completed to read-write status.

[0063] In this embodiment, when the original PG is sorted, the PG EC protection mode is changed, and the PG is set to read-only state. According to the EC protection mode adjusted as needed, a PG reconstruction list is generated. The disks in the PG reconstruction list do not overlap with the disks in the original PG disk list. Figure 2 PG126 in, when there is a read request, EC decoding is performed according to mode 8+2 from the disk list of PG126, the corresponding data in the Disk shard is read, and a list of all the Chunks with data in PG is obtained (which can also be obtained one by one). Then, the complete Chunk data is read, and the complete Chunk data is encoded using the new protection mode 4+2. The newly encoded data is written to the reconstructed Disk according to the EC shard. Finally, the status of the PG that has been sorted is changed to the read-write status.

[0064] Secondly, after the original PG is set to read-only state, if there is a read request, EC decoding is performed according to the original PG EC protection mode from the original PG disk list, and the corresponding data in the Disk segment is read; otherwise, the original PG still performs EC mode conversion in read-only state.

[0065] In this implementation, the original PG is set to read-only state and can still receive read requests. At this time, EC decoding will be performed from the original PG disk list according to the original PG EC protection mode, and then the corresponding data in the Disk shard will be read. The PG in this state does not receive write requests.

[0066] In a preferred embodiment, after the PG data of the original PG is sorted out, the disks in the original Disk list are notified through the corresponding Chunksaver thread to reclaim the space occupied by the Chunks and delete the Chunk information in the Disk;

[0067] Among them, the client does not change the PG status information, and sends a read request to the disk in the original PG disk list to read data. After the disk in the original PG disk list receives the request;

[0068] If the Chunk data has been deleted, the Chunk saver thread corresponding to the Disk will return an error message, and the client will refresh the PG view to read the Chunk shard data from the new Disk.

[0069] If Disk does not receive the chunk deletion message, it indicates that the chunk exists and the corresponding data is returned.

[0070] In this implementation, after the PG data of the original PG is sorted out, the client may not change the PG status information in time, and a read request may be sent to the Disk in the original PG disk list to read the data. After the Disk in the original PG disk list receives the request, there will be two situations. First, the Chunk data has been deleted. At this time, the Disk (corresponding Chunk saver thread) will return an error message, and the client refreshes the PG view to the new Disk to read the Chunk fragment data. Second, the Disk has not received the Chunk deletion information, and the Chunk exists, then the corresponding data is returned. Since the data has not been overwritten, the data correctness is guaranteed.

[0071] See also Figure 3 The present invention further provides an intelligent EC processing method, which is applied to the above-mentioned intelligent EC processing device, comprising:

[0072] S1. Obtain the needs of the upper-layer application, and when the upper-layer application needs to write data, generate a Chunk allocation request, and then allocate new Chunk space from the PG in the read-write state according to the Chunk allocation request;

[0073] S2. Obtain the expansion node, perform EC conversion, update the PG protection mode, change the original PG state to read-only, and continue to receive read requests;

[0074] S3: After the original PG is organized into the new EC mode, the chunk allocation module continues to request chunks from the original PG for upper-layer applications.

[0075] S4. Notify the disks in the original Disk list to reclaim the space occupied by the Chunk and delete the Chunk information in the Disk.

[0076] As described in the above steps S1-S4, first obtain the needs of the upper-level application to ensure that the write request for data is met. When the upper-level application needs to write data, a Chunk allocation request is generated to ensure that there is enough space to store the data. According to the Chunk allocation request, new Chunk space is allocated from the PG in the read-write state to ensure data storage and management. Then, the expansion node is obtained and the EC conversion is performed to ensure stable storage of the data. At the same time, the PG protection mode is updated to protect the security and integrity of the data. Then, the original PG state is changed to read-only state to prevent data from being accidentally modified. At the same time, read requests continue to be received to meet the user's data reading needs. After that, data sorting is performed on the original PG, and the data is sorted according to the new EC mode for better management and storage of data. After the original PG is sorted into the new EC mode, the Chunk allocation module continues to apply for Chunks from the original PG to the upper-level application to ensure the stability and reliability of data storage and management. Finally, the disk in the original Disk list is notified to reclaim the space occupied by the Chunk to avoid space waste, and the Chunk information in the Disk is deleted to ensure the consistency and accuracy of data storage and management.

[0077] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, apparatus, article, or method comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, apparatus, article, or method. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, apparatus, article, or method comprising the element.

[0078] The foregoing is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art may make various improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained herein shall, unless otherwise specified or limited, be implemented in accordance with conventional means in the art.

Claims

1. An intelligent EC processing device, comprising a Chunk allocation module, a PG view management module, a Pool storage resource pool, a Disk list, and a ChunkSaver thread, characterized in that: The Chunk allocation module is used to allocate new data storage space to the storage system for additional writes and to write new user data; The PG view management module is used to obtain the PG state, wherein the PG state includes a read-only state, a read-write state, and an uncertain state; The Pool storage resource pool includes multiple PGs, and when the Pool storage resource pool is created, the total number of the PGs is synchronously determined; The Disk list is used to list all PGs, where each PG corresponds to a unique number; The ChunkSaver thread is used to process Disk information, and each ChunkSaver thread is responsible for processing one or more Disk information; Before the execution of the Chunk allocation module, the requirements of the upper-layer application are obtained, and when the upper-layer application needs to write data, a Chunk allocation request is generated and sent to the Chunk allocation module, and the Chunk allocation module allocates the newly added Chunk space to the upper-layer application; The specific process of the Chunk allocation module allocating the newly added Chunk space to the upper-layer application is as follows: Obtain the PG status corresponding to the newly added Chunk space and the PG space occupancy; Get the allowed allocation threshold and compare it with the PG space occupancy. When the PG state is read-write state, if the PG space occupancy is greater than the allowed allocation threshold, allocate Chunks from the PG to the upper-level application. Otherwise, continue to match the next PG or add PG.

2. The intelligent EC processing device according to claim 1, characterized in that: When the Pool storage resource pool executes node expansion, EC conversion is performed, and a new PG is added, and the protection mode of the new PG is set to the EC mode that needs to be converted. The Chunk allocation module applies for Chunks from the PG in the new EC mode to the upper-level application, and then organizes the PG data of the original PG. After the original PG is organized into the new EC mode, the Chunk allocation module continues to apply for Chunks from the original PG to the upper-level application.

3. The intelligent EC processing device according to claim 2, characterized in that: The EC conversion includes node expansion and node reduction. The node expansion is from within the node to between nodes, and the EC ratio between nodes is gradually increased. The node reduction is to adjust the PGEC ratio.

4. The intelligent EC processing device according to claim 2, characterized in that: The steps for arranging PG data on the original PG are as follows: Set the original PG to read-only mode, then adjust it to the new EC mode, and generate a PG reconstruction list; Obtain a list of all chunks containing data in the original PG, read the complete chunk data, and then re-encode the read chunk data using the new PGEC protection mode. Write the re-encoded data to the reconstructed disk in EC shards until all chunk data has been re-EC-encoded. Update the disk list of the new PG to the reconstructed disk list, change the PG protection mode, and update the PG version information at the same time, and then change the status of the PG after the sorting is completed to read-write status.

5. The intelligent EC processing device according to claim 4, characterized in that: After the original PG is set to read-only state, if there is a read request, EC decoding is performed from the original PG disk list according to the original PGEC protection mode, and the corresponding data in the Disk segment is read. Otherwise, the original PG still performs EC mode conversion in read-only state.

6. The intelligent EC processing device according to claim 4, characterized in that: After the PG data of the original PG is sorted out, the corresponding Chunksaver thread is used to notify the disks in the original Disk list to reclaim the space occupied by the Chunks and delete the Chunk information in the Disk; Among them, the client does not change the PG status information, and sends a read request to the disk in the original PG disk list to read data. After the disk in the original PG disk list receives the request; If the Chunk data has been deleted, the Chunksaver thread corresponding to the Disk will return an error message, and the client will refresh the PG view to read the Chunk shard data from the new Disk; If Disk does not receive the chunk deletion message, it indicates that the chunk exists and the corresponding data is returned.

7. The intelligent EC processing device according to claim 4, characterized in that: The disks in the PG reconstruction list and the disks in the original PG disk list do not overlap with each other.

8. An intelligent EC processing method, applied to the intelligent EC processing device according to any one of claims 1 to 7, characterized in that: include: Obtain the needs of the upper-layer application, and when the upper-layer application needs to write data, generate a Chunk allocation request, and then allocate new Chunk space from the PG in the read-write state according to the Chunk allocation request; Get the expanded node, perform EC conversion, update the PG protection mode, change the original PG state to read-only, and continue to receive read requests; After the original PG is organized into the new EC mode, the chunk allocation module continues to request chunks from the original PG for upper-layer applications. Notify the disks in the original Disk list to reclaim the space occupied by the Chunk and delete the Chunk information in the Disk.

Citation Information

Patent Citations

  • Video data storage method and device, computer equipment and storage medium

    CN113791735A