Data management method and device, computer equipment and storage medium

Abstract

The invention relates to a data management method and device, computer equipment and a storage medium. The method comprises the steps of obtaining a logic data block corresponding to a to-be-processed file in response to a received file storage instruction, and performing erasure code coding on the logic data block to obtain a physical data block and a verification block; obtaining the size of the logic data block and a preset value, and determining storage position types of the physical data block and the verification block according to a comparison result of the size of the logic data block and the preset value; in response to the storage position type being the first storage position type, storing the physical data block and the verification block to a first storage volume of the first type of hard disk; and in response to the storage position type being a second storage position type, storing the physical data block and the verification block to a second storage volume of a second type of hard disk, the first type of hard disk being a solid state disk, and the second type of hard disk being a mechanical hard disk, by adopting the method, the data file read-write performance can be improved.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a data management method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In the field of distributed storage, solving the performance challenge of small files is usually achieved through append-only aggregation schemes. These schemes require merging files after deletion to free up space, which places a significant burden on disk I / O. In real-world production environments, where files of varying sizes are mixed, especially with a high proportion of large files, the repeated merging and the resulting I / O overhead can impact the latency of business requests. Summary of the Invention

[0003] Therefore, it is necessary to provide a highly reliable and low-overhead data management method, apparatus, computer equipment, and storage medium to address the aforementioned technical problems.

[0004] To address the aforementioned technical problems, firstly, a data management method is provided, the method comprising: In response to receiving a file storage instruction, the logical data block corresponding to the file to be processed is obtained, and the logical data block is encoded with erasure coding to obtain the physical data block and the check block. Obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value; In response to a storage location type of the first storage location type, the physical data block and the parity block are stored on the first storage volume of the first type of hard disk; in response to a storage location type of the second storage location type, the physical data block and the parity block are stored on the second storage volume of the second type of hard disk, wherein the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

[0005] In one embodiment, the preset values ​​include a first preset value and a second preset value. Obtaining the size of the logical data block and the preset value, and determining the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value, includes: If the size of the logical data block is less than or equal to the first preset value, the physical data block and the check block are appended to the first storage volume of the first type of hard disk. If the size of the logical data block is equal to the second preset value, the physical data block and the check block are appended to the second storage volume of the second type of hard disk.

[0006] In one embodiment, the method further includes: Obtain the used storage capacity of the first storage volume of the first type of hard disk, and compare the used storage capacity of the first storage volume of the first type of hard disk with a preset storage threshold; In response to the fact that the used storage capacity of the first storage volume of the first type of hard disk is greater than or equal to a preset storage threshold, the first storage volume is sealed and the sealed storage volume is divided into multiple stripes. The multiple stripes are then encoded with erasure codes and stored in the third storage volume of the second type of hard disk.

[0007] In one embodiment, the method further includes: Obtain the metadata descriptor information corresponding to the physical data block. The metadata descriptor information includes the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. The metadata descriptor consists of a metadata identifier, version number, metadata bitmap, serial number, and checksum. The size of the metadata is calculated based on the space occupied by the metadata descriptors and the number of metadata descriptors in each storage volume; Determine whether the entire metadata can be stored in memory based on its size.

[0008] In one embodiment, determining whether the metadata can be fully stored in memory based on its size includes: Get the memory capacity and calculate the ratio of the metadata size to the memory capacity; If the percentage is less than or equal to a preset ratio, the entire metadata will be stored in memory.

[0009] In one embodiment, the physical data block includes multiple physical slices, and the method further includes: In response to receiving a file update instruction, determine the initial data file corresponding to the file to be updated; The file to be updated is compared with the initial data file to obtain the difference file; Determine the target physical slice corresponding to the difference file; Update the data of the target physical slice using the difference file.

[0010] In one embodiment, the method further includes: In response to receiving a data file deletion command, obtain the storage location of the physical data block corresponding to the data file to be deleted; In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the first storage volume of the first type of hard disk, the physical data block corresponding to the data file to be deleted is marked for deletion, the physical data blocks in the first storage volume that have not been marked for deletion are merged to obtain the updated first storage volume, and the original first storage volume is deleted from the first type of hard disk. In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the second storage volume of the second type of hard disk, the physical data block corresponding to the data to be deleted is marked with holes. In response to receiving the file system release command, the physical data block marked with holes is released.

[0011] To address the aforementioned technical problems, a second aspect provides a data management device, comprising: The acquisition module is used to respond to the received file storage instruction, acquire the logical data block corresponding to the file to be processed, encode the logical data block with erasure coding, and obtain the physical data block and the check block; The determination module is used to obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value. A storage module is configured to, in response to a storage location type of a first storage location type, store physical data blocks and parity blocks on a first storage volume of a first type of hard disk; and in response to a storage location type of a second storage location type, store physical data blocks and parity blocks on a second storage volume of a second type of hard disk, wherein the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

[0012] To address the aforementioned technical problems, a third aspect provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: the processor executes the steps of the method described in the first aspect.

[0013] In order to solve the above-mentioned technical problems, in a fourth aspect, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method described in the first aspect.

[0014] In this application, upon receiving a file storage instruction, the system retrieves the logical data block corresponding to the file to be processed, performs erasure coding on the logical data block to obtain a physical data block and a check block, retrieves the size and preset value of the logical data block, and determines the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value, and stores the physical data block and the check block on a first storage volume of a first type of hard disk if the storage location type is a first storage location type, and stores the physical data block and the check block on a second storage volume of a second type of hard disk if the storage location type is a second storage location type. The first type of hard disk is a solid-state drive (SSD), and the second type of hard disk is a mechanical hard disk (HDD). In this way, by distinguishing the size of the logical data block, the physical data block and check block corresponding to the logical data block with lower IO consumption are stored in the SSD, and the physical data block and check block corresponding to the logical data block of a fixed size are stored in the HDD. This enables small files to have higher read and write performance, while avoiding the IO consumption caused by large-scale merging, thereby improving the read and write performance of data files. Attached Figure Description

[0015] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating data management methods in related technologies; Figure 2 This is a flowchart illustrating a data management method in one embodiment; Figure 3 This is a flowchart illustrating the data management method in another embodiment; Figure 4 This is a schematic diagram of the data volume distribution in one embodiment; Figure 5 This is a schematic diagram of the data volume structure in one embodiment; Figure 6 This is a schematic diagram of the structure of a data management system in one embodiment; Figure 7 This is a structural block diagram of a data management device in one embodiment; Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] In the field of distributed storage, related technologies primarily address the performance challenges of small files through append-only aggregation schemes. Append-only optimization is a data processing technique mainly used to improve data read and write performance. With append-only optimization, once data is written, it is not modified or deleted; new data is typically appended to the end of existing data. When using this aggregation scheme, after deleting a file, space needs to be freed up by merging the remaining files. Figure 1 In this process, the data files include (file1, file2, file3, file4, file5). The deleted files (file3, file5) in the data files are marked for deletion (del), and then removed from the data files. The remaining undeletable files (file1, file2, file4) are then merged. This repeated merging of numerous files leads to high I / O consumption.

[0019] In one embodiment, such as Figure 2 As shown, a data management method is provided, which includes the following steps: Step 101: In response to receiving a file storage instruction, obtain the logical data block corresponding to the file to be processed, encode the logical data block using erasure coding, and obtain the physical data block and the check block.

[0020] Step 102: Obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value.

[0021] Step 103: In response to the storage location type being the first storage location type, the physical data block and the parity block are stored on the first storage volume of the first type of hard disk; in response to the storage location type being the second storage location type, the physical data block and the parity block are stored on the second storage volume of the second type of hard disk, where the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

[0022] The preset values ​​include a first preset value and a second preset value. Obtaining the size of the logical data block and the preset value, and determining the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value, includes: if the size of the logical data block is less than or equal to the first preset value, storing the physical data block and the check block in an append manner on the first storage volume of the first type of hard disk; if the size of the logical data block is equal to the second preset value, storing the physical data block and the check block in an append manner on the second storage volume of the second type of hard disk.

[0023] A logical data block typically refers to the smallest unit of operation in a file system; it's the result of addressing physical device data from the operating system's perspective. The size of a logical data block is variable, depending on the file system settings, and is usually an integer multiple of the physical data block size. This design allows the file system to manage data efficiently. A physical data block, also known as a sector, is the smallest unit of operation on disk hardware. The operating system reads data in blocks; logical data blocks and physical data blocks are equal in size and integer multiples of sectors. Files are stored non-contiguously on the disk; inodes map logical data blocks to physical data blocks, improving file system efficiency.

[0024] A volume is a commonly used data management unit in distributed storage. It is designed to organize data into a unified management method, so that data operations on disk (including data reading and writing, data repair, data migration, etc.) can be managed in a unified and centralized manner.

[0025] A block is the basic unit of data storage and management, used to divide large files into fixed-size blocks and distribute them across multiple nodes. This design optimizes the storage, parallel processing, and fault tolerance of large-scale data.

[0026] Specifically, the files to be processed can be user-created or edited files, data query results, file backups, application caches, and so on.

[0027] After receiving the file to be processed, its size can be obtained. It can then be determined whether the file size is less than or equal to a first preset value, such as 10MB. If the file size is less than or equal to the first preset value, it indicates that the file is very small and belongs to an irregular size category. In this case, the file to be processed is the logical data block corresponding to the file to be processed. Erasure coding is performed on the logical data block to obtain the corresponding physical data block and checksum block. The physical data block and checksum block are then appended to the first storage volume of the solid-state drive.

[0028] If the size of the file to be processed is greater than the first preset value, it is considered a large file and needs to be split into multiple sub-files of a fixed size (second preset value). These multiple sub-files are the multiple logical data blocks corresponding to the file to be processed. That is, each sub-file corresponds to one logical data block. Erasure coding is performed on the multiple logical data blocks to obtain the physical data blocks and check blocks corresponding to the multiple logical data blocks. The physical data blocks and check blocks are then appended to the second storage volume of the mechanical hard disk.

[0029] In practical applications, when splitting a file to be processed into sub-files of a fixed size, there may also be parts of the file to be processed that do not meet the fixed size requirement. In this case, the parts of the file to be processed that do not meet the fixed size requirement are also treated as logical data blocks corresponding to the parts of the file to be processed. The logical data blocks are then encoded with erasure codes to obtain the physical data blocks and check blocks corresponding to the logical data blocks. The physical data blocks and check blocks are then appended to the first storage volume.

[0030] In a specific example, such as Figure 3 As shown, once a large file (shown as bigfile) is obtained, it is first divided into multiple logical data blocks (shown as S1, including blocks11, 12, 13, 14, 21, 22, 23, and 24). Using erasure coding (EC), the logical data blocks (shown as S2, blocks11, 12, 13, and 14) are encoded into physical data blocks (shown as S3, blocks11, 12, 13, and 14) and check blocks (shown as blocks15 and 16). The check blocks can be used to store different physical data blocks on different physical volumes (shown as S3). In this way, when any physical data block is damaged, the damaged physical data block can be recovered from the undamaged data blocks and the check blocks.

[0031] In this application, a third storage volume is also set on the mechanical hard disk. Specifically, the used storage capacity of the first storage volume of the first type of hard disk is obtained, and the used storage capacity of the first storage volume of the first type of hard disk is compared with a preset storage threshold. In response to the first storage volume of the first type of hard disk being greater than or equal to the preset storage threshold, the first storage volume is sealed and the sealed storage volume is divided into multiple stripes. The multiple stripes are then encoded with erasure codes and stored in the third storage volume of the second type of hard disk.

[0032] Please see Figure 3This application sets up three types of data volumes, including a first data volume (normal volume) set on a solid-state drive, a second data volume (regular volume) set on a mechanical hard drive, and a third data volume (sunken volume) set on a mechanical hard drive.

[0033] Disks include hard disk drives (HDDs) and solid-state drives (SSDs). HDDs are traditional storage devices that use a mechanical arm and rotating disks. They work by having a read / write head search for and retrieve data on the rotating disk. SSDs are a newer type of storage device that uses flash memory technology. They contain no moving parts and instead use chips to store data. Because there is no mechanical movement, SSDs have faster read / write speeds and shorter response times.

[0034] In this application, smaller files are stored in a solid-state drive (SSD), while larger files are stored in fixed-size blocks in a second storage volume on a mechanical hard drive (HDD). Additionally, a sub-volume of a regular volume is set up in the HDD. When there is too much data stored in the regular volume, the data in the regular volume can be sealed and stored in the sub-volume, which can further save space.

[0035] Since the first data volume and the second data volume store different data, this application also sets different data deletion methods for the first data volume and the second data volume.

[0036] Specifically, in response to receiving a data file deletion command, the storage location of the physical data block corresponding to the data file to be deleted is obtained; in response to the storage location of the physical data block corresponding to the data file to be deleted being a first storage volume of a first type of hard disk, the physical data block corresponding to the data file to be deleted is marked for deletion, the physical data blocks in the first storage volume that have not been marked for deletion are merged to obtain an updated first storage volume, and the original first storage volume is deleted from the first type of hard disk; in response to the storage location of the physical data block corresponding to the data file to be deleted being a second storage volume of a second type of hard disk, the physical data block corresponding to the data file to be deleted is marked for hole-free operation, and in response to receiving a file system release command, the physical data block marked for hole-free operation is released.

[0037] Upon receiving a data file deletion command, the system can parse the command to obtain the storage location of the physical data block corresponding to the file to be deleted. If the storage location of the physical data block is on the first storage volume, the file to be deleted is removed from the data file, and any remaining files are aggregated. If the storage location is on the second storage volume, a hole-punching technique can be used to return space to the file system. That is, when the physical data block corresponding to the file is deleted, the file system does not immediately release these blocks but marks them as "holes." These marked "holes" are released at an appropriate time, thus reducing the actual disk space occupied. The second data volume setup supports good write performance through append-only writes and uses page-aligned hole punching to immediately release disk space, balancing performance and storage utilization.

[0038] This application employs a first data volume: files or blocks of irregular size are written to the solid-state drive (SSD) in an append-only manner. When deleting a file, the file to be deleted is removed from the data volume, and any remaining files are aggregated. The advantage is that write requests are transformed into append-only writes, which benefits the performance of sequential disk writes. Since these are small files, the amount of data recycled is very small, and the resulting I / O consumption is negligible. A second data volume is set up: large files are written to the mechanical hard drive (HDD) in a fixed-size, append-only manner. During writes and deletions, space is returned to the file system through perforations. This method is suitable for writing larger files in fixed-size blocks. The advantage is that it avoids I / O contention caused by garbage collection. Thus, by setting up multiple data volumes to correspond to multiple data management methods and adapt to different file sizes, the I / O consumption problem introduced by file aggregation schemes in scenarios with a large number of mixed-size files can be solved.

[0039] In one embodiment, the physical data block includes multiple physical slices, and the method further includes: in response to receiving a file update instruction, determining the initial data file corresponding to the file to be updated; comparing the file to be updated with the initial data file to obtain a difference file; determining the target physical slice corresponding to the difference file; and updating the data of the target physical slice using the difference file.

[0040] like Figure 5As shown in the diagram, PhyVlume0, PhyVlume1, PhyVlume2...PhyVlume127 are data files, and PhyVlumeMeta0, PhyVlumeMeta1, PhyVlumeMeta2...PhyVlumeMeta127 are the metadata files corresponding to the data files. Each PhyVlume data file contains multiple data blocks (Block0, Block1, Block2...Block8189 in the diagram), and each data block contains multiple physical slices (silce0, silce1, silce2,...silce(N-1)). Each metadata file contains multiple metadata descriptors (BlockDesc0, BlockDesc1, BlockDesc2,...BlockDesc8191). Each metadata descriptor BlockDesc is composed of magic (metadata identifier), version (version number), bitmap (metadata bitmap), seqld (sequence number), checksum (checksum), etc.

[0041] In the data volume design, there are multiple data files on the disk (each data file is 32GB by default, but configurable), and each data file corresponds to a metadata file. To reduce management granularity, each data file is composed of multiple volume physical blocks, and the volume physical blocks correspond to the logical blocks of the business files. To reduce read / write amplification from overwrites, the internal layout of a physical block is divided into multiple slices, each typically 256KB in size. A slice is the smallest granularity for writing to disk. Each block corresponds to a metadata blockDesc, and each blockDesc occupies 128 bytes.

[0042] This configuration allows the system to determine the differences between the file to be updated and the initial data file after receiving a file update instruction, thereby obtaining a difference file. Based on this difference file, the system can modify the content in each physical block at the slice level, which improves both file update efficiency and accuracy.

[0043] In one embodiment, metadata descriptor information corresponding to physical data blocks can be obtained. The metadata descriptor information includes the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. The metadata descriptor consists of a metadata identifier, version number, metadata bitmap, serial number, and checksum. The size of the metadata is calculated based on the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. Based on the size of the metadata, it is determined whether the metadata can be fully stored in memory.

[0044] In a specific example, the actual space occupied by metadata is approximately 128 / (1024 * 1024 * 4) = 0.003% (calculated as: BlockDesc space occupied / Block data space occupied), which is much smaller than the space consumption of metadata in a standard file system. Assuming a regular volume size of 32GB, the metadata size is: 128 * (32GB / 4MB) = 1024KB (calculated as: metadata descriptor space occupied * number of metadata descriptors in each data volume). In this example, the metadata descriptor space occupied is the metadata descriptor size.

[0045] In one embodiment, determining whether the metadata can be fully stored in memory based on its size includes: obtaining the memory capacity and calculating the ratio of the metadata size to the memory capacity; if the ratio is less than or equal to a preset ratio, then storing the metadata in memory.

[0046] In a specific example, assume the data redundancy method is EC4:2 (the data is divided into 4 physical data blocks and 2 additional check blocks are generated), each physical data block is 4MB in size, each slice is 256K in size, the pool security level is node-level, and each node is 8TB*12=96TB.

[0047] We can conclude that with EC4:2, the logical block size = physical block size * 4 = 4MB * 4 = 16MB. One logical block corresponds to 6 (4+2) physical blocks. Since the pool's security level is node-level, the 6 physical blocks are distributed across 6 different nodes.

[0048] From the above text, "Based on a regular volume of 32GB, the metadata size is: 128 * (32GB / 4MB) = 1024KB (calculation formula: BlockDesc space occupied * number of BlockDesc in each volume)," we know that the metadata size corresponding to a 32GB volume is 1024KB; therefore, a node has 96TB = 96 * 1024GB; therefore, the number of volumes in a node is 96 * 1024 / 32; the total metadata size is: 96 * 1024 / 32 * 1024KB = 3072 * 1m = 3GB. Assuming a machine has 96TB of disk space available for storing data, when fully written, the size of the metadata would be: (100*1024) / 32 * 1024KB = 3GB. Comparing the size of the metadata with the memory capacity, we obtain the calculated percentage of the metadata size to the memory capacity. Here, memory capacity refers to the maximum memory capacity, typically 256GB. When the metadata size is 3GB, the percentage of the metadata to the memory capacity is 3GB / 256GB ≈ 1%. When the percentage is 1%, even if the entire metadata is stored in memory, it will not occupy too much memory space and will have little impact on memory usage.

[0049] Traditional distributed storage systems require a full load of metadata (such as block mapping tables and checksum information) for both physical and logical volumes when the volume server starts up. This results in: significantly longer metadata loading times for large volumes, leading to high startup latency; and excessive memory consumption due to redundant metadata, resulting in high memory pressure.

[0050] This application allows for the full loading of all physical volume metadata descriptor structures into memory during the volume server startup phase, with the metadata descriptors periodically persisted via write-ahead logs or snapshots. This ensures consistency during fault recovery. During the logical volume loading phase, only thinned metadata is loaded, and only the free space management structure and seqId / genId indexes are initialized. At runtime: new blocks are quickly allocated using the logical volume's free space information, and seqId / genId are updated after allocation. The latest version is determined by comparing seqId, and genId avoids writing expired copies. By preloading physical volume BlockDesc and thinned logical volume metadata, fast startup and low memory overhead are achieved, while the seqId / genId mechanism ensures consistency across multiple replicas.

[0051] This application acquires a file to be processed. When the file is large, it is split into multiple logical data blocks. EC encoding is performed on each logical data block to generate at least one physical data block and at least one check block. A mapping relationship is established between physical and logical data blocks, which are mapped through metadata. The physical and logical data blocks are recorded, including their corresponding record block IDs, EC group numbers, and fault domain locations. When physical data blocks are distributed across different fault domains (e.g., different zones or racks), they are written to the disk file system's regularized volume data file in a page-aligned manner. During file storage writes, the content of each physical data block is appended to the volume data file, along with accompanying description information, which is simultaneously stored on disk and in memory. During file updates, each physical block is modified at the slice level. Modifications at the slice level (e.g., 1MB) only rewrite the physical data block corresponding to the affected slice and update the EC encoding group. During deletion, a fallocate punchhole is directly applied to the physical block.

[0052] This application provides a data management system, such as Figure 6 As shown, the system includes a logical volume management module and a data volume management module.

[0053] The logical volume management module includes a block allocator (illustrated as Block Allocator), which manages the mapping between logical data block IDs and physical data blocks; an EC splitting and computation (illustrated as EC OP), which performs block splitting and encoding / decoding; and the management of the write queue (illustrated as Write API queue) and the send queue (illustrated as Send queue), which are used for write request buffering and priority scheduling.

[0054] The physical volume management module includes a metadata cache (Metacache shown in the diagram) for accelerating block location lookups; a volume descriptor management module (DescManager shown in the diagram) for managing metadata descriptors; a block operation management module (Block Ops shown in the diagram) for managing physical data blocks; and a disk scheduling management module (Disk Schedule shown in the diagram) for merging adjacent I / O requests and optimizing throughput.

[0055] Logical volume management and physical volume management communicate via Grpc (Grpc Connection Pool). Grpc communication implements the encapsulated block operation protocol, supporting cross-node data transmission.

[0056] For example, assuming a client writes a 1GB file, the logical volume management module splits the file into 16 logical data blocks (64MB / block). Each logical data block is EC encoded to generate 9 physical data blocks (6 data + 3 checksums), distributed across 9 zones. The physical volume management module appends the physical data blocks to the volume files of each zone, recording the metadata location. When overwriting a logical data block, only the physical data blocks of the corresponding slice are updated, and the checksum blocks are recalculated. Upon deletion, a hole is created to release space, and the metadata marks it as reusable.

[0057] It should be understood that, although Figure 2 , Figure 3 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , Figure 3 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0058] In one embodiment, such as Figure 7 As shown, a data management device is provided, including: an acquisition module 70, a determination module 71, and a storage module 72, wherein: The acquisition module 70 is used to respond to the received file storage instruction by acquiring the logical data block corresponding to the file to be processed, performing erasure coding on the logical data block to obtain the physical data block and the check block; The determination module 71 is used to obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value. Storage module 72 is configured to, in response to a storage location type of a first storage location type, store physical data blocks and parity blocks on a first storage volume of a first type of hard disk; and in response to a storage location type of a second storage location type, store physical data blocks and parity blocks on a second storage volume of a second type of hard disk, wherein the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

[0059] In one embodiment, another implementation of the data management method that the above-mentioned device can achieve includes the following specific steps: The preset values ​​include a first preset value and a second preset value. The size of the logical data block and the preset value are obtained. Based on the comparison between the size of the logical data block and the preset value, the storage location type of the physical data block and the check block is determined, including: If the size of the logical data block is less than or equal to the first preset value, the physical data block and the check block are appended to the first storage volume of the first type of hard disk. If the size of the logical data block is equal to the second preset value, the physical data block and the check block are appended to the second storage volume of the second type of hard disk.

[0060] In one embodiment, another implementation of the data management method that the above-mentioned device can achieve includes the following specific steps: The method also includes: Obtain the used storage capacity of the first storage volume of the first type of hard disk, and compare the used storage capacity of the first storage volume of the first type of hard disk with a preset storage threshold; In response to the fact that the used storage capacity of the first storage volume of the first type of hard disk is greater than or equal to a preset storage threshold, the first storage volume is sealed and the sealed storage volume is divided into multiple stripes. The multiple stripes are then encoded with erasure codes and stored in the third storage volume of the second type of hard disk.

[0061] In one embodiment, another implementation of the data management method that the above-mentioned device can achieve includes the following specific steps: The method also includes: Obtain the metadata descriptor information corresponding to the physical data block. The metadata descriptor information includes the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. The metadata descriptor consists of a metadata identifier, version number, metadata bitmap, serial number, and checksum. The size of the metadata is calculated based on the space occupied by the metadata descriptors and the number of metadata descriptors in each storage volume; Determine whether the entire metadata can be stored in memory based on its size.

[0062] In one embodiment, another implementation of the data management method that the above-mentioned device can achieve includes the following specific steps: Determining whether the metadata can be fully stored in memory based on its size includes: Get the memory capacity and calculate the ratio of the metadata size to the memory capacity; If the percentage is less than or equal to a preset ratio, the entire metadata will be stored in memory.

[0063] In one embodiment, another implementation of the data management method that the above-mentioned device can achieve includes the following specific steps: Physical data blocks consist of multiple physical slices, and the method also includes: In response to receiving a file update instruction, determine the initial data file corresponding to the file to be updated; The file to be updated is compared with the initial data file to obtain the difference file; Determine the target physical slice corresponding to the difference file; Update the data of the target physical slice using the difference file.

[0064] In one embodiment, another implementation of the data management method that the above-mentioned device can achieve includes the following specific steps: The method also includes: In response to receiving a data file deletion command, obtain the storage location of the physical data block corresponding to the data file to be deleted; In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the first storage volume of the first type of hard disk, the physical data block corresponding to the data file to be deleted is marked for deletion, the physical data blocks in the first storage volume that have not been marked for deletion are merged to obtain the updated first storage volume, and the original first storage volume is deleted from the first type of hard disk. In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the second storage volume of the second type of hard disk, the physical data block corresponding to the data to be deleted is marked with holes. In response to receiving the file system release command, the physical data block marked with holes is released.

[0065] For specific limitations regarding the data management device, please refer to the limitations on the data management method above, which will not be repeated here. Each module in the aforementioned data management device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0066] In one embodiment, this application also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium. The computer program includes program instructions, and when the program instructions are executed by a computer, the computer is able to execute the data management methods provided by the above methods.

[0067] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data used in data management methods. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a data management method.

[0068] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0069] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps: Step 101: In response to receiving a file storage instruction, obtain the logical data block corresponding to the file to be processed, encode the logical data block using erasure coding, and obtain the physical data block and the check block. Step 102: Obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size and the preset value. Step 103: In response to the storage location type being the first storage location type, the physical data block and the parity block are stored on the first storage volume of the first type of hard disk; in response to the storage location type being the second storage location type, the physical data block and the parity block are stored on the second storage volume of the second type of hard disk, where the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

[0070] In one embodiment, the processor, when executing a computer program, also performs the following steps: The preset values ​​include a first preset value and a second preset value. The size of the logical data block and the preset value are obtained. Based on the comparison between the size of the logical data block and the preset value, the storage location type of the physical data block and the check block is determined, including: If the size of the logical data block is less than or equal to the first preset value, the physical data block and the check block are appended to the first storage volume of the first type of hard disk. If the size of the logical data block is equal to the second preset value, the physical data block and the check block are appended to the second storage volume of the second type of hard disk.

[0071] In one embodiment, the processor, when executing a computer program, also performs the following steps: The method also includes: Obtain the used storage capacity of the first storage volume of the first type of hard disk, and compare the used storage capacity of the first storage volume of the first type of hard disk with a preset storage threshold; In response to the fact that the used storage capacity of the first storage volume of the first type of hard disk is greater than or equal to a preset storage threshold, the first storage volume is sealed and the sealed storage volume is divided into multiple stripes. The multiple stripes are then encoded with erasure codes and stored in the third storage volume of the second type of hard disk.

[0072] In one embodiment, the processor, when executing a computer program, also performs the following steps: The method also includes: Obtain the metadata descriptor information corresponding to the physical data block. The metadata descriptor information includes the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. The metadata descriptor consists of a metadata identifier, version number, metadata bitmap, serial number, and checksum. The size of the metadata is calculated based on the space occupied by the metadata descriptors and the number of metadata descriptors in each storage volume; Determine whether the entire metadata can be stored in memory based on its size.

[0073] In one embodiment, the processor, when executing a computer program, also performs the following steps: Determining whether the metadata can be fully stored in memory based on its size includes: Get the memory capacity and calculate the ratio of the metadata size to the memory capacity; If the percentage is less than or equal to a preset ratio, the entire metadata will be stored in memory.

[0074] In one embodiment, the processor, when executing a computer program, also performs the following steps: Physical data blocks consist of multiple physical slices, and the method also includes: In response to receiving a file update instruction, determine the initial data file corresponding to the file to be updated; The file to be updated is compared with the initial data file to obtain the difference file; Determine the target physical slice corresponding to the difference file; Update the data of the target physical slice using the difference file.

[0075] In one embodiment, the processor, when executing a computer program, also performs the following steps: The method also includes: In response to receiving a data file deletion command, obtain the storage location of the physical data block corresponding to the data file to be deleted; In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the first storage volume of the first type of hard disk, the physical data block corresponding to the data file to be deleted is marked for deletion, the physical data blocks in the first storage volume that have not been marked for deletion are merged to obtain the updated first storage volume, and the original first storage volume is deleted from the first type of hard disk. In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the second storage volume of the second type of hard disk, the physical data block corresponding to the data to be deleted is marked with holes. In response to receiving the file system release command, the physical data block marked with holes is released.

[0076] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor: Step 101: In response to receiving a file storage instruction, obtain the logical data block corresponding to the file to be processed, encode the logical data block using erasure coding, and obtain the physical data block and the check block. Step 102: Obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size and the preset value. Step 103: In response to the storage location type being the first storage location type, the physical data block and the parity block are stored on the first storage volume of the first type of hard disk; in response to the storage location type being the second storage location type, the physical data block and the parity block are stored on the second storage volume of the second type of hard disk, where the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

[0077] In one embodiment, when the computer program is executed by a processor, it also performs the following steps: The preset values ​​include a first preset value and a second preset value. The size of the logical data block and the preset value are obtained. Based on the comparison between the size of the logical data block and the preset value, the storage location type of the physical data block and the check block is determined, including: If the size of the logical data block is less than or equal to the first preset value, the physical data block and the check block are appended to the first storage volume of the first type of hard disk. If the size of the logical data block is equal to the second preset value, the physical data block and the check block are appended to the second storage volume of the second type of hard disk.

[0078] In one embodiment, when the computer program is executed by a processor, it also performs the following steps: The method also includes: Obtain the used storage capacity of the first storage volume of the first type of hard disk, and compare the used storage capacity of the first storage volume of the first type of hard disk with a preset storage threshold; In response to the fact that the used storage capacity of the first storage volume of the first type of hard disk is greater than or equal to a preset storage threshold, the first storage volume is sealed and the sealed storage volume is divided into multiple stripes. The multiple stripes are then encoded with erasure codes and stored in the third storage volume of the second type of hard disk.

[0079] In one embodiment, when the computer program is executed by a processor, it also performs the following steps: The method also includes: Obtain the metadata descriptor information corresponding to the physical data block. The metadata descriptor information includes the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. The metadata descriptor consists of a metadata identifier, version number, metadata bitmap, serial number, and checksum. The size of the metadata is calculated based on the space occupied by the metadata descriptors and the number of metadata descriptors in each storage volume; Determine whether the entire metadata can be stored in memory based on its size.

[0080] In one embodiment, when the computer program is executed by a processor, it also performs the following steps: Determining whether the metadata can be fully stored in memory based on its size includes: Get the memory capacity and calculate the ratio of the metadata size to the memory capacity; If the percentage is less than or equal to a preset ratio, the entire metadata will be stored in memory.

[0081] In one embodiment, when the computer program is executed by a processor, it also performs the following steps: Physical data blocks consist of multiple physical slices, and the method also includes: In response to receiving a file update instruction, determine the initial data file corresponding to the file to be updated; The file to be updated is compared with the initial data file to obtain the difference file; Determine the target physical slice corresponding to the difference file; Update the data of the target physical slice using the difference file.

[0082] In one embodiment, when the computer program is executed by a processor, it also performs the following steps: The method also includes: In response to receiving a data file deletion command, obtain the storage location of the physical data block corresponding to the data file to be deleted; In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the first storage volume of the first type of hard disk, the physical data block corresponding to the data file to be deleted is marked for deletion, the physical data blocks in the first storage volume that have not been marked for deletion are merged to obtain the updated first storage volume, and the original first storage volume is deleted from the first type of hard disk. In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the second storage volume of the second type of hard disk, the physical data block corresponding to the data to be deleted is marked with holes. In response to receiving the file system release command, the physical data block marked with holes is released.

[0083] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0084] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0085] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A data management method, characterized in that, include: In response to receiving a file storage instruction, the logical data block corresponding to the file to be processed is obtained, and the logical data block is encoded with erasure coding to obtain a physical data block and a check block. Obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value; In response to the storage location type being a first storage location type, the physical data block and the check block are stored on a first storage volume of a first type of hard disk; in response to the storage location type being a second storage location type, the physical data block and the check block are stored on a second storage volume of a second type of hard disk, wherein the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

2. The method according to claim 1, characterized in that, The preset values ​​include a first preset value and a second preset value. The process of obtaining the size of the logical data block and the preset value, and determining the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value, includes: If the size of the logical data block is less than or equal to the first preset value, the physical data block and the check block are appended to the first storage volume of the first type of hard disk. If the size of the logical data block is equal to the second preset value, the physical data block and the check block are appended to the second storage volume of the second type of hard disk.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the used storage capacity of the first storage volume of the first type of hard disk, and compare the used storage capacity of the first storage volume of the first type of hard disk with a preset storage threshold; In response to the fact that the used storage capacity of the first storage volume of the first type of hard disk is greater than or equal to a preset storage threshold, the first storage volume is sealed and the sealed storage volume is divided into multiple stripes. The multiple stripes are then encoded with erasure codes and stored in the third storage volume of the second type of hard disk.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the metadata descriptor information corresponding to the physical data block. The metadata descriptor information includes the space occupied by the metadata descriptor and the number of metadata descriptors in each storage volume. The metadata descriptor consists of a metadata identifier, version number, metadata bitmap, serial number, and checksum. The size of the metadata is calculated based on the space occupied by the metadata descriptors and the number of metadata descriptors in each storage volume; The size of the metadata determines whether the entire metadata can be stored in memory.

5. The method according to claim 4, characterized in that, The step of determining whether the metadata can be fully stored in memory based on its size includes: Obtain the memory capacity and calculate the ratio of the metadata size to the memory capacity; If the percentage is less than or equal to a preset ratio, the metadata is stored in memory.

6. The method according to claim 1, characterized in that, The physical data block comprises multiple physical slices, and the method further includes: In response to receiving a file update instruction, determine the initial data file corresponding to the file to be updated; The file to be updated is compared with the initial data file to obtain the difference file; Determine the target physical slice corresponding to the difference file; Update the data of the target physical slice using the difference file.

7. The method according to claim 1, characterized in that, The method further includes: In response to receiving a data file deletion command, obtain the storage location of the physical data block corresponding to the data file to be deleted; In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the first storage volume of the first type of hard disk, the physical data block corresponding to the data file to be deleted is marked for deletion, the physical data blocks in the first storage volume that have not been marked for deletion are merged to obtain an updated first storage volume, and the original first storage volume is deleted from the first type of hard disk. In response to the fact that the physical data block corresponding to the data file to be deleted is stored in the second storage volume of the second type of hard disk, the physical data block corresponding to the data to be deleted is marked with holes. In response to receiving the file system release command, the physical data block marked with holes is released.

8. A data management device, characterized in that, The device includes: The acquisition module is used to, in response to receiving a file storage instruction, acquire the logical data block corresponding to the file to be processed, encode the logical data block using erasure coding, and obtain the physical data block and the check block. The determination module is used to obtain the size and preset value of the logical data block, and determine the storage location type of the physical data block and the check block based on the comparison result between the size of the logical data block and the preset value. A storage module is configured to, in response to the storage location type being a first storage location type, store the physical data block and the check block on a first storage volume of a first type of hard disk; and in response to the storage location type being a second storage location type, store the physical data block and the check block on a second storage volume of a second type of hard disk, wherein the first type of hard disk is a solid-state drive and the second type of hard disk is a mechanical hard disk.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images