Data storage method and device, electronic device and program product

Abstract

The present invention discloses a data storage method, device, electronic device, and program product, relating to the field of data processing technology. The method comprises: upon detecting that a target client has written target data, splitting the target data into multiple sub-data of preset lengths according to a preset data format; determining preset protection information for each sub-data based on a preset data structure; appending the preset protection information to the end of the sub-data to generate multiple target sub-data; determining a storage disk for each target sub-data based on a volume identifier and a data type; and storing each target sub-data on the storage disk. The present invention solves the technical problem of the inability to effectively verify data in related technologies.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data storage method and device, electronic equipment, and program product. Background Art

[0002] To enhance the integrity and consistency of data in the storage system, current data storage systems attach additional protection information (DIF) to each data block. This information is referred to as the Data Integrity Field (DIF) to detect and prevent data tampering or corruption during transmission.

[0003] The current DIF adds 8 bytes of protection information to each data block (such as a 512-byte or 4096-byte sector). Figure 1 is a schematic diagram of an optional DIF structure according to the related art, such as Figure 1 As shown, an 8-byte DIF can be added after the data. The main components of these 8 bytes are: 2-byte Logical Block Guard (that is, the CRC (Cyclic Redundancy Check) checksum of the data block, used to detect data integrity); 2-byte Logical Block Application Tag (an application tag defined by the application or operating system for specific purposes); 4-byte Logical Block Reference Tag (usually records the lower 32 bits of the logical block address of the sector written to ensure that the data is written to the correct physical sector).

[0004] However, the current DIF is relatively simple in data integrity verification and can only ensure the legitimacy of the currently written 512 or 4K data. It is generally used for data communication verification or judgment of silent data changes. It cannot guarantee the validity of end-to-end data at the business level, nor can it describe the relevant usage characteristics of the data, resulting in low accuracy in data verification and low efficiency in data storage.

[0005] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0006] The embodiments of the present invention provide a data storage method and device, an electronic device, and a program product, so as to at least solve the technical problem in the related art that data cannot be effectively verified.

[0007] According to one aspect of an embodiment of the present invention, a data storage method is provided, which is applied to a distributed block storage system, comprising: upon detecting that a target client writes target data, splitting the target data into multiple sub-data of preset length according to a preset data format; determining preset protection information for each sub-data based on a preset data structure, and splicing the preset protection information behind the sub-data to generate multiple target sub-data, wherein the preset protection information includes at least: a volume identifier and a data type; determining a storage disk for each target sub-data based on the volume identifier and the data type, wherein the storage disk is located in the back-end storage of the distributed block storage system; and storing each target sub-data on the storage disk.

[0008] Furthermore, based on the preset data structure, the step of determining preset protection information for each sub-data includes: when the target client writes the target data to the front-end storage of the distributed block storage system, obtaining metadata of the target data; based on the metadata, determining the volume identifier and current version information of the target data; based on the volume identifier, determining the data type of the target data; and based on the volume identifier, the current version information and the data type, determining the preset protection information.

[0009] Furthermore, the step of determining the storage disk of each target sub-data based on the volume identification and data type includes: determining the storage pool set of the target sub-data based on the volume identification, wherein the storage pool set includes: a first-class pool, a second-class pool, and a third-class pool, the access speed of the first-class pool is greater than that of the second-class pool, and the access speed of the second-class pool is greater than that of the third-class pool; determining the target pool from the storage pool set based on the data type, wherein the target pool includes: multiple disks; determining the storage disk from the target pool based on the logical address in the preset protection information.

[0010] Furthermore, based on the data type, the step of determining the target pool from the storage pool set includes: when the data type is a first type, determining the first type of pool in the storage pool set as the target pool; when the data type is a second type, determining the second type of pool in the storage pool set as the target pool, wherein the frequency of accessing the target sub-data indicated by the first type is greater than the target sub-data indicated by the second type; when the data type is a third type, determining the third type of pool in the storage pool set as the target pool, wherein the frequency of accessing the target sub-data indicated by the second type is greater than the target sub-data indicated by the third type.

[0011] Furthermore, the step of storing each target sub-data in the storage disk includes: when the storage capacity of the front-end storage of the distributed block storage system reaches a preset storage threshold, caching the preset protection information of the target sub-data in the preset cache; reading the volume identifier and data type from the preset cache; and storing the target sub-data and the preset protection information in the storage disk based on the volume identifier and data type.

[0012] Furthermore, the process of storing each target sub-data to the storage disk also includes: when the number of times the same storage disk under the volume indicated by the volume identifier is written reaches a preset number, reading the current version information in the preset protection information; comparing the current version information written to the storage disk each time, and retaining the target sub-data indicated by the latest version information in the storage disk.

[0013] Furthermore, before determining the preset protection information of each sub-data based on the preset data structure, it also includes: adding multiple preset byte fields to the initial data structure to obtain a preset data structure, wherein the preset data structure includes at least: a check field, a version information field, a volume information field, a logical address field, and a data type field.

[0014] According to another aspect of an embodiment of the present invention, a data storage device is provided, which is applied to a distributed block storage system, including: a splitting unit, for splitting the target data into multiple sub-data of preset length according to a preset data format when detecting that a target client writes target data; a first determination unit, for determining preset protection information for each sub-data based on a preset data structure, and splicing the preset protection information behind the sub-data to generate multiple target sub-data, wherein the preset protection information includes at least: a volume identifier and a data type; a second determination unit, for determining a storage disk for each target sub-data based on the volume identifier and the data type, wherein the storage disk is located in the back-end storage of the distributed block storage system; and a storage unit, for storing each target sub-data in the storage disk.

[0015] Furthermore, the first determination unit includes: a first acquisition module, used to obtain metadata of the target data when the target client writes the target data to the front-end storage of the distributed block storage system; a first determination module, used to determine the volume identification and current version information of the target data based on the metadata; a second determination module, used to determine the data type of the target data based on the volume identification; and a third determination module, used to determine preset protection information based on the volume identification, current version information and data type.

[0016] Furthermore, the second determination unit includes: a fourth determination module, used to determine the storage pool set of the target sub-data based on the volume identifier, wherein the storage pool set includes: a first-class pool, a second-class pool, and a third-class pool, the access speed of the first-class pool is greater than that of the second-class pool, and the access speed of the second-class pool is greater than that of the third-class pool; a fifth determination module, used to determine the target pool from the storage pool set based on the data type, wherein the target pool includes: multiple disks; a sixth determination module, used to determine the storage disk from the target pool based on the logical address in the preset protection information.

[0017] Furthermore, the fifth determination module includes: a first determination submodule, used to determine the first type of pool in the storage pool set as the target pool when the data type is the first type; a second determination submodule, used to determine the second type of pool in the storage pool set as the target pool when the data type is the second type, wherein the frequency of accessing the target sub-data indicated by the first type is greater than the target sub-data indicated by the second type; and a third determination submodule, used to determine the third type of pool in the storage pool set as the target pool when the data type is the third type, wherein the frequency of accessing the target sub-data indicated by the second type is greater than the target sub-data indicated by the third type.

[0018] Furthermore, the storage unit includes: a first cache module, used to cache the preset protection information of the target sub-data to a preset cache when the storage capacity of the front-end storage of the distributed block storage system reaches a preset storage threshold; a first reading module, used to read the volume identifier and data type from the preset cache; and a first storage module, used to store the target sub-data and the preset protection information to the storage disk based on the volume identifier and data type.

[0019] Furthermore, the storage device also includes: a second reading module, which is used to read the current version information in the preset protection information when the number of times the same storage disk under the volume indicated by the volume identifier is written reaches a preset number during the process of storing each target sub-data to the storage disk; a first comparison module, which is used to compare the current version information written to the storage disk each time, and retain the target sub-data indicated by the latest version information in the storage disk.

[0020] Furthermore, the storage device also includes: a first adding module, which is used to add multiple preset byte fields to the initial data structure before determining the preset protection information of each sub-data based on the preset data structure to obtain a preset data structure, wherein the preset data structure includes at least: a check field, a version information field, a volume information field, a logical address field, and a data type field.

[0021] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above data storage methods.

[0022] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement any of the above-mentioned data storage methods.

[0023] In the present invention, when it is detected that the target client writes the target data, the target data is split into multiple sub-data of preset length according to the preset data format, and the preset protection information of each sub-data is determined based on the preset data structure, and the preset protection information is spliced ​​to the back of the sub-data to generate multiple target sub-data. Based on the volume identifier and data type, the storage disk of each target sub-data is determined, and each target sub-data is stored in the storage disk, thereby solving the technical problem in the related art that the data cannot be effectively verified.

[0024] In the present invention, an enhanced data integrity storage strategy is adopted. By appending preset protection information (i.e., protection information determined according to the improved DIF structure (i.e., preset data structure) to the end of each data block of a preset length, the purpose of comprehensively verifying end-to-end data integrity is achieved. The storage disk for each target sub-data can then be accurately determined based on the volume identifier and data type of the preset protection information to achieve accurate data storage, thereby achieving the technical effect of improving data legitimacy and storage efficiency in the data storage system. This further solves the technical problem that the current DIF can only detect the legitimacy of the currently written data block, but cannot guarantee the validity of business-level data, resulting in low data playback efficiency and an inability to optimize storage based on data characteristics. The enhanced DIF design not only ensures data accuracy, but also optimizes the data playback process, achieves efficient classified storage of data, reduces storage costs, and improves system performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0026] Figure 1 is a schematic diagram of an optional DIF structure according to the related art;

[0027] Figure 2 is a flow chart of an optional data storage method according to an embodiment of the present invention;

[0028] Figure 3 is a schematic diagram of an optional distributed block storage system structure according to an embodiment of the present invention;

[0029] Figure 4 is a schematic diagram of an optional data storage structure according to an embodiment of the present invention;

[0030] Figure 5 is a schematic diagram of an optional enhanced DIF data structure according to an embodiment of the present invention;

[0031] Figure 6is a schematic diagram of an optional data storage device according to an embodiment of the present invention;

[0032] Figure 7 This is a hardware structure block diagram of an electronic device (or mobile device) for a data storage method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," and the like in the present invention are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0035] It should be noted that the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected and involved in the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with the relevant laws, regulations and standards of the relevant regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation portals for users to choose to authorize or refuse. For example, an interface is set up between this system and the relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain the relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0036] Based on the current 8-byte DIF data structure, the present invention constructs a 32-byte enhanced DIF, adding data-related end-to-end feature information (such as data source information, version information and data type information). In this way, in addition to completing the legitimacy verification of its own data, it can improve the integrity of the back-end storage data verification and enhance the efficiency of data storage.

[0037] The present invention will be described in detail below with reference to various embodiments.

[0038] Example 1

[0039] According to an embodiment of the present invention, an embodiment of a data storage method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0040] Figure 2 is a flow chart of an optional data storage method according to an embodiment of the present invention, such as Figure 2 As shown, the method includes the following steps:

[0041] Step S201 : when it is detected that the target client writes target data, the target data is split into a plurality of sub-data of preset lengths according to a preset data format.

[0042] In an embodiment of the present invention, when a distributed block storage system receives large data write requests from a target client (which could be a server or any data source), the system detects these write operations and then, based on a predefined data format (a predefined data splitting and reassembly method to ensure data can be processed and stored according to design requirements), splits the data into several sub-data blocks of a predetermined length (e.g., 4KB) at the system's front-end storage layer. This process, called "data sharding," enables more efficient data management and protection. For example, if a client sends a 16KB block of data, the system will split it into four 4KB sub-data blocks.

[0043] Step S202: Based on the preset data structure, determine the preset protection information of each sub-data, and splice the preset protection information to the end of the sub-data to generate multiple target sub-data, wherein the preset protection information at least includes: volume identification and data type.

[0044] In this embodiment of the present invention, after data segmentation, the system adds preset protection information based on the data structure of the enhanced DIF (i.e., the preset data structure). This protection information is a 32-byte enhanced DIF, containing fields such as a CRC checksum, data version information (Data version), volume ID, data type (Type), and the data's logical block address (Volume LBA). This information is appended to each sub-data to form the target sub-data. For example, for each 4KB sub-data, the system generates a corresponding 32-byte enhanced DIF.

[0045] The volume ID identifies the logical volume to which the data belongs, ensuring that the correct volume is located during data playback (for back-end storage). The data type describes the access frequency and importance of the data, such as cold data, warm data, or hot data, and is used for categorized back-end data storage.

[0046] Step S203: determining the storage disk of each target sub-data based on the volume identifier and the data type, wherein the storage disk is located in the backend storage of the distributed block storage system.

[0047] In an embodiment of the present invention, based on the volume identifier and data type information contained in the target sub-data, the system can determine which type of storage disk (located in the backend storage of a distributed block storage system, such as a physical storage device or logical storage unit in the backend storage) the data should be stored on. For example, hot data should be flushed back to a hot pool (a high-performance storage pool, such as NVMe disks (Non-Volatile Memory Express Disks)), warm data to a warm pool (a medium-performance storage pool, such as SSD disks (Solid State Drive Disks)), and cold data to a cold pool (a low-performance storage pool, such as HDD disks (Hard Disk Drive Disks)). This ensures data storage efficiency and cost-effectiveness.

[0048] Step S204: store each target sub-data in a storage disk.

[0049] In an embodiment of the present invention, each target sub-data with enhanced DIF will be stored on a designated storage disk in the back-end storage, thereby achieving data persistence, ensuring data security, integrity and classified storage, and improving the overall performance and cost-effectiveness of the system.

[0050] In summary, an enhanced data integrity storage strategy can be adopted to achieve the purpose of comprehensive end-to-end data integrity verification by appending preset protection information (i.e., protection information determined according to the improved DIF structure (i.e., preset data structure)) to the end of each data block of a preset length. Then, the storage disk of each target sub-data can be accurately determined based on the volume identifier and data type of the preset protection information to achieve accurate data storage, thereby achieving the technical effect of improving the data legitimacy and storage efficiency in the data storage system, and further solving the technical problem that the current DIF can only detect the legitimacy of the currently written data block and cannot guarantee the validity of the business-level data, resulting in low data playback efficiency and the inability to optimize storage according to data characteristics. Through the enhanced DIF design, not only the accuracy of the data is ensured, but also the data playback process is optimized, and efficient classified storage of data is achieved, which reduces storage costs and improves system performance.

[0051] In order to improve the accuracy of determining preset protection information, in the data storage method provided in the first embodiment of the present application, when the target client writes target data to the front-end storage of the distributed block storage system, the metadata of the target data is obtained; based on the metadata, the volume identifier and current version information of the target data are determined; based on the volume identifier, the data type of the target data is determined; and based on the volume identifier, the current version information and the data type, the preset protection information is determined.

[0052] In the embodiment of the present invention, the distributed block storage system is a pre-built storage system, including: a protocol layer, a volume layer, a front-end storage, and a back-end storage. Figure 3 FIG. 1 is a schematic diagram of an optional distributed block storage system structure according to an embodiment of the present invention. Figure 3As shown in the figure, the distributed block storage system consists of four layers: the first layer is the data protocol layer, which provides block devices (i.e., devices storing data) to clients through protocols such as ISCSI (a network protocol), NVMF (i.e., a storage protocol), or Vhost (i.e., an efficient storage access protocol); the second layer is the volume management layer (volume layer), for example, Volume1, Volume2, and Volume3, which is used to manage the mapping information between the logical space of data and the physical space; the third layer is the front-end storage, including LogStore (log storage area), which is used for high-performance storage to quickly complete data storage and return it to the client. The front-end storage can be either power-saving memory or NVMe (Non-Volatile Memory Express) high-performance distributed storage; the fourth layer is the back-end storage. For data security and balancing storage performance and cost, data can be placed on storage with higher redundancy and stored in corresponding pools based on the access characteristics of the data. Generally, higher performance means higher cost. For example, the hot pool (high-performance pool) corresponds to NVMe Disks storage disks, warm pools (pools with normal performance) correspond to SSD Disks storage disks, and cold pools (pools with relatively poor performance) correspond to HDD Disks storage disks.

[0053] In this embodiment of the present invention, when a target client begins sending data write requests to a distributed block storage system, the system first captures and analyzes these requests, extracting metadata about the target data. This metadata includes descriptive information about the data, such as its size, creation time, and file type. It also includes metadata related to data storage and management, such as the volume ownership and version information. For example, if the target client sends a write request for a file, the system will analyze the write request and retrieve the file-related metadata.

[0054] Then, based on the metadata, the logical volume (volume ID) to which the target data belongs and the current version information of the data can be determined. The volume ID is a unique identifier used to distinguish different logical volumes. The version information records the write time or serial number of the data and is used to track the change history of the data. For example, if the target data belongs to volume 1 and is the most recently written data in the volume, the system will mark the volume ID of the target data as "Volume 1" and record its version information as "Version 10". After determining the volume ID of the target data, the type of data can be further determined based on this information. The data type can be cold data, warm data, or hot data, depending on the access frequency and importance of the data. For example, if the volume ID indicates that the data comes from a frequently accessed volume, the data type may be marked as "hot data".

[0055] In some optional embodiments, the data type is pre-associated with the volume identifier. For example, the data of a certain volume is hot data, and the data of a certain volume is cold data. The data type can also be specified when the client writes the target data.

[0056] In this embodiment of the present invention, the volume identifier, current version information, and data type are integrated to generate preset protection information. This information is appended to the target data as part of the target data. For example, the system-generated preset protection information for the target data may include: "Volume 1" (volume identifier), "Version 10" (current version information), "Hot" (hot data type), and a calculated CRC checksum.

[0057] For example, when the foreground data (i.e., data written by the client) is written to the front-end storage, it can be split according to a preset data format (i.e., the preset data length and the length of the information attached to each data). For example, if it is split according to 4K data, a 32-byte buffer space (increased space) is inserted after the 4K data to store information related to the enhanced DIF data. Figure 4 is a schematic diagram of an optional data storage structure according to an embodiment of the present invention, such as Figure 4 As shown in the figure, each 4K data (such as Data1, Data2, ..., DataN) is attached with Footer DIF (i.e. enhanced DIF) information, including: volume-related information, version information and data type. The data type can be bound to a certain volume (for example, the data of a certain volume is all hot data, and the data of a certain volume is all cold data), or it can be specified by the client when sending data. Finally, the data of this 32-bit enhanced DIF is written to the storage disk together with the 4K data.

[0058] In this embodiment, metadata is first obtained, and then the volume identifier, version information, and data type of the target data are determined based on the metadata, and finally preset protection information (i.e., enhanced DIF) is generated to ensure that the storage of data in the distributed block storage system is not only safe and reliable, but also highly optimized. The storage location can be intelligently selected based on the characteristics and version history of the data. This not only improves the data reading and writing efficiency and reduces storage costs, but also enhances the integrity and consistency protection of the data.

[0059] In order to improve the accuracy of determining the storage disk for each target sub-data, in the data storage method provided in Example 1 of the present application, a storage pool set of the target sub-data is determined based on the volume identifier, wherein the storage pool set includes: a first-class pool, a second-class pool, and a third-class pool, the access speed of the first-class pool is greater than that of the second-class pool, and the access speed of the second-class pool is greater than that of the third-class pool; based on the data type, a target pool is determined from the storage pool set, wherein the target pool includes: multiple disks; based on the logical address in the preset protection information, a storage disk is determined from the target pool.

[0060] In this embodiment of the present invention, after the system identifies the logical volume to which the target subdata belongs based on its volume ID, it further queries the storage pool configuration to determine a set of storage pools that matches the volume ID. This set consists of three different types of storage pools: a first-type pool (a high-speed access pool (hot pool)), a second-type pool (a medium-speed access pool (warm pool)), and a third-type pool (a low-speed access pool (cold pool)). Each type of pool consists of multiple physical or logical disks, with access speeds decreasing in descending order. The data type field (Type) in the preset protection information is then analyzed to determine whether the target subdata is cold, warm, or hot data. Based on the data type, the system selects an appropriate storage pool from the set of storage pools as the target pool. For example, if the target subdata is marked as hot, the system selects the first-type pool (a high-speed access pool) as the target pool; if it is warm, the system selects the second-type pool; if it is cold, the system selects the third-type pool.

[0061] In this embodiment of the present invention, the preset protection information includes the logical address (VolumeLBA) of the target subdata, which identifies the data's location within the logical volume. The system uses this logical address, combined with information about the target pool, to determine the specific storage disk where the target subdata should be stored. For example, the logical address can be mapped to a physical address to locate the appropriate physical disk within the target pool to store the data. For example, if the target pool is a high-speed access pool, the system will use the logical address to locate the most available and optimized NVMe disks within the pool to store the target subdata.

[0062] In this embodiment, an intelligent and efficient data storage strategy is implemented by determining a set of storage pools, selecting a target pool based on data type, and locating specific storage disks based on logical addresses. This strategy fully considers data access characteristics, classifying and storing it on the most appropriate storage medium. This improves data read and write speeds, reduces storage costs, and enhances data management and access performance. By precisely locating storage locations, data storage efficiency is improved and storage resource allocation is optimized.

[0063] In order to improve the accuracy of determining the target pool, in the data storage method provided in Example 1 of the present application, when the data type is the first type, the first type of pool in the storage pool set is determined as the target pool; when the data type is the second type, the second type of pool in the storage pool set is determined as the target pool, wherein the frequency of accessing the target sub-data indicated by the first type is greater than the target sub-data indicated by the second type; when the data type is the third type, the third type of pool in the storage pool set is determined as the target pool, wherein the frequency of accessing the target sub-data indicated by the second type is greater than the target sub-data indicated by the third type.

[0064] In this embodiment of the present invention, when a data write operation occurs, the system first parses the data type field (Type) in the enhanced DIF to determine the type of the target sub-data: Type 1 (hot data), Type 2 (warm data), or Type 3 (cold data). This classification is based on the expected frequency of future access to the data.

[0065] In this embodiment of the present invention, if the target sub-data is marked as type 1 (hot data), the system searches for the first type of pool (high-speed access pool) in the storage pool set. The first type of pool consists of a series of high-performance storage devices, such as NVMe disks, which provide the fastest access speeds to meet the performance requirements of frequent read and write operations for hot data. After determining that the first type of pool is the target pool for the hot data, the system selects an appropriate high-speed storage disk from the first type of pool to store the target sub-data based on the logical address information in the enhanced DIF.

[0066] In this embodiment of the present invention, for target sub-data marked as type 2 (warm data), the system locates the second-type pool (medium-speed access pool) in the storage pool set as the target pool. This type 2 pool is typically composed of SSD disks, which have slightly slower access speeds than NVMe, but still superior to traditional hard drives. This makes it suitable for data storage that is accessed frequently but not as frequently as hot data. Based on the logical address of the target sub-data, the system selects a medium-speed storage disk from the second-type pool for persistent data storage.

[0067] In this embodiment of the present invention, when the data type is identified as type 3 (cold data), the system selects a type 3 pool (a low-speed access pool) from the storage pool set as the target pool. This type of pool typically uses lower-cost HDD disks as storage media and is suitable for storing rarely accessed data. Based on the logical address of the target subdata, the system selects a low-speed storage disk in the type 3 pool for the data write operation.

[0068] In this embodiment, intelligent data storage is achieved through meticulous data type classification and matching storage pools. Hot data is placed in the first pool, which offers the fastest response speeds; warm data is stored in the second pool, which offers moderate access speeds; and cold data is stored in the third pool, which is more cost-effective. This strategy not only effectively allocates storage resources at different performance levels, reducing unnecessary data migration, but also stores data on the optimal storage media based on actual data access needs, thereby improving the efficiency, performance, and cost-effectiveness of the storage system.

[0069] In order to accurately store each target sub-data on the storage disk, in the data storage method provided in the first embodiment of the present application, when the storage capacity of the front-end storage of the distributed block storage system reaches a preset storage threshold, the preset protection information of the target sub-data is cached in the preset cache; the volume identifier and data type are read from the preset cache; and based on the volume identifier and data type, the target sub-data and the preset protection information are stored on the storage disk.

[0070] In this embodiment of the present invention, in a distributed block storage system, front-end storage is responsible for receiving and temporarily storing user data. To efficiently manage and utilize back-end storage resources, this embodiment proposes a strategy for migrating target sub-data and its preset protection information (enhanced DIF) to back-end storage when the front-end storage capacity reaches a preset storage threshold (a pre-set upper limit for efficient operation of the front-end storage. Once this threshold is reached, the system triggers data migration).

[0071] In this embodiment of the present invention, when the front-end storage receives continuous data write operations, the system monitors the LogStore storage capacity in real time. If the LogStore storage capacity reaches a preset storage threshold, for example, 80% fullness, the system initiates data migration to ensure that the performance and responsiveness of the front-end storage are not affected by storage space shortages.

[0072] In this embodiment of the present invention, when the data volume reaches a preset threshold, the system caches the 32-byte enhanced DIF information following the target sub-data into a preset cache. This preset cache can be a specific memory area or cache device. The purpose is to temporarily store this critical protection information during data playback and migration to backend storage for subsequent use.

[0073] The volume ID and data type information for each target sub-data can then be read from the preset cache. The volume ID is used to identify the logical volume to which the data belongs, while the data type indicates the access characteristics of the data (cold, warm, hot). The specific back-end storage pool and storage disk where the target sub-data and its preset protection information should be stored are then determined based on the read volume ID and data type information. For example, if the target sub-data is marked as hot data and its volume ID points to a high-speed access pool, the system will extract the target sub-data and its enhanced DIF from the preset cache and then store it on the NVMe disks in the high-speed access pool.

[0074] In this embodiment, when the front-end storage capacity reaches a preset storage threshold, the system automatically caches the preset protection information for the target sub-data and, based on the data's volume identifier and type, stores it on the most appropriate back-end storage medium. This not only avoids excessive use of front-end storage space but also ensures that data is stored in the most cost-effective storage pool based on its access characteristics. This strategy improves data storage efficiency and reduces storage costs. Furthermore, through the use of enhanced DIF, it further ensures data integrity and consistency.

[0075] In order to further improve the efficiency of data storage, in the data storage method provided in Example 1 of the present application, in the process of storing each target sub-data in the storage disk, when the number of times the same storage disk under the volume indicated by the volume identifier is written reaches a preset number of times, the current version information in the preset protection information is read; the current version information of each write to the storage disk is compared, and the target sub-data indicated by the latest version information is retained in the storage disk.

[0076] In an embodiment of the present invention, when the amount of data in the front-end storage reaches a certain threshold and playback and storage of the target sub-data to the back-end storage disk are required, the system begins the storage process. When the system prepares to write each target sub-data to the back-end storage, it checks the volume ID corresponding to the target sub-data and tracks the number of writes to the same storage disk under that volume. It then compares this count with a preset number (a pre-set threshold used to determine whether to read and compare the current version information of the target sub-data). If the number of writes to the same storage disk reaches the preset number, the system reads the data version information (DataVersion) in the preset protection information (i.e., enhanced DIF) of the target sub-data to ensure data version control and prevent redundant storage of old versions of data, thereby saving storage space and improving efficiency. After reading the current version information of multiple target sub-data, the system compares the version information. If multiple data blocks with the same logical block address (Volume LBA) are found, a determination is made based on the data version information of each data block to ensure that only the latest version (with the highest version number) of the target sub-data is retained on the storage disk. In this way, the storage of duplicate data can be avoided, the consumption of storage space can be reduced, and the timeliness and accuracy of the data can be guaranteed.

[0077] For example, when the data stored on the front end reaches a certain level, the data is replayed from the front-end storage, and all the data information of the enhanced DIF is read and cached in the cache (i.e., a high-speed cache). It is classified according to the volume ID information (Volume ID) and inserted into the memory of each volume. If there are multiple overwrites at the same location of the volume, according to the version information (DataVersion) of the enhanced DIF, the metadata and data only need to retain the latest version of the data to reduce the amount of data flushed back, and according to the data type (Type) of the enhanced DIF, the hot data is flushed back to the hot pool, the warm data is flushed back to the warm pool, and the cold data is flushed back to the cold pool.

[0078] In this embodiment, the distributed block storage system effectively identifies and stores the latest version of data, avoiding data redundancy and storage waste caused by multiple overwrites. As data is stored on the storage disk, the system monitors the number of writes to the same disk and reads the current version information of the target sub-data, ensuring that only the latest version of the data is retained. This achieves efficient use of storage space and precise control of data versions. This not only improves storage performance and reduces unnecessary I / O (input and output) operations, but also enhances data integrity and consistency, improving the overall efficiency and reliability of the storage system.

[0079] In order to accurately construct the preset data structure, in the data storage method provided in Example 1 of the present application, before determining the preset protection information of each sub-data based on the preset data structure, it also includes: adding multiple preset byte fields to the initial data structure to obtain a preset data structure, wherein the preset data structure includes at least: a check field, a version information field, a volume information field, a logical address field, and a data type field.

[0080] In the embodiment of the present invention, in order to enhance data integrity and storage efficiency, a plurality of preset byte fields are added to the initial data structure to form a preset data structure (ie, enhanced DIF).

[0081] Specifically, we can first analyze the existing data structure, which is a standard data layout that includes user data and basic protection information such as CRC checksum. These data structures are generally suitable for direct storage operations, but their functionality may be limited in complex and dynamic distributed environments. Then, we add a 32-byte preset field at the end of each data block to form a preset data structure. These 32 bytes include the following fields:

[0082] Check field: CRC checksum of the data, used to ensure the integrity of the data during transmission.

[0083] Version information field: records the version information of data written, used to distinguish the data versions under the same logical address written at different time points.

[0084] Volume information field: records the volume information of the data source, which is used to accurately track the logical ownership of the data and ensure the validity of end-to-end data at the business level.

[0085] Logical address field: records the logical block address of the data on the logical volume, ensuring that the data can be correctly mapped to a specific physical storage location.

[0086] Data type field: describes the access characteristics of the data, for example, cold data, warm data, or hot data.

[0087] The aforementioned fields were then integrated into the original data structure to create an enhanced DIF data structure. This new data structure not only provides basic data protection but also includes rich metadata information, enabling the storage system to perform more refined management and optimization based on data characteristics.

[0088] Figure 5 : is a schematic diagram of an optional enhanced DIF data structure according to an embodiment of the present invention, such as Figure 5As shown, 32 bytes are added to the end of each 512-byte or 4KB data to describe data integrity. The enhanced DIF mainly consists of: a 4-byte CRC (data checksum), an 8-byte Data version (data write version information), an 8-byte Volume ID (data source volume information), an 8-byte Volume LBA (data source volume logical address), a 2-byte Type (indicating data characteristics, such as cold, warm, or hot data), and a 2-byte Reversed (reserved field for future expansion).

[0089] In this embodiment, by adding preset fields to the initial data structure to form a preset data structure (enhanced DIF), the distributed block storage system achieves more comprehensive data protection and more efficient data management. This improvement not only strengthens data integrity verification but also improves data playback efficiency and reduces redundant data storage by introducing features such as version control, volume information, and data type identification. This ensures that data is stored on the most appropriate storage medium based on its access characteristics, thereby optimizing the use of storage resources. Furthermore, the use of enhanced DIF simplifies the storage system's complexity in handling data migration, data recovery, and data query, improving the system's overall performance and cost-effectiveness.

[0090] In an embodiment of the present invention, by adding volume-related information (Volume ID) in the DIF, the end-to-end source of data can be accurately recorded. By adding version information (Data Version) of the end-to-end data in the DIF, the latest written valid data can be identified in a storage system that adopts the append writing method. By adding the characteristics of the end-to-end data type (Type) in the DIF, when flushing, the data can be stored on a storage medium with a higher cost-effectiveness according to the characteristics of the data.

[0091] In some optional embodiments, in addition to being applied to storage systems written by append, the enhanced DIF can also be applied to the field of distributed volume data communication. Through the information of the enhanced DIF, the specified volume information and version can be identified to make some enhanced data legitimacy judgments.

[0092] In this embodiment, by appending an incremental DIF to the data, the integrity of the end-to-end data can be verified without worrying about reading the wrong volume data. Moreover, in a storage system written in append mode, after the volume data is overwritten multiple times, the most recently written valid data can be identified, thereby improving the efficiency of data playback.

[0093] The following describes it in detail with reference to another embodiment.

[0094] Example 2

[0095] A data storage device provided in this embodiment includes multiple implementation units, each implementation unit corresponding to each implementation step in the above-mentioned embodiment 1.

[0096] Figure 6 is a schematic diagram of an optional data storage device according to an embodiment of the present invention, such as Figure 6 As shown, the storage device may include: a splitting unit 60, a first determining unit 61, a second determining unit 62, and a storage unit 63.

[0097] The splitting unit 60 is configured to split the target data into a plurality of sub-data of preset lengths according to a preset data format when detecting that the target client has written the target data;

[0098] The first determining unit 61 is configured to determine preset protection information for each sub-data based on a preset data structure, and append the preset protection information to the end of the sub-data to generate a plurality of target sub-data, wherein the preset protection information includes at least a volume identifier and a data type;

[0099] A second determining unit 62 is configured to determine a storage disk for each target sub-data based on the volume identifier and the data type, wherein the storage disk is located in a backend storage of the distributed block storage system;

[0100] The storage unit 63 is configured to store each target sub-data in a storage disk.

[0101] The above-mentioned storage device can achieve the purpose of comprehensive end-to-end data integrity verification by appending preset protection information (i.e., protection information determined according to the improved DIF structure (i.e., preset data structure)) to the end of each data block of preset length. Then, based on the volume identifier and data type of the preset protection information, the storage disk of each target sub-data can be accurately determined to achieve accurate data storage, thereby achieving the technical effect of improving data legitimacy and storage efficiency in the data storage system. This further solves the technical problem that the current DIF can only detect the legitimacy of the currently written data block and cannot guarantee the validity of business-level data, resulting in low data playback efficiency and the inability to optimize storage according to data characteristics. The enhanced DIF design not only ensures data accuracy, but also optimizes the data playback process, achieves efficient classified storage of data, reduces storage costs, and improves system performance.

[0102] Optionally, the first determination unit includes: a first acquisition module, used to obtain metadata of the target data when the target client writes the target data to the front-end storage of the distributed block storage system; a first determination module, used to determine the volume identification and current version information of the target data based on the metadata; a second determination module, used to determine the data type of the target data based on the volume identification; and a third determination module, used to determine preset protection information based on the volume identification, current version information and data type.

[0103] Optionally, the second determination unit includes: a fourth determination module, used to determine the storage pool set of the target sub-data based on the volume identifier, wherein the storage pool set includes: a first-class pool, a second-class pool, and a third-class pool, the access speed of the first-class pool is greater than that of the second-class pool, and the access speed of the second-class pool is greater than that of the third-class pool; a fifth determination module, used to determine the target pool from the storage pool set based on the data type, wherein the target pool includes: multiple disks; a sixth determination module, used to determine the storage disk from the target pool based on the logical address in the preset protection information.

[0104] Optionally, the fifth determination module includes: a first determination submodule, used to determine the first type of pool in the storage pool set as the target pool when the data type is the first type; a second determination submodule, used to determine the second type of pool in the storage pool set as the target pool when the data type is the second type, wherein the frequency of accessing the target sub-data indicated by the first type is greater than the target sub-data indicated by the second type; and a third determination submodule, used to determine the third type of pool in the storage pool set as the target pool when the data type is the third type, wherein the frequency of accessing the target sub-data indicated by the second type is greater than the target sub-data indicated by the third type.

[0105] Optionally, the storage unit includes: a first cache module, used to cache the preset protection information of the target sub-data to a preset cache when the storage capacity of the front-end storage of the distributed block storage system reaches a preset storage threshold; a first reading module, used to read the volume identifier and data type from the preset cache; and a first storage module, used to store the target sub-data and the preset protection information to the storage disk based on the volume identifier and data type.

[0106] Optionally, the storage device also includes: a second reading module, which is used to read the current version information in the preset protection information when the number of times the same storage disk under the volume indicated by the volume identifier is written reaches a preset number during the process of storing each target sub-data to the storage disk; a first comparison module, which is used to compare the current version information written to the storage disk each time, and retain the target sub-data indicated by the latest version information in the storage disk.

[0107] Optionally, the storage device also includes: a first adding module, which is used to add multiple preset byte fields to the initial data structure before determining the preset protection information of each sub-data based on the preset data structure to obtain a preset data structure, wherein the preset data structure includes at least: a check field, a version information field, a volume information field, a logical address field, and a data type field.

[0108] The above-mentioned storage device can also include a processor and a memory. The above-mentioned splitting unit 60, the first determination unit 61, the second determination unit 62, the storage unit 63, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize the corresponding functions.

[0109] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be provided, and each target sub-data is stored in the storage disk by adjusting the kernel parameters.

[0110] The above-mentioned memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0111] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: when detecting that a target client writes target data, splitting the target data into multiple sub-data of preset length according to a preset data format, determining preset protection information for each sub-data based on a preset data structure, and splicing the preset protection information to the back of the sub-data to generate multiple target sub-data, determining the storage disk for each target sub-data based on a volume identifier and a data type, and storing each target sub-data on the storage disk.

[0112] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements any of the above data storage methods.

[0113] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned data storage method.

[0114] Figure 7 FIG. 1 is a hardware structure block diagram of an electronic device (or mobile device) for a data storage method according to an embodiment of the present invention. Figure 7 As shown, the electronic device may include one or more processors (e.g., Figure 7 Processors 702a, 702b, ..., 702n, etc., which may include but are not limited to processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), and memory 704 for storing data. In addition, the system may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports in the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. It will be understood by those skilled in the art that Figure 7 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 7 More or fewer components than shown, or with Figure 7 Different configurations shown.

[0115] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0116] The embodiments or examples of the present disclosure are not exhaustive, but are merely illustrations of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be arbitrarily combined; in addition, the various embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a certain embodiment or example can be arbitrarily combined with the optional methods or optional examples of other embodiments or examples.

[0117] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0118] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, and can be electrical or other forms.

[0119] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0120] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0121] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), a mobile hard drive, a magnetic disk, or an optical disk.

[0122] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that: Applied to distributed block storage systems, including: When detecting that the target client writes target data, splitting the target data into a plurality of sub-data of preset lengths according to a preset data format; Determining preset protection information for each sub-data based on a preset data structure, and splicing the preset protection information onto the back of the sub-data to generate a plurality of target sub-data, wherein the preset protection information includes at least: a volume identifier and a data type; determining the preset protection information for each sub-data based on the preset data structure includes: obtaining metadata of the target data when the target client writes the target data to the front-end storage of the distributed block storage system; determining the volume identifier and current version information of the target data based on the metadata; determining the data type of the target data based on the volume identifier; and determining the preset protection information based on the volume identifier, the current version information, and the data type; Determining a storage disk for each target sub-data based on the volume identifier and the data type, wherein the storage disk is located in a backend storage of the distributed block storage system; storing each of the target sub-data in the storage disk; The preset protection information is a 32-byte enhanced DIF, which also includes: CRC check, data version information, and logical address of the data.

2. The storage method according to claim 1, wherein: The step of determining a storage disk for each target sub-data based on the volume identifier and the data type includes: Determining a storage pool set for the target sub-data based on the volume identifier, wherein the storage pool set includes: a first-type pool, a second-type pool, and a third-type pool, wherein the access speed of the first-type pool is greater than that of the second-type pool, and the access speed of the second-type pool is greater than that of the third-type pool; Based on the data type, determining a target pool from the storage pool set, wherein the target pool includes: a plurality of disks; The storage disk is determined from the target pool based on the logical address in the preset protection information.

3. The storage method according to claim 2, characterized in that: The step of determining a target pool from the set of storage pools based on the data type includes: In a case where the data type is a first type, determining the first type pool in the storage pool set as the target pool; In a case where the data type is a second type, determining the second type pool in the storage pool set as the target pool, wherein a frequency of accessing the target sub-data indicated by the first type is greater than a frequency of accessing the target sub-data indicated by the second type; When the data type is the third type, the third type pool in the storage pool set is determined as the target pool, wherein the frequency of accessing the target sub-data indicated by the second type is greater than the frequency of accessing the target sub-data indicated by the third type.

4. The storage method according to claim 1, wherein: The step of storing each target sub-data to the storage disk comprises: When the storage capacity of the front-end storage of the distributed block storage system reaches a preset storage threshold, caching the preset protection information of the target sub-data into a preset cache; Reading the volume identifier and the data type from the preset cache; Based on the volume identifier and the data type, the target sub-data and the preset protection information are stored in the storage disk.

5. The storage method according to claim 1, wherein: The process of storing each target sub-data in the storage disk further includes: When the number of times the same storage disk under the volume indicated by the volume identifier has been written reaches a preset number, reading the current version information in the preset protection information; The current version information written into the storage disk each time is compared, and the target sub-data indicated by the latest version information is retained in the storage disk.

6. The storage method according to claim 1, wherein: Before determining the preset protection information of each sub-data based on the preset data structure, the method further includes: A plurality of preset byte fields are added to the initial data structure to obtain the preset data structure, wherein the preset data structure at least includes: a check field, a version information field, a volume information field, a logical address field, and a data type field.

7. A data storage device, characterized in that: Applied to distributed block storage systems, including: a splitting unit, configured to split the target data into a plurality of sub-data of preset lengths according to a preset data format when detecting that the target client has written target data; A first determining unit is configured to determine preset protection information for each sub-data based on a preset data structure, and to concatenate the preset protection information onto the back of the sub-data to generate a plurality of target sub-data, wherein the preset protection information includes at least a volume identifier and a data type; determining the preset protection information for each sub-data based on the preset data structure includes: obtaining metadata of the target data when the target client writes the target data to the front-end storage of the distributed block storage system; determining the volume identifier and current version information of the target data based on the metadata; determining the data type of the target data based on the volume identifier; and determining the preset protection information based on the volume identifier, the current version information, and the data type; a second determining unit, configured to determine a storage disk of each target sub-data based on the volume identifier and the data type, wherein the storage disk is located in a backend storage of the distributed block storage system; A storage unit, configured to store each of the target sub-data in the storage disk; The preset protection information is a 32-byte enhanced DIF, which also includes: CRC check, data version information, and logical address of the data.

8. A computer program product, characterized in that The invention comprises a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the data storage method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: The device comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data storage method described in any one of claims 1 to 6.

Images