Data storage method and device of distributed system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a data storage method and device of a distributed system, electronic equipment and a storage medium, and relates to the technical field of IT and software, the method comprises the steps that a target data packet is received, the target data packet comprises a target data block and target verification information corresponding to the target data block, and the target verification information corresponds to the target data block; the local data file comprises a verification storage space; and obtaining a block relation between the target data block and the last data block in the local data file, writing the target data block into the local data file according to the block relation, and writing the target verification information into the verification storage space, so that the data storage structure of the data file is optimized, and the verification efficiency of the data file is improved. According to the method, the verification storage space used for storing the verification information is reserved in the data file, and the data and the corresponding verification information are stored in the same file, so that the performance of data disk operation is improved, the time consumed for writing the data file is saved, the number of opened file streams is reduced, and the management overhead of a file system is reduced.

Description

Technical Field

[0001] This invention relates to the fields of IT and software technology, and in particular to a data storage method for a distributed system, a data storage device for a distributed system, an electronic device, and a computer-readable storage medium. Background Technology

[0002] In a distributed file system, to ensure data integrity and consistency, each data block is typically accompanied by a corresponding CRC (Cyclic Redundancy Check) record to verify whether data corruption has occurred during inter-node transmission. Typically, the CRC and other metadata information for a data block are stored in a separate metadata file, while the actual data is stored in the data file. The primary function of the metadata file is to record relevant information about the data block, helping the system manage, verify, and access data. The data persistence operation involves writing the data block and its metadata to disk to ensure data durability and consistency.

[0003] However, since CRC information is only stored in the metadata file, writing data to disk requires not only writing the data file to disk but also writing an additional metadata file storing the CRC. In large-scale storage scenarios, the efficiency of data write-to-disk operations directly impacts the performance and stability of the entire system. Especially in multi-threaded environments, performing a large number of data write-to-disk operations simultaneously can lead to competition and conflicts between data nodes, thus affecting write-to-disk efficiency. Furthermore, inefficient disk write-to-disk operations consume more CPU and memory resources, causing I / O operation latency and potentially blocking the request queue in high-concurrency scenarios, thereby reducing overall system performance. Summary of the Invention

[0004] This invention provides a data storage method, apparatus, electronic device, and computer-readable storage medium for distributed systems, to solve or partially solve the problems of low data write-to-disk efficiency, high system management overhead, and poor performance under high performance and high concurrency pressure.

[0005] This invention discloses a data storage method for a distributed system, comprising: A target data packet for a local data file is received. The target data packet includes at least a target data block and target verification information corresponding to the target data block. The local data file includes at least a verification storage space for storing the verification information. Obtain the block relationship between the target data block and the last data block in the local data file, write the target data block into the local data file according to the block relationship, and write the target verification information into the verification storage space.

[0006] In this embodiment of the invention, by optimizing the data storage structure of the data file and reserving verification storage space in the data file for storing verification information, the performance of data write-to-disk operation is improved when high-concurrency read and write operations are performed in a large-scale storage cluster. This saves the time spent writing data files and also reduces the number of open file streams, thereby reducing the management overhead of the file system.

[0007] Among some feasible implementation methods are: Obtain the file parameters corresponding to the local data file, wherein the file parameters include at least the data capacity, the block size of the data blocks, and the size of the verification information; Based on the data capacity, the block size, and the information size, the verification storage space for storing verification information is calculated in the local data file.

[0008] In this embodiment of the invention, by optimizing the data storage structure, during the data write-to-disk process, data nodes can write data blocks and corresponding verification information into the same file, thereby improving the performance of the data write-to-disk operation, saving the time spent writing data files, and reducing the number of open file streams, thus reducing the management overhead of the file system.

[0009] In some feasible implementations, the verification storage space for storing verification information is calculated in the local data file based on the data capacity, the block size, and the information size, using the following formula: Check Sum Disk Len=ceil(Block Size / Chunk Size)*Bytes Per Check Sum Wherein, Check Sum Disk Len is the size of the check storage space, Block Size is the data capacity of the data file, Chunk Size is the block size required to generate a check message, and BytesPer Check Sum is the information size of each check message.

[0010] In this embodiment of the invention, by optimizing the data storage structure, during the data write-to-disk process, data nodes can write data blocks and corresponding verification information into the same file, thereby improving the performance of the data write-to-disk operation, saving the time spent writing data files, and reducing the number of open file streams, thus reducing the management overhead of the file system.

[0011] In some feasible implementations, the local data file further includes a data storage space for storing data blocks, wherein writing the target data blocks into the local data file according to the block relationship, and writing the target verification information into the verification storage space, includes: If the target data block belongs to the same block as the last data block in the local data file, and the target data block is a complete block, then the target data block is truncated, and the truncated data is appended to the data storage space of the local data file. Based on the first write position of the target data block in the data storage space, determine the first offset address of the first write position in the verification storage space; The target verification information is written into the verification storage space according to the first offset address, overwriting the currently stored verification information.

[0012] In this embodiment of the invention, data retransmission and correction scenarios are effectively handled, avoiding waste of storage space, while the real-time nature of verification information is ensured through precise address mapping.

[0013] In some feasible implementations, the step of writing the target data into the local data file in blocks according to the block relationship, and writing the target verification information into the verification storage space, includes: If the target data block is not a complete block, and it is a continuous block with the last data block in the local data file, then the target data block is appended to the data storage space of the local data file, and the last data block in the local data file is merged with the target data block into a merged block. Based on the second write position of the merged block in the data storage space, determine the second offset address of the second write position in the verification storage space; The target verification information is used to calculate the merge verification information corresponding to the merged block; The merged verification information is written into the verification storage space according to the second offset address.

[0014] In this embodiment of the invention, boundary cases in streaming writing are effectively handled. Through dynamic merging and recalculation, it is effectively ensured that data writing of any size can generate correct verification information, thus guaranteeing the validity and accuracy of data writing.

[0015] In some feasible implementations, the step of writing the target data into the local data file in blocks according to the block relationship, and writing the target verification information into the verification storage space, includes: If the target data block is a complete block and does not belong to the same block as the last data block in the local data file, then the target data block is appended to the data storage space of the local data file. Based on the third write position of the target data block in the data storage space, determine the third offset address of the third write position in the verification storage space; The target verification information is written into the verification storage space according to the third offset address.

[0016] In this embodiment of the invention, since the data block belongs to a complete computation block, the data node does not need to introduce any additional computational overhead during the writing process, and can be directly written to the local data file in the predetermined writing order, thus ensuring the efficiency of data writing.

[0017] Among some feasible implementation methods are: Update the metadata information in the local data file, the metadata information including at least the current write length of the data storage space and the current write status of the verification space.

[0018] In this embodiment of the invention, updating metadata helps to reduce the management overhead of the file system.

[0019] This invention also discloses a data storage device for a distributed system, comprising: A data packet receiving module is used to receive a target data packet for a local data file. The target data packet includes at least a target data block and target verification information corresponding to the target data block. The local data file includes at least a verification storage space for storing the verification information. The data writing module is used to obtain the block relationship between the target data block and the last data block in the local data file, and to write the target data block into the local data file according to the block relationship, and to write the target verification information into the verification storage space.

[0020] Among some feasible implementation methods are: The file parameter acquisition module is used to acquire the file parameters corresponding to the local data file. The file parameters include at least the data capacity, the block size of the data blocks, and the size of the verification information. The storage space calculation module is used to calculate the verification storage space for storing verification information in the local data file based on the data capacity, the block size, and the information size.

[0021] In some feasible implementations, the storage space calculation module can be specifically implemented using the following formula: Check Sum Disk Len=ceil(Block Size / Chunk Size)*Bytes Per Check Sum Wherein, Check Sum Disk Len is the size of the check storage space, Block Size is the data capacity of the data file, Chunk Size is the block size required to generate a check message, and BytesPer Check Sum is the information size of each check message.

[0022] In some feasible implementations, the local data file further includes data storage space for storing data blocks, and the data writing module is specifically used for: If the target data block belongs to the same block as the last data block in the local data file, and the target data block is a complete block, then the target data block is truncated, and the truncated data is appended to the data storage space of the local data file. Based on the first write position of the target data block in the data storage space, determine the first offset address of the first write position in the verification storage space; The target verification information is written into the verification storage space according to the first offset address, overwriting the currently stored verification information.

[0023] In some feasible implementations, the data writing module is specifically used for: If the target data block is not a complete block, and it is a continuous block with the last data block in the local data file, then the target data block is appended to the data storage space of the local data file, and the last data block in the local data file is merged with the target data block into a merged block. Based on the second write position of the merged block in the data storage space, determine the second offset address of the second write position in the verification storage space; The target verification information is used to calculate the merge verification information corresponding to the merged block; The merged verification information is written into the verification storage space according to the second offset address.

[0024] In some feasible implementations, the data writing module is specifically used for: If the target data block is a complete block and does not belong to the same block as the last data block in the local data file, then the target data block is appended to the data storage space of the local data file. Based on the third write position of the target data block in the data storage space, determine the third offset address of the third write position in the verification storage space; The target verification information is written into the verification storage space according to the third offset address.

[0025] Among some feasible implementation methods are: An update module is used to update the metadata information in the local data file. The metadata information includes at least the current write length of the data storage space and the current write status of the verification space.

[0026] This invention also discloses an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; The memory is used to store computer programs; When the processor executes a program stored in the memory, it implements the method described in the embodiments of the present invention.

[0027] This invention also discloses a computer-readable storage medium storing instructions that, when executed by one or more processors, cause the processors to perform the methods described in this invention.

[0028] The embodiments of the present invention have the following advantages: In this embodiment of the invention, when a data node receives a target data packet for a local data file, it can parse the corresponding target data block and the target verification information corresponding to the target data block from the target data packet. The local data file includes a verification storage space for storing verification information. During the data writing process, the data node can obtain the block relationship between the target data block and the last data block in the local data file, and write the target data block into the local data file and the target verification information into the verification storage space according to the block relationship. By optimizing the data storage structure of the data file and reserving verification storage space for storing verification information in the data file, the performance of data write operations can be improved when high-concurrency read and write operations are performed in a large-scale storage cluster. By storing the data and the corresponding verification information in the same file, the writing time of the data file can be saved. In addition, the number of open file streams can be reduced, and the management overhead of the file system can be reduced. Attached Figure Description

[0029] Figure 1 This is a flowchart illustrating the steps of a data storage method for a distributed system provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the data packet structure provided in an embodiment of the present invention; Figure 3 This is a structural block diagram of a data storage device for a distributed system provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] As an example, in typical distributed file systems (such as Hadoop HDFS, Ceph, etc.), data is divided into fixed-size blocks and stored across multiple data nodes. To ensure data integrity, a cyclic redundancy check (CRC) code is calculated for each data block during transmission and storage. Traditional architectures store the data block and the CRC code in two separate files: the data file stores the actual content, and the metadata file stores the corresponding CRC value. This separate storage architecture exposes significant performance bottlenecks in high-concurrency write scenarios. When multiple write threads operate simultaneously, each thread needs to maintain two file streams (a data file stream and a metadata file stream) and perform two independent disk write operations. Writing to the metadata file becomes a shared resource contention point, leading to frequent lock contention and I / O waits between threads, causing a sharp decline in the efficiency of data write operations to disk.

[0032] In response, this invention fundamentally solves the aforementioned performance problem by reorganizing the physical storage relationship between data files and verification information. Specifically, when a data node receives a target data packet for a local data file, it can parse the corresponding target data block and the target verification information corresponding to that target data block from the target data packet. The local data file includes verification storage space for storing verification information. During the data writing process, the data node can obtain the block relationship between the target data block and the last data block in the local data file, and write the target data block to the local data file and the target verification information to the verification storage space according to the block relationship. By optimizing the data storage structure of the data file and reserving verification storage space for storing verification information in the data file, the performance of data write operations is improved when high-concurrency read and write operations are performed in a large-scale storage cluster. By storing data and corresponding verification information in the same file, the time spent writing data files is saved. Furthermore, the number of open file streams can be reduced, thus lowering the management overhead of the file system.

[0033] Reference Figure 1 The diagram illustrates a flowchart of a data storage method for a distributed system provided in an embodiment of the present invention, which may specifically include the following steps: Step 101: Receive a target data packet for a local data file. The target data packet includes at least a target data block and target verification information corresponding to the target data block. The local data file includes at least a verification storage space for storing the verification information. In a distributed system architecture, data nodes and the system as a whole form a multi-layered, structured collaborative relationship. They are not simply components to the whole, but rather, through functional division, responsibility allocation, and collaborative mechanisms, jointly construct a data storage and management platform with scalability, high reliability, and high performance.

[0034] In a distributed system, as a complete software architecture, the core lies in integrating multiple independent computing resources into a logically unified system through distributed algorithms and communication protocols. In this architecture, data nodes play the role of the core execution unit of the data plane. Optionally, a distributed system may include a system management layer and a data storage layer. The system management layer can consist of a metadata server, a master node, or a coordinator component, responsible for global namespace management, data distribution strategies, node status monitoring, and access scheduling. This layer defines the system's behavioral logic and consistency model, serving as the system's "brain." The data storage layer can consist of numerous data nodes, responsible for specific tasks such as persistent storage of entity data, local I / O operations, data integrity verification, and replica maintenance. This layer is the system's "flesh and blood" and "skeleton," the physical repository of data. Data nodes operate under the instructions of the management layer, and the two maintain state synchronization through heartbeat mechanisms, block reporting, and other protocols, jointly forming a centralized management and distributed execution collaborative system.

[0035] In this embodiment of the invention, especially in large-scale high-concurrency storage scenarios, the distributed file system needs to perform a large amount of data writing. As the execution node, the data node needs to write the contents contained in the received data packet into the corresponding data file under the instruction of the management layer to complete the corresponding data disk writing operation.

[0036] In its implementation, after receiving a target data packet for a local data file, a data node can parse the packet to obtain the target data block and its corresponding checksum (CRC) information. This allows it to write the target data block to the local data file. Furthermore, since the local data file has reserved storage space for writing checksum information, the data node can also write the checksum information to this storage space during the process of writing the target data block. By storing both the data and its corresponding checksum in the same file, the performance of the data write operation is improved, the time spent writing to the data file is saved, and the number of open file streams is reduced, lowering the file system management overhead.

[0037] Reference Figure 2 The diagram illustrates the structure of a data packet provided in this embodiment of the invention. The data packet received by a data node can include at least a data header, data blocks, and verification information. The data header carries necessary control information, including but not limited to the data packet sequence number, data length, logical offset within the data block, operation type identifier, and a flag indicating whether it is the end of a data block. The presence of the data header enables the receiver to accurately understand the context and processing requirements of the data packet. The data block is a fragment of actual user data to be stored. In a distributed file system, a complete data block typically consists of multiple consecutive data packets, each containing a portion of the data block's content. The verification information is a CRC checksum calculated by the data sender based on the content of this data block. This verification information ensures data integrity during transmission and serves as a basis for verifying data correctness during storage.

[0038] In some feasible implementations, to improve the efficiency of data persistence to disk, in this embodiment of the invention, the local data file undergoes structural optimization during the creation phase. The local data file is physically divided into two logical areas: a verification storage space and a data storage space. The verification storage space stores verification information, while the data storage space stores the actual user data content. The verification storage space is located in a continuous area at the beginning of the file, while the data storage space can be located in the area following the verification storage space. By optimizing the data storage structure, during the data persistence process, data nodes can write data blocks and corresponding verification information into the same file, improving the performance of the data persistence operation, saving time spent writing data files, and reducing the number of open file streams, thus lowering the management overhead of the file system.

[0039] Optionally, for verification storage space, after the local data file is created, the required size of the verification storage space can be accurately calculated. Then, based on the calculated size, corresponding storage space can be reserved in the local data file to ensure sufficient and non-wasteful storage space for all possible verification information. In specific implementation, the file parameters corresponding to the local data file can be obtained. The file parameters include at least the data capacity, the block size of the data blocks, and the information size of the verification information. Then, based on the data capacity, block size, and information size, the verification storage space used to store the verification information can be calculated in the local data file.

[0040] Among these, data capacity refers to the maximum size of a complete logical data block, such as the common 128MB or 256MB in HDFS. This parameter determines the maximum capacity of the file data area; block size refers to the size of the data segment on which a single checksum is calculated. The system divides the data block into multiple contiguous calculation units according to this size, with each unit generating an independent CRC value. Typical values ​​are 512 bytes or 1KB; checksum size refers to the storage space occupied by each CRC value, usually 4 bytes (corresponding to a 32-bit CRC). In the specific implementation, the calculation method for checksum storage space can be obtained using the following formula: Check Sum Disk Len=ceil(Block Size / Chunk Size)*Bytes Per Check Sum Wherein, Check Sum Disk Len is the size of the check storage space, Block Size is the data capacity of the data file, Chunk Size is the block size required to generate a check message, and BytesPer Check Sum is the information size of each check message.

[0041] For example, suppose a data block is configured as follows: Block Size = 128 MB = 134,217,728 bytes; Chunk Size = 512 bytes; Bytes per Check Sum = 4 bytes; The check storage space size is: Check Sum Disk Len = ceil(134217728 / 512)*4=ceil(262144)*4=262144*4=1048576 bytes (i.e. 1MB). That is, when the system creates a data file, it can reserve the first 1MB of space specifically for storing the CRC check code. The actual user data will be written starting from the file offset of 1MB.

[0042] Step 102: Obtain the block relationship between the target data block and the last data block in the local data file, and write the target data block into the local data file according to the block relationship, and write the target verification information into the verification storage space.

[0043] For the parsed target data blocks and target verification information, when writing to the local data file, the data node can first obtain the block relationship between the target data block and the last data block in the local data file. This block relationship can be used to characterize the logical relationship between the newly arrived data block and the existing data in the local data file. Through this block relationship, the appropriate data processing and writing strategy can be determined to ensure data correctness and storage efficiency. Then, the data node writes the target data block to the local data file and writes the target verification information to the verification storage space based on the corresponding writing strategy. By optimizing the data storage structure of the data file and reserving verification storage space in the data file for storing verification information, the performance of data write operations can be improved when high-concurrency read and write operations are performed in a large-scale storage cluster. By storing data and corresponding verification information in the same file, the writing time of the data file can be saved. In addition, the number of open file streams can be reduced, and the management overhead of the file system can be reduced.

[0044] The analysis of block relationships is mainly based on the comparison between the offset information in the packet header and the current state of the local data file. It can include at least: ① continuation and completion of the same block: the new data and the last data block belong to the same computing unit, and the data is sufficient to complete the unit; ② continuous but incomplete blocks: the new data is immediately after the last block, but it is still insufficient to form a complete computing unit, either alone or after merging; ③ the beginning of a completely new block: the new data belongs to a completely new computing unit and has no direct continuity with the existing blocks.

[0045] In some feasible implementations, if the target data block and the last data block in the local data file belong to the same block, and the target data block is a complete block, then the target data block is truncated, and the truncated data is appended to the data storage space of the local data file. Then, based on the first write position of the target data block in the data storage space, the first offset address of the first write position in the verification storage space is determined. Then, the target verification information is written into the verification storage space according to the first offset address, overwriting the currently stored verification information. In this process, data retransmission and correction scenarios are effectively handled, storage space is avoided, and the real-time performance of the verification information is ensured through accurate address mapping.

[0046] In its implementation, when a data node determines that the target data block and the last data block of the local file belong to the same computational block, and that the received data constitutes a complete computational unit, the data node first analyzes the overlapping portion between the target data block and the last data block. By comparing the packet offset and the current file length, it calculates the number of bytes to be truncated, thus ensuring that duplicate data is not stored and maintaining data consistency. Next, the data node appends the truncated new data to the corresponding data storage area of ​​the local data file. This "append" operation is relative to the data area; the data node maintains the current write pointer of the data storage area to ensure the continuity of data on the physical medium. Finally, while writing the target data block, the data node can also calculate the corresponding check information storage address based on the starting position of the calculated block in the data area using a predefined mapping relationship. The mapping formula is: CRC_Offset = (Data_Offset / ChunkSize) *BytesPerCheckSum, where Data_Offset is the logical offset of the target data block in the data area (the size of the check storage space needs to be subtracted from the physical offset). Then, the data node can directly write the target check information in the data packet to the calculated address, overwriting any old check values ​​that may already exist at that location. This in-situ update operation ensures that the check information is consistent with the latest data content.

[0047] For example, assuming the calculated chunk size (ChunkSize) is 512 bytes and each checksum (BytesPerCheckSum) is 4 bytes, the overwrite process for the same chunk can include: 1. Initial state: In the local data file, 300 bytes of incomplete data have been written to a certain computation block (e.g., the Kth block). Its corresponding temporary CRC (which may be calculated based on the incomplete data or is an initial value) has been stored in the Kth slot of the check storage space (offset address = K * 4 bytes).

[0048] 2. Receiving data packets: The data node received a target data packet. The packet header information shows that this data packet is the complete data (512 bytes) of the Kth computation block, and the starting offset corresponds exactly to the beginning of the Kth block.

[0049] 3. Processing procedure: ① Analysis and Truncation: The system detects that the first 300 bytes of the Kth block already exist locally. Therefore, the first 300 bytes of the newly arrived 512 bytes of data are duplicates. The system performs a truncation operation, retaining only the last 512-300=212 bytes of new data.

[0050] ② Data writing: Append these 212 bytes of new data after the existing 300 bytes in the Kth block of the data storage area. At this point, the data in the Kth block becomes a complete 512 bytes.

[0051] ③CRC writing: Calculate the logical offset of the Kth block in the data area: Data_Offset = K * 512.

[0052] Calculate the address of its CRC in the check memory space: CRC_Offset=(K*512) / 512*4=K*4.

[0053] The target check information, calculated based on the complete 512 bytes, carried in the data packet, is written to the CRC_Offset address of the check storage space, overwriting the previous incomplete old CRC value.

[0054] In the above process, data correction is efficiently completed through truncation and overwrite, ensuring the eventual consistency of data with CRC and avoiding the storage of duplicate data.

[0055] In some feasible implementations, if the target data block is not a complete block and is a continuous block with the last data block in the local data file, the target data block is appended to the data storage space of the local data file, and the last data block in the local data file is merged with the target data block into a merged block. Then, based on the second write position of the merged block in the data storage space, the second offset address of the second write position in the verification storage space is determined. At the same time, the merged verification information corresponding to the merged block is calculated using the target verification information, and then the merged verification information is written into the verification storage space according to the second offset address. In this process, the boundary cases in streaming writing are effectively handled. Through dynamic merging and recalculation, it is effectively ensured that data writing of any size can generate correct verification information, thus guaranteeing the validity and accuracy of data writing.

[0056] In the specific implementation, when a data node detects that the target data block itself does not constitute a complete data block, but its data logic is immediately adjacent to the last data block in the local data file, the data node can write the target data block and target verification information according to the standard process of processing partial data during sequential writing. Specifically, the data node can first append the target data block to the end of the data storage area and update the current write pointer corresponding to the local data file. At the same time, if the data node recognizes that the target data block and the last data block can be merged into a complete computation block, the data node can perform the corresponding memory merging operation. For example, it can read the last incomplete computation block from the data storage area into the memory buffer, and then append the target data block to the existing data in the buffer, ensuring that the size of the merged data block is exactly equal to ChunkSize. Simultaneously, while writing the target data block, since the target check information carried by the data packet is only valid for itself and cannot represent the complete data after merging, the data node needs to recalculate the CRC check value for the merged complete data block to ensure that the stored check information can truly reflect the state of the complete data content. Then, based on the location of the merged block in the data storage area, the corresponding storage address is calculated, and the newly calculated merged check information is written to the corresponding location in the check storage space. This effectively handles the boundary cases in streaming writing. Through dynamic merging and recalculation, it effectively ensures that data writing of any size can generate correct check information, guaranteeing the validity and accuracy of data writing.

[0057] For example, assuming Chunk Size = 512 bytes and Bytes Per Check Sum = 4 bytes, the merging process for incomplete chunks can include: 1. Initial state: The last computed block (the Mth block) of the local data file has been written 500 bytes and is not yet full. Due to the incomplete block, its CRC may not have been calculated or may be invalid.

[0058] 2. Receiving data packets: The data node receives a target data packet containing 100 bytes of data. After parsing, this data should be written immediately after the 500th byte of the Mth block.

[0059] 3. Processing procedure: ① Data appending and merging: First, append these 100 bytes of data to the data storage area, making the total length of the Mth block reach 500 + 100 = 600 bytes. This exceeds ChunkSize.

[0060] The system recognizes that the Mth block is full (512 bytes) and identifies the extra 600-512=88 bytes as the beginning of the M+1th block.

[0061] Specific operation: The original 500 bytes of the Mth block and the first 12 bytes of the new data (512-500=12) are merged in memory to form a complete 512-byte block. The remaining 88 bytes of the new data are temporarily stored as the starting point of the (M+1)th block.

[0062] ②CRC recalculation and writing: The CRC is recalculated on the merged 512-byte complete data to obtain the merged verification information.

[0063] Calculate the CRC address of the Mth block: CRC_Offset = M * 4.

[0064] Write the newly calculated merged verification information to this address.

[0065] In the above process, the case where the data packet size is not an integer multiple of ChunkSize was properly handled. Through dynamic merging and recalculation, it was ensured that each complete calculation block has the correct CRC, thus guaranteeing the integrity of the data.

[0066] In some feasible implementations, if the target data block is a complete block and does not belong to the same block as the last data block in the local data file, the target data block is appended to the data storage space of the local data file. Then, based on the third write position of the target data block in the data storage space, the third offset address of the third write position in the verification storage space is determined. Then, the target verification information is written to the verification storage space according to the third offset address. In this process, since the data block belongs to a complete computation block, the data node does not need to introduce any additional computational overhead during the writing process and can be directly written to the local data file in the predetermined writing order, thus ensuring the efficiency of data writing.

[0067] In the implementation, when the target data block is a complete computational block and logically initiates a new computational unit, the data node can directly append the entire data block to the end of the data region. Since the data itself is a complete computational unit, no truncation or merging operations are required. Simultaneously, based on the starting position of the new data in the data region, its corresponding target address in the verification storage space is calculated using a standard mapping formula. Then, the target verification information carried in the data packet is directly written to the calculated address. Because the data packet itself is complete and the verification information is calculated based on this complete data, no recalculation or verification is required.

[0068] For example, assuming Chunk Size = 512 bytes and Bytes Per Check Sum = 4 bytes, the direct append process for a complete chunk can include: 1. Initial state: The local data file has been completely written to N computation blocks, and the current pointer to the data area is at N * 512 bytes. Valid CRC checks have been written to the first N slots of the verification storage space.

[0069] 2. Receiving data packets: The data node receives a target data packet containing a complete 512-byte data block, which is logically the N+1th computation block.

[0070] 3. Processing procedure: ① Data writing: Append these 512 bytes of data directly to the end of the data storage area. The data area pointer is updated to (N+1) * 512.

[0071] ②CRC writing: Calculate the CRC address of the (N+1)th block: CRC_Offset = (N+1) * 4.

[0072] The target verification information, which is calculated based on the 512 bytes of complete data contained in the data packet, is directly written to the CRC_Offset address of the verification storage space.

[0073] In the above process, there is no need for data truncation, merging, or CRC recalculation; the data is directly appended, maximizing the throughput of sequential writes.

[0074] In addition, after writing the data and verification information, the data node also needs to maintain relevant metadata information. This metadata information can include lightweight information that supports file management and recovery. Specifically, this can be achieved by updating the metadata information in the local data file. The metadata information includes at least the current write length of the data storage space and the current write status of the verification space. Compared to traditional CRC metadata files, the metadata processed in this way has lower management overhead and is more conducive to reducing the management overhead of the file system.

[0075] In a specific implementation, the updated metadata may include: current length of the data area: records the total amount of valid data that has been written to the data area. This information is used to determine the starting position of the next write and to quickly locate the data boundary when reading the file; check space write status: uses an efficient data structure (such as a bitmap) to record which positions in the check storage space have been validly written. Each bit represents a CRC storage unit, and setting it indicates that valid check information has been written; file integrity flag: after the data block is completely written, sets the corresponding flag bit to indicate that the data block is ready to provide services, etc. This invention does not limit this.

[0076] It should be noted that the embodiments of the present invention include, but are not limited to, the examples described above. It is understood that those skilled in the art can make further settings according to actual needs under the guidance of the ideas in the embodiments of the present invention, and the present invention does not limit such settings.

[0077] In this embodiment of the invention, when a data node receives a target data packet for a local data file, it can parse the corresponding target data block and the target verification information corresponding to the target data block from the target data packet. The local data file includes a verification storage space for storing verification information. During the data writing process, the data node can obtain the block relationship between the target data block and the last data block in the local data file, and write the target data block into the local data file and the target verification information into the verification storage space according to the block relationship. By optimizing the data storage structure of the data file and reserving verification storage space for storing verification information in the data file, the performance of data write operations can be improved when high-concurrency read and write operations are performed in a large-scale storage cluster. By storing the data and the corresponding verification information in the same file, the writing time of the data file can be saved. In addition, the number of open file streams can be reduced, and the management overhead of the file system can be reduced.

[0078] To enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention, the following examples are provided for illustrative purposes: As an example, the data storage layout on a data node is as follows: The starting write position of the CRC in the data file and the amount of space to be reserved for the CRC are calculated based on the block size information in the metadata file. When the data node writes blocks to the data file, it can also write the corresponding CRC to the reserved positions, as follows: [CRC0][CRC1][CRC2][...][CRCN][...][data0][data1][data2][...][dataN] Furthermore, as the number of packets received by the data node increases and the amount of data written each time varies, the offset of the data in the data file will change. At this time, it is necessary to process the data into blocks and perform CRC checks according to different situations before writing it into the data file.

[0079] In one case: Local data file: [meta][123crc][4crc].......

[123] [4] Received data packet: [header][456crc][789crc]...

[456]

[789] ... The data is truncated and the CRC is recalculated before being written. Data file after writing: [meta][123crc][456crc][789crc]...

[123]

[456]

[789] ... In another case: Local data file: [meta][123crc][4crc].......

[123] [4] Data packet received: [header][567crc][89crc]...

[567]

[89] ... Data is appended directly during writing, and the CRC is recalculated before writing: Data file after writing: [meta][123crc][456crc][789crc]...

[123]

[456]

[789] ... In another case: Local data file: [meta][123crc][456crc]...

[123]

[456] Data packet received: [header][789crc]...

[789] ... Data is appended directly during writing, and CRC is also appended directly: Data file after writing: [meta][123crc][456crc][789crc]...

[123]

[456]

[789] ... Through the above process, by optimizing the data storage structure of the data file and reserving verification storage space in the data file for storing verification information, the performance of data write-to-disk operations is improved when high-concurrency read and write operations are performed in a large-scale storage cluster. This saves the time spent writing data files and also reduces the number of open file streams, thereby reducing the management overhead of the file system.

[0080] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0081] Reference Figure 3 The diagram illustrates a structural block diagram of a data storage device for a distributed system provided in an embodiment of the present invention, which may specifically include the following modules: The data packet receiving module 301 is used to receive a target data packet for a local data file. The target data packet includes at least a target data block and target verification information corresponding to the target data block. The local data file includes at least a verification storage space for storing the verification information. The data writing module 302 is used to obtain the block relationship between the target data block and the last data block in the local data file, and to write the target data block into the local data file according to the block relationship, and to write the target verification information into the verification storage space.

[0082] Among some feasible implementation methods are: The file parameter acquisition module is used to acquire the file parameters corresponding to the local data file. The file parameters include at least the data capacity, the block size of the data blocks, and the size of the verification information. The storage space calculation module is used to calculate the verification storage space for storing verification information in the local data file based on the data capacity, the block size, and the information size.

[0083] In some feasible implementations, the storage space calculation module can be specifically implemented using the following formula: Check Sum Disk Len=ceil(Block Size / Chunk Size)*Bytes Per Check Sum Wherein, Check Sum Disk Len is the size of the check storage space, Block Size is the data capacity of the data file, Chunk Size is the block size required to generate a check message, and BytesPer Check Sum is the information size of each check message.

[0084] In some feasible implementations, the local data file further includes data storage space for storing data blocks, and the data writing module is specifically used for: If the target data block belongs to the same block as the last data block in the local data file, and the target data block is a complete block, then the target data block is truncated, and the truncated data is appended to the data storage space of the local data file. Based on the first write position of the target data block in the data storage space, determine the first offset address of the first write position in the verification storage space; The target verification information is written into the verification storage space according to the first offset address, overwriting the currently stored verification information.

[0085] In some feasible implementations, the data writing module is specifically used for: If the target data block is not a complete block, and it is a continuous block with the last data block in the local data file, then the target data block is appended to the data storage space of the local data file, and the last data block in the local data file is merged with the target data block into a merged block. Based on the second write position of the merged block in the data storage space, determine the second offset address of the second write position in the verification storage space; The target verification information is used to calculate the merge verification information corresponding to the merged block; The merged verification information is written into the verification storage space according to the second offset address.

[0086] In some feasible implementations, the data writing module is specifically used for: If the target data block is a complete block and does not belong to the same block as the last data block in the local data file, then the target data block is appended to the data storage space of the local data file. Based on the third write position of the target data block in the data storage space, determine the third offset address of the third write position in the verification storage space; The target verification information is written into the verification storage space according to the third offset address.

[0087] Among some feasible implementation methods are: An update module is used to update the metadata information in the local data file. The metadata information includes at least the current write length of the data storage space and the current write status of the verification space.

[0088] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0089] In addition, this invention also provides an electronic device, including: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the various processes of the above-described distributed system data storage method embodiments and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0090] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the data storage method embodiments of the distributed system described above, and achieves the same technical effect. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0092] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, EEPROM, Flash, and eMMC, etc.) containing computer-usable program code.

[0093] Embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0096] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0097] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0098] The present invention has provided a detailed description of a data storage method and a data storage device for a distributed system. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data storage method of a distributed system, characterized by, The method comprises the following steps: receiving a target data packet for a local data file, wherein the target data packet comprises at least a target data chunk and target check information corresponding to the target data chunk, and the local data file comprises at least a check storage space for storing check information; obtaining a chunk relationship between the target data chunk and a last data chunk in the local data file, and writing the target data chunk into the local data file and the target check information into the check storage space according to the chunk relationship.

2. The method of claim 1, wherein, The method further comprises the following steps: obtaining file parameters corresponding to the local data file, wherein the file parameters comprise at least a data capacity, a chunk size of a data chunk and an information size of check information; calculating the check storage space for storing check information in the local data file according to the data capacity, the chunk size and the information size.

3. The method of claim 2, wherein, The check storage space for storing check information in the local data file is calculated according to the data capacity, the chunk size and the information size by the following formula: Check Sum Disk Len = ceil(Block Size / Chunk Size)*Bytes Per Check Sum wherein Check Sum Disk Len is the size of the check storage space, Block Size is the data capacity of the data file, Chunk Size is the chunk size required for generating one check information, and Bytes Per Check Sum is the information size of each check information.

4. The method of claim 1, wherein, The local data file further comprises a data storage space for storing data chunks, and the method of writing the target data chunk into the local data file and the target check information into the check storage space comprises the following steps: if the target data chunk and the last data chunk in the local data file belong to the same chunk and the target data chunk is a complete chunk, truncating the target data chunk and appending the truncated data to the data storage space of the local data file; determining a first writing position of the target data chunk in the data storage space, and a first offset address of the first writing position in the check storage space; writing the target check information into the check storage space according to the first offset address, and covering the stored current check information.

5. The method of claim 4, wherein, The method of writing the target data chunk into the local data file and the target check information into the check storage space comprises the following steps: if the target data chunk is not a complete chunk and belongs to a continuous chunk with the last data chunk in the local data file, appending the target data chunk to the data storage space of the local data file, and merging the last data chunk in the local data file and the target data chunk into a merged chunk. determining a second offset address of the second write position in the check storage space based on the second write position of the merged chunk in the data storage space; calculating merged check information corresponding to the merged chunk by using the target check information; writing the merged check information into the check storage space according to the second offset address.

6. The method of claim 1, wherein, The writing of the target data chunk into the local data file and the writing of the target check information into the check storage space according to the chunk relationship comprises: if the target data chunk is a complete chunk and does not belong to the same chunk as the last data chunk in the local data file, appending the target data chunk to the data storage space of the local data file; determining a third offset address of the third write position in the check storage space based on the third write position of the target data chunk in the data storage space; writing the target check information into the check storage space according to the third offset address.

7. The method according to any one of claims 4 to 6, characterized in that, Further comprising: updating metadata information in the local data file, wherein the metadata information at least comprises a current write length of the data storage space and a current write state of the check storage space.

8. A data storage apparatus of a distributed system, characterized by, Comprise: a data packet receiving module configured to receive a target data packet for a local data file, wherein the target data packet at least comprises a target data chunk and target check information corresponding to the target data chunk, and the local data file at least comprises a check storage space configured to store check information; a data writing module configured to obtain a chunk relationship between the target data chunk and a last data chunk in the local data file, and write the target data chunk into the local data file and write the target check information into the check storage space according to the chunk relationship.

9. An electronic device, comprising: comprise a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete communication with each other through the communication bus; the memory is configured to store a computer program; the processor is configured to execute the program stored on the memory, and implement the method in any one of claims 1-7. 10.A computer readable storage medium having instructions stored thereon, the instructions, when executed by one or more processors, cause the processors to perform the method in any one of claims 1-7.

Images