Data storage method and device, computer device and medium

Abstract

This application applies to the field of blockchain technology and provides a data storage method, apparatus, computer device, and medium. The method includes: receiving batch data, which includes multiple data write operations; determining a current log file in a database; if the remaining data capacity of the current log file is greater than or equal to the data size of the batch data, then writing the batch data into the current log file; if the remaining data capacity is less than the data size, then writing the batch data into a recycling log file in the database, the recycling log file being a log file where batch data has been previously written and all written batch data has been written to a persistent database; writing the contents of a target file to a disk, the target file including the current log file and the recycling log file; and writing the batch data into memory. This method can improve the data storage efficiency of the database while ensuring the reliability of data storage.

Description

Technical Field

[0001] This application belongs to the field of blockchain technology, and in particular relates to a data storage method, device, computer equipment and medium. Background Technology

[0002] Data storage involves storing data received or generated by a device onto a storage medium. When performing a task, the device can retrieve the corresponding data from the storage medium and perform operations based on the retrieved data. Storing data in memory offers high data access efficiency, improving the device's computing efficiency, but data is easily lost after a crash and restart. Storing data on disk ensures data is less prone to loss, but low data access efficiency leads to slower task execution. Therefore, achieving a balance between data storage efficiency and reliability is challenging. Summary of the Invention

[0003] In view of this, embodiments of this application provide a data storage method, apparatus, computer equipment, and medium to improve data storage efficiency.

[0004] A first aspect of this application provides a data storage method, including:

[0005] Receive batch data, wherein the batch data includes multiple data write operations;

[0006] Determine the current log file in the database;

[0007] If the remaining data capacity of the current log file is greater than or equal to the data size of the batch data, then the batch data is written to the current log file;

[0008] If the remaining data capacity is less than the data size, the batch data is written to the recycling log file in the database. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written to the persistent database.

[0009] Write the contents of the target file to the disk, wherein the target file includes the current log file and the recycling log file;

[0010] The batch data is written into memory.

[0011] A first aspect of this application provides a data storage device, including:

[0012] The receiving module is used to receive batch data, which includes multiple data write operations;

[0013] The determination module is used to determine the current log file in the database;

[0014] The first storage module is configured to write the batch data into the current log file if the remaining data capacity of the current log file is greater than or equal to the data size of the batch data.

[0015] The second storage module is used to write the batch data into the recycling log file in the database if the remaining data capacity is less than the data size. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written into the persistent database.

[0016] The disk writing module is used to write the file content of the target file to the disk, wherein the target file includes the current log file and the recycling log file;

[0017] A caching module is used to write the batch data into memory.

[0018] A third aspect of this application provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0019] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0020] A fifth aspect of this application provides a computer program product that, when run on a computer device, causes the computer device to perform the method described in the first aspect.

[0021] Compared with the prior art, the embodiments of this application have the following advantages:

[0022] Data can be processed in batches during storage. Batch data processing can include multiple data write operations, enabling batch storage of data, thereby reducing the number of disk flushes and improving data storage efficiency. When batch data is received, the current log file in the database can be determined. If the remaining data capacity of the current log file is greater than or equal to the size of the batch data, the batch data can be stored in the current log file. If the remaining data capacity of the current log file is less than the batch data, it means that the current log file cannot store the batch data, and the batch data can be stored in the reclaimed log file. The reclaimed log file is a log file where batch data has been previously written and all written batch data has been written to the persistent database. Storing batch data in the reclaimed log file enables the reuse of the log file, thereby avoiding the infinite expansion of the log file in extreme cases. Then, the contents of the log file are written to the disk, thus preventing the loss of batch data after the database restarts. After the log file is written to the disk, the batch data can be written to memory, facilitating fast data access. Based on the log file, this embodiment can first store data on the disk and then store it in memory, ensuring the reliability of data storage while improving data storage efficiency. Furthermore, when writing log files to disk, it is not necessary to write the log file's metadata to the disk, such as the file size, file modification time, etc. Only the file content of the log file needs to be written, thereby reducing disk input and output and improving the database's data storage efficiency. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0024] Figure 1 This is a flowchart illustrating the steps of a data storage method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of a log file in a database provided in an embodiment of this application;

[0026] Figure 3 This is a schematic diagram of another log file in a database provided in an embodiment of this application;

[0027] Figure 4 This is a flowchart illustrating the steps of another data storage method provided in an embodiment of this application;

[0028] Figure 5 This is a schematic diagram of another type of log file in a database provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of a data storage device provided in an embodiment of this application;

[0030] Figure 7 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0031] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details.

[0032] In other cases, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted to avoid unnecessary detail from obscuring the description of this application.

[0033] Memory offers high access efficiency, but data in memory is lost after a power outage and restart. Data on disk persists after a power outage and restart, but access efficiency is low. Therefore, to ensure efficiency while preventing data loss, data can be written to memory simultaneously with the disk storage. Responding to user requests based on the data in memory improves efficiency. Alternatively, data can be written to a disk file before being written to memory, thus preventing data loss after a database restart. These disk files are typically journals or write-ahead logs (WAL).

[0034] Generally speaking, logs do not need to provide query functionality, but only need to restore memory during a crash restart. Therefore, the log structure is often an append-only structure. This structure does not have good query performance, but it can provide fast writing, so log writing will not cause a significant performance loss.

[0035] In computer operating systems, file write operations are not directly written to disk; instead, they are first written to the page cache. That is, data is written to the page cache before being written to the log file. The operating system then writes the page cache data to disk in the background. The existence of the page cache, combined with the sequential append structure of the log file, ensures that writing to disk before data is written to memory does not degrade performance. However, this introduces a problem: even if data has been written to the log file, it may not have been written to disk but remains in the page cache. Therefore, if an operating system crashes, such as a server power outage, the data will be lost. This can lead to unacceptable problems such as database unavailability after restarting and loss of user databases.

[0036] Therefore, FSync can be enabled in the database system. FSync forces the computer system to write the entire metadata and content of the log file to disk when writing data to the log file. Since a single disk write is approximately 60 times slower than a memory write, the drawback of FSync is also very obvious: the database write efficiency will be significantly reduced.

[0037] Furthermore, in traditional organization methods, data in memory can be organized in a certain way, with a one-to-one correspondence between in-memory data structures and log files. The creation and deletion of in-memory data structures are accompanied by the creation and deletion of log files. In addition, due to the sequential append structure of log files, multiple updates to data in memory can cause log files to grow indefinitely. Therefore, with FSync enabled, the memory and log organization of traditional KeyValue databases inevitably leads to performance degradation.

[0038] For example, in a blockchain, in-memory data can be organized using in-memory tables (MemTables). There is a one-to-one correspondence between an in-memory MemTable and a log file in the database; that is, for every MemTable created, a corresponding log file needs to be created, and for every MemTable deleted, a corresponding log file needs to be deleted. A MemTable is a KeyValue type in-memory data structure. In a MemTable, updates or deletions of duplicate keys are performed in-place. That is, even after 100 Put(X=100) operations, the MemTable will still only contain one record of data related to X. In contrast, log files are append-only; that is, after 100 Put(X=100) operations, the log file will store 100 records related to X. In extreme scenarios, this can cause the log file to grow indefinitely.

[0039] Based on this, this application proposes a data storage method to avoid the unlimited expansion of log files and improve data storage efficiency. The technical solution of this application will be described below through specific embodiments.

[0040] Reference Figure 1 The diagram illustrates a step flow of a data storage method provided in an embodiment of this application. The execution subject of this embodiment can be a computer device, such as a tablet computer, wearable device, vehicle device, laptop computer, super mobile personal computer, server, etc. The computer device can deploy a blockchain network, making it a blockchain node device. Figure 1 The method shown may specifically include the following steps:

[0041] S101, Receive batch data, the batch data including multiple data write operations.

[0042] Computer devices can receive and store data sent from other devices or input data from users. Based on the stored data, they can respond to the needs of other users or respond to user requests. Generally, during the execution of a task, a computer device can receive data, which is typically used to perform the current task. To improve the computer's response speed, data can be stored in memory first, facilitating fast data access. However, memory is volatile; therefore, if the database is shut down and restarted, the data in memory will be lost and cannot be retrieved, which is unacceptable. Therefore, data can be stored on disk first, and then stored in memory, thereby improving the computer device's response speed while avoiding data loss.

[0043] The batch data mentioned above can include multiple write operations, meaning it comprises multiple data sets that need to be stored. The batch data can have a size, which determines its storage location. Batch data can be data sent from other devices; for example, it can be a document sent from another device to a computer. Batch data can also be user-inputted data, such as text typed into a document by a user.

[0044] For example, the method in this embodiment can be applied to a blockchain system, where the batch processing data can be batch transaction data. If multiple transactions are completed in the blockchain system, the data from these transactions will form batch processing data, which can be sent to various blockchain node devices. Each node device in the blockchain system can reach a consensus on the transaction results based on the batch processing data. When all node devices agree on the results of the batch processing data, the batch processing data will not be rolled back and can be committed, thus being written to a persistent database. During the consensus or other processing of the batch processing data by the node devices, to facilitate fast data access, the data can be stored in memory to accelerate computational efficiency.

[0045] S102, determine the current log file in the database.

[0046] A database can include multiple log files. These log files store data, and the data stored in them is used to restore data in memory after a database restart. When the database initially starts, no log files are present; a new log file can be created at this time. In this embodiment, the created log file can have a fixed data capacity, which can be preset, for example, 10MB. When the existing log file can no longer store data, a new log file can be created; that is, the database can include multiple log files. Each log file can have a corresponding filename, and the database can include a log file directory, which can include the filenames of each log file. For example, when the database initially starts, a 10MB log file can be created, named 0.log; when the data capacity of 0.log is insufficient, another 10MB log file, 1.log, can be created and used to continue storing batch processing data. In this case, the log file directory can include both 0.log and 1.log.

[0047] Log files can also have attribute information that identifies whether a log file is a reclaimed log file. A reclaimed log file is one that previously stored batch data, and all of that batch data has been committed to the persistent database. For example, when the batch data stored in a log file has reached a consensus based on a consensus mechanism, the data can be committed to the persistent database, and the data in the persistent database will not be changed. Each newly created log file can have a first attribute, meaning it's a non-reclaimed log file. A non-reclaimed log file can store batch data that hasn't yet been committed to the persistent database or no data at all. When all the batch data stored in a non-reclaimed log file has been committed to the persistent database, the attribute information of the non-reclaimed log file can be changed to a second attribute, and the non-reclaimed log file becomes a reclaimed log file. When a reclaimed log file re-stores uncommitted batch data, the attribute information of the reclaimed log file can be changed back to the first attribute, and the reclaimed log file becomes a non-reclaimed log file. The attribute information of a log file can be represented in various ways. For example, the attribute information can be identified by the filename suffix; the filename suffix for a non-reclaimed log file can be .log, and the filename suffix for a reclaimed log file can be .rc.

[0048] When storing batch data, log files can store data sequentially, meaning they start storing the next batch of data from the storage location where the previous batch data was stored. Therefore, to quickly locate the storage location, the storage location where the previous batch data was stored can be recorded; this location can serve as the write location for the next batch of data. Since a database can contain multiple log files, the name of the log file corresponding to this write location can also be recorded, allowing for quick location of the log file at that write location. The log file corresponding to the write location is the current log file mentioned above.

[0049] The database can include log identification information, which may include the filename and write position of the current log file. The write position of the log file can be characterized by the actual size of the data stored in the current log file. For example, if the current log file contains 8MB of data after the previous log file was stored, then the write position could be 8MB. The data capacity of the current log file is fixed; therefore, the remaining data capacity of the current log file can be determined based on the data capacity and the write position.

[0050] S103, if the remaining data capacity of the current log file is greater than or equal to the data size of the batch data, then the batch data is written to the current log file.

[0051] If the current log file's data size is greater than or equal to the batch data size, the batch data can be directly written to the current log file.

[0052] In a blockchain system, batch data can be stored as a block in a log file. Each block can have a block identifier, which is determined by the storage order of the batch data. For example, if the block identifier for the last batch of data written to the current log file is 5, then the block identifier for the next batch of data written to the current log file will be 6, and so on. The log file can sequentially store multiple blocks of data.

[0053] S104, if the remaining data capacity is less than the data size, then the batch data is written to the recycling log file in the database. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written to the persistent database.

[0054] If the remaining data capacity of the current log file is less than the size of the batch data, it means that the batch data cannot be stored in the current log file. In this case, it can be determined whether a reclaimed log file exists in the database. Specifically, log files with the second attribute can be searched in the database. If a log file with the second attribute exists, it indicates that a reclaimed log file exists in the database. For example, the log file directory in the database can be traversed to search for log files with the .rc extension; if a log file with the .rc extension exists, it indicates that a reclaimed log file exists.

[0055] If a recycling log file exists in the database, batch data can be stored in that file. Specifically, the filename of a recycling log file in the database can be used as the filename of the current log file in the log identification information, and the initial position of that recycling log file can be used as the write position for the log identification information. For example, if one of the recycling log files is named 0.rc, the log identification information can be updated to (0.rc, 0MB), and then the batch data can be stored in 0.rc.

[0056] If the recycling log file does not exist in the database, a new log file can be created, and the batch data can be stored in the new log file. Specifically, a new log file can be created in the database using a fixed-size storage area. This log file can have a filename, and the filename of the current log file in the log identification information can be updated to the filename of the new log file and the initial position of the new log file, thereby writing the batch data into the new log file. For example, a log file 3.log can be created, and then the file identification information can be updated to (3.log, 0MB), and the batch data can then be stored in 3.log.

[0057] After storing the batch data in the log file, the log identification information in the database can be updated. First, it can be determined whether the current log file has remaining storage space. If the current log file has remaining storage space, the write position can be updated based on the actual data size of the current log file. If the current log file does not have remaining storage space, it can be determined whether a reclaimed log file exists in the database. If a reclaimed log file exists in the database, the name of the reclaimed log file can be used as the filename of the current log file in the log identification information, and the initial position of the reclaimed log file can be used as the write position in the log identification information.

[0058] Figure 2 This is a schematic diagram of a log file in a database provided in an embodiment of this application, such as... Figure 2As shown, the database contains three log files: 0.log, 1.log, and 2.log. Each file has a capacity of 10MB. 0.log stores block data from block numbers 0-40, 1.log stores block data from block numbers 41-60, and 2.log stores block data from block number 61. The actual data size of 0.log and 1.log is 10MB, meaning they are full. The actual data size of 2.log is 8MB, meaning it is not yet full. At this point, the log identifier in the database can be (2.log, 8MB). If a batch of data of 1MB is received, it can be determined that the batch data can be stored in 2.log based on 10MB - 8MB > 1MB. After storing this batch data as block number 62 in 2.log, the log identifier in the database can be updated to (2.log, 9MB).

[0059] When block number 40 is committed, all data in 0.log will be committed to the persistent database. At this point, 0.log can be renamed to 0.rc, where 0.rc is the reclaimed log file. The log files in the database can be configured as follows: Figure 3 As shown, if a batch of data with a size of 2MB is received, it can be determined that the batch of data cannot be stored in 2.log based on 10MB-9MB<2MB. At this time, it can be determined that there is a recycling log file 0.rc in the database. Then, the log identification information is updated to (0.rc, 0MB). After storing the batch of data as a block with block number 63 in 0.rc, 0.rc can be changed to 0.log, and the log identification information in the database is updated to (0.log, 2MB).

[0060] In another possible implementation, the log file's data capacity can be expanded once. If the remaining data capacity of the current log file is less than the size of the batch data, the log file can be expanded so that the remaining data capacity of the current log file equals the size of the batch data, thereby storing the batch data in the current log file.

[0061] For example, in Figure 3In the database shown, if a batch of data of size 2MB is received, it can be determined that this batch of data cannot be stored in 2.log based on the formula 10MB-9MB<2MB. In this case, 2.log can be expanded to 11MB, and then the batch of data can be stored in 2.log as block data with block number 63. After that, it can be determined that the database contains a recycling log file 0.rc, and the log identification information in the database can be updated to (0.rc, 0MB).

[0062] S105, write the file content of the target file to the disk, the target file including the current log file and the recycling log file.

[0063] In computer operating systems, file write operations are not directly written to the disk; instead, they are first written to the page cache. That is, data is written to the page cache before being written to the log file, and the operating system then writes the page cache data to the disk in the background. Therefore, when the computer system returns information that batch data has been written to the log file, the batch data may have only been written to the page cache instead of the disk. If batch data is only written to the page cache and not to the disk, then if the computer system crashes, such as when the server loses power, the data in the page cache will be lost, potentially rendering the database unusable.

[0064] To avoid this situation, in this embodiment, an FDataSync operation can be performed, forcing the computer system to perform a persistence operation on the log file, writing the batch data in the log file to the disk. When FSync is executed, the contents of the log file, as well as the file's metadata, such as file size and modification time, are written to the disk; while FDataSync only writes the file content to the disk. Compared to FSync, using FDataSync in this application reduces one interaction with the disk.

[0065] When writing log files to disk, only the file content of the log file needs to be written. Since the size and storage area of ​​the log file are fixed in this embodiment, the metadata corresponding to the log file is also fixed. Therefore, unlike traditional log files, in this application, only the file content of the log file needs to be written to disk, and the metadata does not need to be written. Traditional log files are appended sequentially, so their data size is constantly changing, meaning the metadata is constantly changing. Therefore, when writing traditional log files to disk, the metadata also needs to be written to disk. In other words, traditional log files require two interactions with the disk when writing to disk. In this embodiment, only one interaction with the disk is needed when writing the log file to disk. That is, when writing the log file to disk, the log file in this application reduces one input / output of the log file's metadata, improving data storage efficiency.

[0066] S106, The batch data is written into memory.

[0067] The memory can contain a MemTable, which can be a skip list. Data can be quickly retrieved from memory based on the MemTable. Batch data can be written to the in-memory MemTable, thus achieving fast data access.

[0068] In this embodiment, the log file and the data structure in memory no longer have a corresponding relationship. The decoupling of the log file and the data structure in memory avoids the uncontrollable size of the recycled log file, ensuring that the log file will not become too large, thus facilitating the reuse of the log file. The log file has a fixed size, thus avoiding infinite expansion in extreme cases. In this embodiment, when writing the log file to the disk, only the file content can be written to the disk, and the metadata can be omitted. This reduces the interaction with the disk when writing the log file to the disk and improves the efficiency of data storage.

[0069] Reference Figure 4 The diagram illustrates a flowchart of another data storage method provided in an embodiment of this application, which may specifically include the following steps:

[0070] S 401, Receive batch data, the batch data including multiple data write operations.

[0071] S 402, Determine the current log file in the database.

[0072] S 403, if the remaining data capacity of the current log file is greater than or equal to the data size of the batch data, then the batch data is written to the current log file.

[0073] S 404, if the remaining data capacity is less than the data size, then the batch data is written to the recycling log file in the database. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written to the persistent database.

[0074] S 405, Write the contents of the target file to the disk, the target file including the current log file and the recycling log file.

[0075] S 406, Write the batch data into memory.

[0076] In this embodiment, S401-S406 are similar to S101-S106 in the previous embodiment and can be referenced interchangeably. They will not be described in detail here.

[0077] S 407, Receive a data submission instruction, the data submission instruction having a corresponding first block identifier.

[0078] In a blockchain system, the consensus module periodically reaches a consensus on the block state. Once consensus is reached, it means that the state of this block and previous blocks is consistent across all nodes and there is no possibility of rollback. This indicates that the corresponding block data can be committed. Commit means that the block data that has reached consensus can be written to the persistent database.

[0079] The computer device can receive a data submission instruction, which may include the block identifier of the block data for which consensus has been reached. This block identifier is the first block identifier mentioned above.

[0080] S 408, write the block data whose block identifier is less than or equal to the first block identifier into the persistent database.

[0081] Block data whose block identifier is less than or equal to the first block identifier can be used as target data. Then, the target data can be retrieved from memory and written to the persistent database.

[0082] S 409, delete block data whose block identifier is less than or equal to the first block identifier from the log file of the database and the memory.

[0083] When block data with block identifiers less than or equal to the first block identifier is written to the persistent database, such block data can be deleted from the database's log files and memory. This is equivalent to archiving the consensus-reached block data in the log files, avoiding excessive data in memory that could impact computational efficiency.

[0084] S 410, the log file in which all the block data is written to the persistent database is used as the recycling log file.

[0085] If all block data in a log file is written to a persistent database, that log file can be used as a retrieval log file. Specifically, if all block data in a log file is written to a persistent database, the log file's attribute information can be modified to give it a second attribute. For example, in... Figure 3 In the scenario shown, all data in blocks 0-40 of 0.log is committed to the persistent database. Therefore, 0.log can be changed to 0.rc, which is the reclaim log file.

[0086] S 411, if some of the block data is written to the log file of the persistent database, then the commit identifier information is updated according to the first block identifier.

[0087] There is a situation where only a portion of the data in a log file is written to the persistent database. For example, in... Figure 3 In the scenario shown, if block 42 is committed, only a portion of the data in 2.log will be committed to the persistent database. At this point, 2.log will still contain uncommitted blocks. In this case, the database can include a commit identifier to indicate the position of the last block in the log file that has been committed to the persistent database. The commit identifier can be identified by the size of the free storage area preceding the log file. For example, if the combined size of blocks 41 and 42 is 3MB, the commit identifier could be (1.log, 3MB).

[0088] S 412, if the database is not being started for the first time, the memory is restored according to the commit identifier information and the write location.

[0089] A database restart can be a normal restart or a restart following a system crash. After a database restart, data in memory will be lost. To ensure the computing speed of the computer, the data in memory can be recovered. The data to be recovered from memory includes uncommitted blocks of data stored in the log file.

[0090] When the database is not starting for the first time, the database's log directory can be read. The log directory includes at least one log file and can include the log file's name, storage information, attribute information, etc. Based on the log directory, the target log file in the database that does not belong to the reclaimed log file can be identified. Then, the block data in the target log file between the position corresponding to the commit identifier information and the write position is restored to memory.

[0091] For example, if the log directory includes 0.rc, 1.log, 2.log, log identifier information is (2.log, 8MB), and commit identifier information is (1.log, 3MB), then all block data after the 3MB position in 1.log can be restored to memory, and the data from the initial position to the 8MB position in 2.log can be restored to memory.

[0092] In a blockchain system, block data that has not yet reached consensus can be stored in the memory and log files of node devices. During the consensus process, node devices can quickly retrieve data from memory, thereby accelerating transaction execution and improving node device performance.

[0093] If a node device detects an error in a block of data—for example, if a node device fails to reach an agreement with other node devices on a particular block—it indicates that the block data may be erroneous, and a rollback of that block is necessary. During the rollback, the block data, as well as all subsequent data, must be deleted from memory and log files.

[0094] When performing data rollback, the computer device can receive a data rollback instruction. The data rollback instruction may include a second block identifier, which can be the block identifier of the data block to be rolled back. Based on the second block identifier, the computer can search for data blocks in the log file whose block identifier is greater than or equal to the second block identifier, and then delete the data blocks whose block identifier is greater than or equal to the second block identifier. At the same time, the corresponding data blocks in memory are also deleted.

[0095] After block data rollback, the log identifier information can be updated based on the second block identifier. Based on the second block identifier, the size of the block data deleted from the log file can be determined, and the new write position can be obtained by subtracting this data size from the write position.

[0096] For example, in Figure 3 In the scenario shown, if the second block identifier is 54, it indicates that blocks 54-62 need to be deleted. If the data size of blocks 54-60 is 5MB, the updated log identifier information will be (1.log, 5MB). At this point, all block data in 2.log is also deleted. Alternatively, 2.log can be updated to a reclaim log file and then used for data storage.

[0097] In this embodiment of the application, the starting position of the block data stored in the log file can be determined based on the commit identifier information, so that when the database restarts, the data that needs to be restored to memory can be determined based on the commit identifier information and the log identifier information.

[0098] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0099] To better illustrate the solution in this application, the following section describes the solution using a complete database data management process.

[0100] Upon initial startup, no log file exists in the database. A new log file named 0.log with a capacity of 10MB can be created and used to store block data from 0 to 40. When storing block 41, there is no remaining space in 0.log. A new log file named 1.log with a capacity of 10MB can be created to store block data from 41 to 60. When storing block 61, there is no remaining space in 0.log. A new log file named 2.log with a capacity of 10MB can be created to store block data from 61. At this point, the log file in the database can be... Figure 2 As shown.

[0101] The database may include a log file directory, which may contain information about the log files in the database. Figure 2 In the corresponding scenario, the log file directory can be as shown in Table 1:

[0102] File Name Storage block data information Actual data size 0.log 0-40 10MB 1.log 41-60 10MB 2.log 61 8MB

[0103] Table 1

[0104] The database includes log identification information. Figure 2 In the corresponding scenario, the log identifier information could be (2.log, 8MB).

[0105] exist Figure 2 In the corresponding scenario, when a write operation receives a batch of data as a parameter, the batch size (Batch_Size) can be determined. Based on the log identifier information, the actual data size of the current log file can be determined to be 8MB, leaving 2MB of remaining space. Comparing Batch_Size with 2MB determines whether 2.log can store the batch. If Batch_Size is less than or equal to 2MB, the batch can be written starting from the 8MB position in 2.log, and the batch can have a consecutive block number of 62. If Batch_Size is 1MB, then the log identifier information at this time is (2.log, 9MB).

[0106] When consensus is reached on block 40, i.e., when checkpoint information for block 40 is generated, block data from 0 to 40 can be committed to the persistent database. This means committing all data in 0.log to the persistent database, deleting all data in 0.log, and changing 0.log to 0.rc. In this embodiment, the log file after data commit is not deleted, but becomes a reclaimed log file. At this point, the log files in the database can be... Figure 3 As shown.

[0107] exist Figure 3 In the scenario shown, the log file directory can be as shown in Table 2:

[0108]

[0109]

[0110] Table 2

[0111] exist Figure 3 In the scenario shown, if we want to commit data from block 52, which corresponds to an 8MB location, the commit identifier can be (1.log, 8MB). The log file directory can then be as shown in Table 3.

[0112] File Name Storage block data information Actual data size 0.rc 0MB 1.log 53-60 2MB 2.log 61-62 8MB

[0113] Table 3

[0114] If the device crashes and restarts at this time, the database will also restart. At this time, you can traverse the log file directory in the database, identify the file with the .log extension, and then store all the block data from 8MB to 10MB in 1.log into memory according to the commit identifier information (1.log, 8MB); according to the log identifier information (2.log, 9MB), store all the data from the initial position to the 9MB position in 2.log into memory.

[0115] If, during the consensus process, block 60 becomes inconsistent with block data on other nodes, it needs to be rolled back, meaning blocks 60-62 need to be deleted. At this point, block 60 can be deleted from 1.log, and blocks 61 and 62 from 2.log. If block 60 is 1MB in size, the log identifier could be (1.log, 9MB). Since all blocks in 2.log have been deleted, 2.log can be renamed to 2.rc, which is a reclaimed log file. The log files in the database can then be... Figure 5 As shown. The log file directory at this point can be as follows:

[0116] As shown in Table 4:

[0117] File Name Storage block data information Actual data size 0.rc 0MB 1.log 53-59 2MB 2.rc 0MB

[0118] Table 4

[0119] If in Figure 4 If the database is restarted in the scenario shown, the non-reclaimed log file 1.log can be obtained by traversing the log file directory. Based on the commit identifier information (1.log, 8MB) and the log identifier information (1.log, 9MB) at this time, the block data of 1.log from 8MB to 9MB can be restored to memory.

[0120] In this embodiment, the log file size remains fixed throughout the entire process without physically changing it. Furthermore, there is no direct correspondence between the log file and the MemTable in memory; both sizes are configurable and do not need to be identical. Therefore, data rollback or commit can be performed midway through the log file process. Even after committing the data, the size of the recycled data file remains unchanged, ensuring that metadata is not written when the log file is written to disk, thus improving data storage efficiency.

[0121] Reference Figure 6 This diagram illustrates a data storage device according to an embodiment of this application, which may specifically include a receiving module 61, a determining module 62, a first storage module 63, a second storage module 64, a disk writing module 65, and a cache module 66, wherein:

[0122] Receiving module 61 is used to receive batch data, the batch data including multiple data write operations;

[0123] Module 62 is used to determine the current log file in the database;

[0124] The first storage module 63 is used to write the batch data into the current log file if the remaining data capacity of the current log file is greater than or equal to the data size of the batch data.

[0125] The second storage module 64 is used to write the batch data into the recycling log file in the database if the remaining data capacity is less than the data size. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written into the persistent database.

[0126] The disk writing module 65 is used to write the file content of the target file to the disk, wherein the target file includes the current log file and the recycling log file;

[0127] The cache module 66 is used to write the batch data into memory.

[0128] In one possible implementation, the above-mentioned device further includes:

[0129] A creation module is used to create a new log file if the recycling log file does not exist in the database, and to store batch data larger than the remaining data capacity in the new log file.

[0130] In one possible implementation, the database includes log identification information, which includes the file name and write position of the current log file. Batch data with a size less than or equal to the remaining data capacity is written to the current log file at the write position. The write position is characterized by the actual data size of the data stored in the current log file. The log file in the database has a fixed data capacity, and the remaining data capacity is determined by the fixed data capacity of the log file and the write position.

[0131] In one possible implementation, the database includes commit identifier information used to identify the position of the last block of data in the log file that has been committed to the persistent database. The block of data has a block identifier and is a batch of data. The apparatus further includes:

[0132] A submission instruction receiving module is used to receive data submission instructions, wherein the data submission instructions have a corresponding first block identifier;

[0133] The persistence module is used to write block data whose block identifier is less than or equal to the first block identifier into the persistent database;

[0134] The first deletion module is used to delete block data from the log file of the database and the memory where the block identifier is less than or equal to the first block identifier;

[0135] The recycling module is used to write all the block data into the log file of the persistent database as the recycling log file;

[0136] The first update module is used to update the commit identifier information according to the first block identifier if some of the block data has been written to the log file of the persistent database.

[0137] In one possible implementation, the above-mentioned device further includes:

[0138] The first startup module is used to create a new log file in the database if the database is started for the first time.

[0139] The second startup module is used to restore the memory based on the commit identifier information and the write location if the database is not being started for the first time.

[0140] In one possible implementation, the second startup module mentioned above includes:

[0141] A read submodule is used to read the log directory of the database, the log directory including at least one log file;

[0142] The determination submodule is used to determine target log files that do not belong to the reclaimed log files;

[0143] The recovery submodule is used to restore the block data in the target log file between the position corresponding to the commit identifier information and the write position to the memory.

[0144] In one possible implementation, the above-mentioned device further includes:

[0145] A rollback instruction receiving module is used to receive data rollback instructions, wherein the data rollback instructions have a corresponding second block identifier;

[0146] The second deletion module is used to delete block data from the log file of the database and the memory where the block identifier is greater than or equal to the second block identifier;

[0147] The second update module is used to update the log identifier information according to the second block identifier.

[0148] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0149] Figure 7 A schematic diagram of the structure of a computer device provided in an embodiment of this application. For example... Figure 7 As shown, the computer device 7 of this embodiment includes: at least one processor 70 ( Figure 7 (Only one is shown) a processor, a memory 71, and a computer program 72 stored in the memory 71 and executable on the at least one processor 70, which, when executing the computer program 72, implements the steps in any of the above method embodiments.

[0150] The computer device 7 can be a desktop computer, laptop, handheld computer, or cloud server, etc. This computer device may include, but is not limited to, a processor 70 and a memory 71. Those skilled in the art will understand that... Figure 7 The computer device 7 is merely an example and does not constitute a limitation on the computer device 7. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0151] The processor 70 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0152] In some embodiments, the memory 71 may be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. In other embodiments, the memory 71 may be an external storage device of the computer device 7, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the computer device 7. Furthermore, the memory 71 may include both internal and external storage units of the computer device 7. The memory 71 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 71 can also be used to temporarily store data that has been output or will be output.

[0153] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0154] This application provides a computer program product that, when run on a computer device, enables the computer device to perform the steps described in the above-described method embodiments.

[0155] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data storage method, characterized in that, include: Receive batch data, wherein the batch data includes multiple data write operations; Identify the current log file in the database. The data stored in the log file is used to restore the data in memory after the database restarts. If the remaining data capacity of the current log file is greater than or equal to the data size of the batch data, then the batch data is written to the current log file; If the remaining data capacity is less than the data size, the batch data is written to the recycling log file in the database. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written to the persistent database. Write the contents of the target file to the disk, wherein the target file includes the current log file and the recycling log file; Write the batch data into memory; The database includes commit identifier information, which is used to identify the position of the last block of data in the log file that has been committed to the persistent database. The block of data has a block identifier and is a batch of data. The method further includes: Receive a data submission instruction, wherein the data submission instruction has a corresponding first block identifier; Write the block data whose block identifier is less than or equal to the first block identifier to the persistent database; Delete block data whose block identifier is less than or equal to the first block identifier from the database log file and the memory; The log file in the persistent database containing all the block data is used as the recycling log file. If some of the block data is written to the log file of the persistent database, the commit identifier information is updated according to the first block identifier.

2. The method as described in claim 1, characterized in that, The method further includes: If the recycling log file does not exist in the database, a new log file is created, and batch data larger than the remaining data capacity is stored in the new log file.

3. The method as described in claim 1, characterized in that, The database includes log identification information, which includes the file name and write position of the current log file. Batch data whose data size is less than or equal to the remaining data capacity is written to the current log file at the write position. The write position is characterized by the actual data size of the data stored in the current log file. The log file in the database has a fixed data capacity, and the remaining data capacity is determined by the fixed data capacity of the log file and the write position.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: If the database is being started for the first time, a new log file is created in the database; If the database is not being started for the first time, the memory is restored based on the commit identifier information and the write location.

5. The method as described in claim 4, characterized in that, The step of restoring the memory based on the commit identifier information and the write location includes: Read the log directory of the database, the log directory including at least one log file; Identify the target log file that does not belong to the reclaimed log file category; The block data in the target log file between the position corresponding to the commit identifier information and the write position is restored to the memory.

6. The method as described in claim 3, characterized in that, The method further includes: Receive a data rollback instruction, wherein the data rollback instruction has a corresponding second block identifier; Delete block data whose block identifier is greater than or equal to the second block identifier from the database log file and the memory; Update the log identifier information according to the second block identifier.

7. A data storage device, characterized in that, include: The receiving module is used to receive batch data, which includes multiple data write operations; The determination module is used to determine the current log file in the database. The data stored in the log file is used to restore the data in memory after the database restarts. The first storage module is configured to write the batch data into the current log file if the remaining data capacity of the current log file is greater than or equal to the data size of the batch data. The second storage module is used to write the batch data into the recycling log file in the database if the remaining data capacity is less than the data size. The recycling log file is a log file in which batch data has been written before and all the written batch data has been written into the persistent database. The disk writing module is used to write the file content of the target file to the disk, wherein the target file includes the current log file and the recycling log file; A caching module is used to write the batch data into memory; The database includes commit identifier information, which is used to identify the position of the last block of data in the log file that has been committed to the persistent database. The block of data has a block identifier and is a batch of data. The device is further used to: Receive a data submission instruction, wherein the data submission instruction has a corresponding first block identifier; Write the block data whose block identifier is less than or equal to the first block identifier to the persistent database; Delete block data whose block identifier is less than or equal to the first block identifier from the database log file and the memory; The log file in the persistent database containing all the block data is used as the recycling log file. If some of the block data is written to the log file of the persistent database, the commit identifier information is updated according to the first block identifier.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

Images