A method and system for saving storage space of a value log system

Abstract

The present invention discloses an implementation method and system for saving storage space of a value log system. When the value log system performs an update operation, the storage redundancy of data can be greatly reduced. When the value log system reads records in a value log file, the time consumption of the reading operation can be less affected.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and in particular to a method and system for saving storage space of a value log system. Background Art

[0002] Databases and various key-value data processing systems based on value logging convert update and delete operations into insert operations, preserving the entire data modification history. In practical applications, if there are a large number of update and delete operations, using value logging can consume significant disk space due to preserving all historical data versions. The reason for this is as follows: Each inserted or updated record in the value logging system stores all key-value data for the corresponding key. When a user performs an update operation, the index is used to locate the location of the previous data in the value log file. After modifying the previous data in memory, the update operation is converted into an insert operation with a deletion mark and an insert operation of the updated data, which are then appended sequentially to the end of the value log file. This implementation leverages the high-performance sequential read and write characteristics of the log file, significantly improving write performance. While the aforementioned update implementation maintains a high write speed, it consumes significant write space. Each update operation, regardless of the amount of content updated, also stores a copy of the key-value data that has not been updated. If the update is repeated N times, in the worst case, N+1 copies of the fields that have not been updated will be stored. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide an implementation method and system for saving storage space of a value log system, which can reduce the occupation of storage medium space and reduce the impact of time-consuming operations on reading value log files.

[0004] To solve the above technical problems, the present invention provides a method for saving storage space of a value log system, comprising the following steps:

[0005] Step 1: When the user updates data, the value log system first uses the index to find the LSN value of the record to be updated, reads the corresponding physical record according to the LSN value, and then cyclically reads the physical record according to the LSN value of the last operation recorded in REC until all column values ​​of the column to be updated are found.

[0006] Step 2: Two judgments will be made to determine how to write the update to disk.

[0007] Step 3: When the user continues to update, repeat the above steps.

[0008] Step 4: After the update is completed, the user needs to read the data in the log. The log system obtains the latest LSN value of the log record corresponding to the query key based on the index.

[0009] Step 5: The value log system reads the latest record of this record according to the LSN value, and reads the record in a loop according to the PREV_LSN in the record until all the Key-Value data required by the user are read;

[0010] Step 6: Combine the values ​​corresponding to all the operations read to obtain the value the user needs to access and return it to the user.

[0011] Preferably, in step 1, the physical record is represented by REC, and the physical length of the complete record, the number of updates, and the LSN of the last update are recorded at the end of the record, which is recorded as PREV_LSN.

[0012] Preferably, in step 2, two judgments are performed to determine how to write the update operation to the disk. Judgment 1: add 1 to the number of updates recorded in the REC read for the first time in step 1 to determine whether the maximum number of partial updates has been exceeded. Judgment 2: add the actual lengths of the different columns of the partial updates in the physical records previously read in a loop and the actual length of the column update operation this time. If the same column is updated multiple times, only the current time is calculated, and the ratio is calculated with the length of the complete physical record recorded in REC to determine whether the maximum ratio of the length of the partial update to the length of the entire physical record has been exceeded. When judgment 1 or judgment 2 is successful, based on the latest values ​​of all columns read in a loop, combined with the modification of the values ​​involved in the update operation, a complete insert record is formed. The number of updates in the tail of the physical record is set to 0. Because this is a complete record, the length of the complete physical record is set to the physical length of this record, and PREV_LSN is set to the LSN of the record before the update, and it is appended to the tail of the value log file. When both judgment 1 and judgment 2 are unsuccessful, this update operation only saves the key of the record to be updated and the updated partial value (such as the column number and column value of the updated column) in the value log file, and puts the update count + 1 at the end of the record. The length of the complete physical record is set to the physical length of the previous complete record, and PREV_LSN is set to the LSN of the physical record before the update.

[0013] Preferably, in step 5, in the best case, the user only needs to use one I / O operation to read the required data. In the worst case, the user needs to read in a loop until the most recent complete record that saves all Key-Value data is found.

[0014] Accordingly, a system for saving storage space in a value log system includes a scheduling module, an indexing module, and a value log storage module. Upon receiving update and query requests from users, the scheduling module calls the indexing module to perform updates and queries, stores the results in the value log storage module, and returns the results to the user.

[0015] Preferably, when a user initiates a request to update a record, the scheduling module will receive the user's update request, first call the index module to query the latest LSN value of the record that needs to be modified based on the Key given by the user, and then find the latest record in the log storage module based on the LSN value. Depending on whether the record contains all the fields that need to be modified, decide whether it is necessary to search forward again through the log storage module through the PREV_LSN in the record. After all records are read, two judgments are made. After the actual updated records are assembled in the memory, they are stored through the value log storage module, and then the record's Key value and the latest LSN value generated by the actual storage are updated to the index module, and finally the execution result is returned to the user.

[0016] Preferably, when a user initiates a query request, the scheduling module will receive the user's query request, first call the index module to query the latest LSN value of the record to be queried based on the key given by the user, and then find the latest record in the log storage module based on the LSN value. Depending on whether the record contains all the fields to be queried, decide whether it is necessary to search forward again through the log storage module through the PREV_LSN in the record. After all records are read, the records are merged in memory and returned to the user.

[0017] The beneficial effects of the present invention are as follows: the present invention improves the update operation performed on the duty system, and optimizes the original writing of a complete record containing modified and unmodified data for each update operation to only writing the modified data into the value log. Assuming that the user creates a record with many fields and frequently modifies several of them, the improved method of the present invention will greatly save the amount of stored data; in addition, the present invention limits the number and size of incomplete records written. This improved method is intended to optimize the query speed. Assuming that there is no restriction on the writing of incomplete records, when the user wants to query the fields that have not been modified during the query, because the complete record only has the oldest inserted record, the user needs to continuously search forward through PREV_LSN until the oldest inserted record is found, which will traverse all modified records, greatly slowing down the query time. Therefore, by limiting the number and size of incomplete records written, a complete record is updated when the restriction conditions are met. In the worst case during the query, the values ​​of all fields can be obtained after finding the most recent complete record, thereby optimizing the impact on the query time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 The present invention is a schematic diagram of a data writing process of a method for saving storage space of a value log system.

[0019] Figure 2 The present invention is a data reading process diagram of a method for saving storage space of a value log system. DETAILED DESCRIPTION

[0020] Figure 1 This is a flowchart of a method for implementing a value log system to save storage space and a system data writing process provided by Example 1 of the present invention. This embodiment 1 can be used to implement a method for implementing a value log system to save storage space and a system data writing process. Figure 1 The method shown specifically comprises the following steps:

[0021] Step 1. When the user updates data, the value log system will first use the index to find the LSN value of the record to be updated, read the corresponding physical record according to the LSN value (the physical record is represented by REC, and the physical length of the complete record, the number of updates, and the LSN of the last update are recorded at the end of the record, recorded as PREV_LSN), and then cyclically read the physical record according to the LSN value of the last operation recorded in REC until all column values ​​of the column to be updated are found.

[0022] Step 2: Add 1 to the update count of the record in the REC first read in step 1 to determine whether the maximum number of partial updates has been exceeded. If so, based on the latest values ​​of all columns read in a loop, combined with the value modifications involved in this update operation, a complete insert record is formed (the update count at the end of the physical record is set to 0, because this is a complete record, the length of the complete physical record is set to the physical length of this record, and PREV_LSN is set to the LSN of the record before the update), and appended to the end of the value log file. If the maximum number of partial updates has not been exceeded, proceed to step 3.

[0023] Step 3. Add up the actual lengths of the different columns that were partially updated in the physical records that were previously read in a loop and the actual length of the column update operation this time (if the same column is updated multiple times, only the current time is calculated), and calculate the ratio with the length of the complete physical record recorded in REC to determine whether it exceeds the maximum ratio of the length of the partial update to the length of the entire physical record. If the maximum ratio is exceeded, based on the latest values ​​of all columns read in the loop, combined with the modification of the values ​​involved in the update operation, a complete insert record is formed (the number of updates in the tail of the physical record is set to 0, because this is a complete record, the length of the complete physical record is set to the physical length of this record, and PREV_LSN is set to the LSN of the record before the update), and append it to the tail of the value log file. If it does not exceed the maximum ratio, proceed to step 4.

[0024] Step 4: This update operation saves only the key of the record to be updated and the updated partial value (such as the column number and column value of the updated column) in the value log file, and places the update count + 1 at the end of the record. The length of the complete physical record is set to the physical length of the previous complete record, and PREV_LSN is set to the LSN of the physical record before the update.

[0025] Example 1:

[0026] The following explains the whole process by taking the update statement in the database as an example. Assume that there is a table T1 with three columns: ID, NAME, and BALANCE. ID is the primary key column:

[0027] Table 1 Column information of table T1

[0028] Column Name Column Type Column meaning id varchar(20) Account ID, primary key name varchar(20) Account Name balance int Account balance

[0029] At this point, three new records have been inserted into the table:

[0030] Table 2 Data information of table T1

[0031] id name balance A001 Zhang San 100 A002 Li Si 200 A003 Wang Wu 300

[0032] Execute the UPDATE statement:

[0033] UPDATE T1 SET NAME='Zhao Liu'WHERE ID=A003;

[0034] S101, find the LSN value of the actual record stored in ID=A003 through the index file, and read this log record through the LSN value. Because it is a complete data record and meets the user's modified fields, no further record search is done.

[0035] S102, assuming that the maximum number of modifications is set to 2, the maximum modification ratio is set to 50%, the current number of modifications is the first, and the current modification ratio is 4 / (4+4+4)*100%=30%, both of which do not meet the judgment conditions, then partially update this record, set the Key update position of the new record in the memory to 1, set the Value to Zhao Liu, set the number of updates at the end of the record to 1, set the complete record length to 12, set PREV_LSN to the LSN value of the previous record, write this record to disk, and finally update the index to the LSN value of this record.

[0036] Execute the UPDATE statement:

[0037] UPDATE T1 SET BALANCE='400'WHERE ID=A003;

[0038] S103: The LSN value of the actual record with ID=A003 is found through the index file, and the log record is read out based on the LSN value. Because this record does not meet all the fields modified by the user, the original INSERT log is found based on the PREV_LSN value at the end of the record. At this point, all the fields modified by the user have been met, so no further search is performed.

[0039] S104, assuming that the maximum number of modifications is set to 2, the maximum modification ratio is set to 50%, the current number of modifications is the second, and the current cumulative modification ratio is 4+4 / (4+4+4)*100%=66%, and one of the two meets the judgment condition, then based on the latest values ​​of all columns read in a loop, combined with the value modifications involved in this update operation, a complete insert record is formed, and the number of updates in the tail of the physical record is set to 0. Because this is a complete record, the complete record length is set to 12, PREV_LSN is set to the LSN of the record before the update, this record is written to disk, and finally the index is updated to the LSN value of this record.

[0040] Figure 2 This is a method for implementing a value log system to save storage space and a flow chart of system data reading provided by embodiment 2 of the present invention. This embodiment 2 can be used to implement a method for implementing a value log system to save storage space and a flow chart of system data reading. Figure 2 The method shown specifically comprises the following steps:

[0041] Step 1: The user needs to read the data in the log. The log system obtains the latest LSN value of the log record corresponding to the query key based on the index.

[0042] Step 2: The value log system reads the most recent record based on the LSN value and loops through records based on the PREV_LSN in the record until it has read all the key-value data required by the user. In the best case, the user only needs a single I / O operation to read the required data. In the worst case, the user needs to loop through the record until the most recent record containing all the key-value data is found.

[0043] Step 3: Combine the values ​​corresponding to all the operations read to get the value the user needs to access and return it to the user.

[0044] Example 2:

[0045] Example 2 is explained by executing query statements after executing two UPDATE commands in Example 1.

[0046] After executing the first UPDATE command in Example 1, execute the query statement:

[0047] SELECT*FROM T1 WHERE ID=A003;

[0048] S201, the value log system obtains the latest LSN value of the record with ID=A003 according to the index.

[0049] S202, if the value log system finds that this record cannot satisfy all the fields queried by the user, it reads the previous record according to the PREV_LSN value in the record. Since the previous record is a complete record and can satisfy all the fields queried by the user, it stops searching for records forward.

[0050] S203, the value log system modifies NAME = 'Zhao Liu' based on the complete record in the memory, and finally returns the combined record to the user.

[0051] After executing the second UPDATE command in Example 1, execute the query statement:

[0052] SELECT*FROM T1 WHERE ID=A003;

[0053] S204, the value log system obtains the latest LSN value of the record with ID=A003 according to the index.

[0054] S205: If the value log system finds that the record is a complete record and can satisfy all the fields queried by the user, the record is directly returned to the user.

[0055] Accordingly, a system for saving storage space in a value log system includes a scheduling module, an indexing module, and a value log storage module. Upon receiving update and query requests from users, the scheduling module calls the indexing module to perform updates and queries, stores the results in the value log storage module, and returns the results to the user.

[0056] When a user initiates a request to update a record, the scheduling module will receive the user's update request. It will first call the index module to query the latest LSN value of the record that needs to be modified based on the key given by the user. Then, it will find the latest record in the log storage module based on the LSN value. Depending on whether the record contains all the fields that need to be modified, it will decide whether it is necessary to search forward again through the log storage module through the PREV_LSN in the record. After all records have been read, two judgments will be made. After the actual updated records are assembled in the memory, they will be stored through the value log storage module. Then, the record's key value and the latest LSN value generated by the actual storage will be updated to the index module. Finally, the execution result will be returned to the user.

[0057] When a user initiates a query request, the scheduling module receives the user's query request and first calls the index module to query the latest LSN value of the record to be queried based on the key provided by the user. Then, the scheduling module finds the latest record in the log storage module based on the LSN value. Depending on whether the record contains all the fields to be queried, it decides whether to search forward again through the log storage module based on the PREV_LSN in the record. After all records are read, the records are merged in memory and returned to the user.

[0058] The value log system of the present invention can greatly reduce data storage redundancy when performing an update operation, and can have a smaller impact on the time consumption of the reading operation when reading records in the value log file.

Claims

1. A method for saving storage space of a value log system, characterized in that: The steps include: Step 1: When the user updates data, the value log system first uses the index to find the LSN value of the record to be updated, reads the corresponding physical record based on the LSN value, and then cyclically reads the physical record based on the LSN value of the previous operation recorded in REC until all column values ​​of the column to be updated are found; Step 2. At this time, two judgments will be made to determine how to write the update operation to the disk; Two judgments will be made to determine how to write the update operation to the disk. Judgment 1: add 1 to the number of updates recorded in the REC read for the first time in step 1 to determine whether it exceeds the maximum number of partial updates. Judgment 2: add the actual lengths of different columns of partial updates in the physical records read in the previous loop and the actual length of this update column operation. If the same column is updated multiple times, only the current time is calculated, and the ratio is calculated with the length of the complete physical record recorded in REC to determine whether it exceeds the maximum ratio of the length of the partial update to the length of the entire physical record. When Judgment 1 or Judgment 2 is successful, based on the latest values ​​of all columns read in the loop, combined with this The update operation involves value modifications and merging to form a complete insert record. The update count at the end of the physical record is set to 0. This is a complete record, so the length of the complete physical record is set to the physical length of this record, and PREV_LSN is set to the LSN of the record before the update. It is appended to the end of the value log file. When both judgment 1 and judgment 2 are unsuccessful, this update operation only saves the key of the record to be updated and the updated partial value, such as the column number and column value of the updated column, in the value log file. The update count + 1 is placed at the end of the record, the length of the complete physical record is set to the physical length of the previous complete record, and PREV_LSN is set to the LSN of the physical record before the update. Step 3: When the user continues to update, repeat the above steps; Step 4: After the update is completed, the user needs to read the data in the log. The log system obtains the latest LSN value of the log record corresponding to the query key based on the index; Step 5: The value log system reads the latest record of this record according to the LSN value, and reads the record in a loop according to the PREV_LSN in the record until all the Key-Value data required by the user are read; Step 6: Combine the values ​​corresponding to all the operations read to obtain the value the user needs to access and return it to the user.

2. The method for saving storage space of a value log system according to claim 1, wherein: In step 1, the physical record is represented by REC, and the physical length of the complete record, the number of updates, and the LSN of the last update are recorded at the end of the record, which is recorded as PREV_LSN.

3. The method for saving storage space of a value log system according to claim 1, wherein: In step 5, in the best case, the user only needs to use one I / O operation to read the required data. In the worst case, the user needs to read in a loop until the most recent complete record that saves all the key-value data is found.

4. A system based on the method for implementing the storage space saving value log system according to claim 1, characterized in that: include: Scheduling module, indexing module and value log storage module; The scheduling module receives the user's update and query requests, calls the index module to update and query, stores them through the value log storage module, and returns the execution results to the user.

5. The system for implementing the method for saving storage space of a value log system according to claim 4, characterized in that: When a user initiates a request to update a record, the scheduling module will receive the user's update request. It will first call the index module to query the latest LSN value of the record that needs to be modified based on the key given by the user. Then, it will find the latest record in the log storage module based on the LSN value. Depending on whether the record contains all the fields that need to be modified, it will decide whether it is necessary to search forward again through the log storage module through the PREV_LSN in the record. After all records have been read, two judgments will be made. After the actual updated records are assembled in the memory, they will be stored through the value log storage module. Then, the record's key value and the latest LSN value generated by the actual storage will be updated to the index module. Finally, the execution result will be returned to the user.

6. The system for implementing the method for saving storage space of a value log system according to claim 4, characterized in that: When a user initiates a query request, the scheduling module receives the user's query request and first calls the index module to query the latest LSN value of the record to be queried based on the key provided by the user. Then, the scheduling module finds the latest record in the log storage module based on the LSN value. Depending on whether the record contains all the fields to be queried, it decides whether to search forward again through the log storage module based on the PREV_LSN in the record. After all records are read, the records are merged in memory and returned to the user.

Images