XLOG log compression method based on OpenGauss platform

Abstract

The present application relates to a kind of XLOG log compression methods based on OpenGauss platform.This method is compressed, only the data after XLOG record header is compressed, after all registered XLOG information needing to be connected to the link list in XLogRecordAssemble function, all data are copied to memory, data are compressed and a new node is generated to replace original node, then setting flag bit.This method is compressed by XLOG, so that when XLOG is landed, disk IO is reduced, while reducing the occupation of disk space, when sending XLOG log, network IO can also be reduced.Testing shows that, in the case of using the present application to open XLOG compression, compression rate can reach about 21.8%, IO performance will be improved by 200%, it can be seen that using the present application can better play the performance of database, improve the work efficiency of database.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of XLOG log compression, and particularly relates to an XLOG log compression method and system based on an OpenGauss platform. BACKGROUND

[0002] Write-Ahead Logging is a standard method to guarantee data integrity. Simply put, the concept of WAL is that changes to data files (including tables and indexes) must first be written to the log, that is, the log record is flushed to permanent storage before being overwritten. Following this process, there is no need to flush data pages to disk at each transaction commit, because the log can be used to recover the database in the event of a crash: any changes that have not been applied to the data pages can be redone (i.e. rolled back) according to the log record.

[0003] As can be seen, using WAL can significantly reduce the number of times the disk is written, because only the log file needs to be flushed to the disk to ensure that the transaction is committed, and each data file modified by the transaction does not need to be flushed to the disk. Since the log file is written continuously, the cost of synchronizing the log is much less than the cost of flushing the data page, especially when the server is processing a large number of small transactions involving different parts of the data storage. In addition, when the server is processing a large number of parallel small transactions, one fsync of the log file can commit multiple transactions.

[0004] The skilled person finds that when the amount of data written by XLOG records is too large or too frequent, the amount of disk write IO data is too large, and the network IO burden is large when sending XLOG logs, and compressing the records before the XLOG data is landed can well avoid this problem.

[0005] OpenGauss is an open source relational database management system, and its kernel originates from PostgreSQL. In OpenGauss, WAL compression can be enabled by setting wal_compression=enable: using LZ compression method to compress XLOG records with page image. In this case, the XLogRecordBlockCompressHeader structure will be added. This function has two advantages and one disadvantage, the advantages are to reduce the I / O overhead of writing records and reduce the consumption of WAL segment files; the disadvantage is to consume more CPU resources to perform compression operations, and it cannot compress other types of XLOG records except page image.

[0006] At present, there is no ideal solution to the above problems. SUMMARY

[0007] In order to overcome the problem of excessive CPU resource consumption and the inability to compress other types of XLOG records except for full-page images in the XLOG log compression method of the OpenGauss platform, a solution is proposed.

[0008] The present application proposes a new XLOG log compression method based on the OpenGauss platform. By compressing XLOG, we reduce disk IO when XLOG is written to disk, reduce the occupation of disk space, and reduce network IO when sending XLOG logs.

[0009] OpenGauss has provided a method for compressing XLOG records (XLOG log data) with full-page images using the LZ compression method. The compression unit of the present method is a single XLOG record. In each XLOG record, the highest bit of xl_tot_len in the structure XLogRecord is used as a flag bit. If the flag bit is 1, it indicates that this is a compressed XLOG record. If the flag bit is 0, it indicates that this is an uncompressed XLOG record. In the code, a mask is used wherever xl_tot_len is used to exclude the influence of the highest bit.

[0010] When compressing, only the data after the XLOG record header (the actual data node data) is compressed. After the XLogRecordAssemble function connects all registered XLOG information that needs to be written to disk to a linked list, all data is copied to a block of memory, the data is compressed, and a new node is generated to replace all the original nodes. Then, the flag bit in the xlog record header is set to 1, marking the highest bit of xl_tot_len as 1.

[0011] When decompressing, only the data after the XLOG record header is decompressed. After the XLogReadRecord function reads a single XLOG record, the data is saved in a block of memory. First, it is determined whether the highest bit of xl_tot_len of the XLOG record is 1. If it is 1, it means that this is a compressed XLOG record that needs to be decompressed, and the decompressed data is copied back to the previous memory. If it is 0, no operation is needed.

[0012] Specifically, in a first aspect, the present application provides an XLOG log compression method based on the OpenGauss platform, comprising:

[0013] S1. Assemble XLOG log data into WAL records and save them in a linked list. The assembled WAL records include header information nodes and actual data nodes.

[0014] S2. Copying the data of the actual data node of the WAL record in the chain table into a piece of memory;

[0015] S3. Compressing the actual data node data in the memory and generating a new node;

[0016] S4. Replacing the original actual data node of the WAL record with the new node obtained in the previous step, and setting a mark bit in the header information node (xlog record header);

[0017] S5. Identifying the mark bit in the header information node of the XLOG log data before reading the XLOG log data;

[0018] S6. When the mark bit in the header information node shows compressed data, decompressing the data and copying the decompressed data into the memory to replace the original compressed data.

[0019] Further, the XLOG log compression method based on the OpenGauss platform of the present application assembles the XLOG log data into a WAL record and saves it in a chain table in step S1, comprising:

[0020] S11. When registering the XLOG log data, first call the XLogBeginInsert function to enter the registration state;

[0021] S12. Call the registration function in XlogRegister to register the XLOG log data in the buffer area;

[0022] S13. Assemble all the data registered in the buffer area into a WAL record through the function XLogRecordAssemble;

[0023] S14. Save the WAL record in the chain table.

[0024] Further, the XLOG log compression method based on the OpenGauss platform of the present application compresses the actual data node data in the memory and generates a new node in step S3, wherein the compression method used is the LZ compression method provided in OpenGauss.

[0025] Further, the XLOG log compression method based on the OpenGauss platform of the present application sets a mark bit in the header information node in step S4, comprising:

[0026] S41. The highest bit of xl_tot_len in the structure body XLogRecord in the XLOG log is used as a mark bit;

[0027] S42. If the XLOG log data is not compressed, mark the highest bit of xl_tot_len as 0;

[0028] S43. If the XLOG log data is compressed, mark the highest bit of xl_tot_len as 1.

[0029] Further, the application based on the OpenGauss platform XLOG log compression method step S5 is described before reading the XLOG log data, first identifying the mark bit in the WAL record header information node, including:

[0030] S51. Before reading the XLOG log data, the XlogReadRecord function calls the ReadPageInternal function to read the XLOG page information into a block of memory;

[0031] S52. The location of the XLOG log data is searched in the memory, and the XLogRecord is generated;

[0032] S53. Identify and judge the mark bit in the XLogRecord.

[0033] Further, the application based on the OpenGauss platform XLOG log compression method step S6 is described to decompress the data, and copy the decompressed data to the memory to replace the original compressed data, which means calling the decompression algorithm to decompress the compressed data of the actual data node in the XLOG log data WAL record, and then copying the decompressed data to the previous memory to replace the original compressed data of the actual data node.

[0034] Secondly, the application provides an XLOG log compression system based on the OpenGauss platform, the system includes:

[0035] Record assembly module: for assembling XLOG log data into WAL records and saving in a linked list, the assembled WAL records include header information nodes and actual data nodes;

[0036] Function management module: for managing and calling XLogBeginInsert function, XlogRegister function, XLogRecordAssemble function, XlogReadRecord function;

[0037] Compression module: for compressing data and generating new nodes;

[0038] Mark bit setting module: for setting mark bits in the header information node of the WAL record;

[0039] Mark bit identification module: for identifying mark bit information;

[0040] Decompression module: used for decompressing data.

[0041] In a third aspect, the present application also provides a computer readable storage medium, wherein the storage medium stores a computer program, and the program is executed by a processor to implement the steps of the XLOG log compression method based on the OpenGauss platform.

[0042] To sum up, the present application proposes a new XLOG log compression method based on the OpenGauss platform. On the basis of the original XLOG, the XLOG compression is used to reduce the disk IO when the XLOG is written to the disk, reduce the occupation of the disk space, and reduce the network IO when the XLOG log is sent. It is tested that in the case of starting the XLOG compression by using the method, the compression rate can reach about 21.8%, the CPU performance will decrease by 0.9%, and the IO performance will increase by 200%. It can be seen that the use of the method can help to better play the performance of the database and significantly improve the work efficiency of the database. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings used in the embodiments of the present application. Obviously, the following drawings are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0044] Figure 1 It is a schematic diagram of the internal structure of the WAL segment file used in the present application.

[0045] Figure 2 It is a schematic diagram of detailed information of the internal structure of the XLOG record in the present application.

[0046] Figure 3 It is a schematic diagram of the overall implementation process of the method of the present application.

[0047] Figure 4 It is a schematic diagram of the implementation process of step S1 of the method of the present application.

[0048] Figure 5 It is a schematic diagram of the implementation process of step S4 of the method of the present application.

[0049] Figure 6 It is a schematic diagram of the implementation process of step S5 of the method of the present application.

[0050] Figure 7 It is a schematic diagram of the composition structure of the XLOG log compression system based on the OpenGauss platform of the present application. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in connection with specific embodiments and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments of the present application. The present application can also be implemented or applied by using different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the present application.

[0052] It should be understood that the scope of protection of the present application is not limited to the following specific embodiments; it should also be understood that the terms used in the embodiments of the present application are used to describe the specific embodiments, rather than to limit the scope of protection of the present application.

[0053] Embodiment: An XLOG log compression method based on an OpenGauss platform

[0054] Term explanation:

[0055] OpenGauss database: an open source relational database management system, which is evolved based on the open source database PostgreSQL.

[0056] WAL segment file: the default size of a WAL segment file is 16 MB, and it is internally divided into multiple pages with a size of 8192B (8 KB). The first page contains header data defined by XLogLongPageHeaderData, and the other pages contain header data defined by XLogPageHeaderData. After the header data of each page, the WAL records are written in descending order (see Figure 1 ) immediately.

[0057] XLOG record: the data part of the XLOG record can be divided into two parts, header and data (see Figure 2 ).

[0058] Firstly, the compression in the process of writing XLOG record is introduced. When inserting XLOG log, the following stages are passed: registering data, inserting data, and writing data to disk. When registering data, XLogBeginInsert function is called to enter the registration state, then various registration functions in XlogRegister are called to register data to the buffer, and then all registered data is assembled into WAL record by function XLogRecordAssemble, and the data is saved in a linked list, including a header information node and an actual data node. The data of the actual data node in the linked list is copied to a piece of memory, compressed by using a compression algorithm, and a new node is generated to replace the original actual data node, and a flag bit is set in the header information node, and the highest bit of xl_tot_len is marked as 1.

[0059] Next, the decompression in the process of reading XLOG record is introduced. When reading record, XlogReadRecord function will call ReadPageInternal to read XLOG page information to a piece of memory, search the position of XLOG record in the memory, and generate XLogRecord. When decompressing, it is judged whether the highest bit of xl_tot_len in XLogRecord is 1. If it is 1, it means that the data in the memory is compressed, and the decompression algorithm is called to decompress the data after the header of XLOG record, and then the decompressed data is copied back to the previous memory to replace the original data after the header of XLOG; otherwise, no operation is performed.

[0060] Specifically, the XLOG log compression method based on the OpenGauss platform of the present application comprises (see Figure 3 ) :

[0061] S1. Assemble XLOG log data into WAL record and save it in a linked list, and the assembled WAL record includes a header information node and an actual data node, comprising (see Figure 4 ) :

[0062] S11. When registering XLOG log data, XLogBeginInsert function is called to enter the registration state;

[0063] S12. The registration functions in XlogRegister are called to register XLOG log data to the buffer;

[0064] S13. All data registered in the buffer is assembled into WAL record by function XLogRecordAssemble;

[0065] S14. The WAL record is saved in a linked list.

[0066] S2. Copy the actual data node data of the WAL record in the linked list to a piece of memory.

[0067] S3. Compress the actual data node data in the memory using the LZ compression method provided in OpenGauss and generate a new node.

[0068] S4. Replace the original actual data node of the WAL record with the new node obtained in the previous step, and set the flag bit in the header information node (xlog record header), including (see Figure 5 ) :

[0069] S41. The highest bit of xl_tot_len in the structure XLogRecord in the XLOG log is used as a flag bit;

[0070] S42. If the XLOG log data is not compressed, mark the highest bit of xl_tot_len as 0;

[0071] S43. If the XLOG log data has been compressed, mark the highest bit of xl_tot_len as 1.

[0072] S5. Before reading the XLOG log data, identify the flag bit in the header information node of the WAL record, including (see Figure 6 ) :

[0073] S51. Before reading the XLOG log data, the XlogReadRecord function calls ReadPageInternal to read the XLOG page information into a piece of memory;

[0074] S52. Search for the location of the XLOG log data in the memory and generate XLogRecord;

[0075] S53. Identify and determine the flag bit in XLogRecord.

[0076] S6. When the flag bit in the header information node shows compressed data, call the decompression algorithm to decompress the compressed data of the actual data node in the WAL record of the XLOG log data, and then copy the decompressed data to the previous memory to replace the original compressed data of the actual data node.

[0077] An XLOG log compression system based on the OpenGauss platform, the system includes (see Figure 7 ) :

[0078] A record assembling module is configured to assemble XLOG log data into a WAL record and save the WAL record in a linked list, and the assembled WAL record comprises a header information node and an actual data node;

[0079] A function management module is configured to manage and call an XLogBeginInsert function, an XlogRegister function, an XLogRecordAssemble function and an XlogReadRecord function;

[0080] A compression module is configured to compress data and generate a new node;

[0081] A flag bit setting module is configured to set a flag bit in the header information node of the WAL record;

[0082] A flag bit identification module is configured to identify flag bit information;

[0083] A decompression module is configured to decompress data.

[0084] The modules are implemented according to the XLOG log compression method based on the OpenGauss platform.

[0085] The embodiments in the present application are described in a progressive manner, and the same or similar parts of the embodiments can be referred to each other.

[0086] The above only describes the embodiments of the present application and is not used to limit the present application. The present application can be variously changed and modified by those skilled in the art. Any modification, replacement and the like within the spirit and principle of the present application shall be included in the protection scope of the claims of the present application.

Claims

1. A method for XLOG log compression based on the OpenGauss platform, characterized in that, The method includes: S1. Assemble the XLOG log data into WAL records and store them in a linked list. The assembled WAL records include header information nodes and actual data nodes. S2. Copy the data of the actual data nodes recorded in the WAL of the linked list to a block of memory; S3. Compress the actual data node data in memory and generate a new node; S4. Replace the original WAL record's actual data node with the new node obtained in the previous step, and set a flag bit in the header information node; S5. Before reading XLOG log data, identify the flag bits in the header information node of its WAL record; S6. When the flag bit in the header information node shows compressed data, decompress the data and copy the decompressed data into memory to replace the original compressed data.

2. The XLOG log compression method based on the OpenGauss platform according to claim 1, characterized in that, Step S1, which involves assembling XLOG log data into WAL records and storing them in a linked list, includes: S11. When registering XLOG log data, first call the XLogBeginInsert function to enter the registration state; S12. Call the registration function in XlogRegister to register the XLOG log data into the buffer; S13. Assemble all the data registered in the buffer into a WAL record using the function XLogRecordAssemble; S14. Store the WAL record in a linked list.

3. The XLOG log compression method based on the OpenGauss platform according to claim 1, characterized in that, Step S3 describes compressing the actual data node data in memory and generating a new node, using the LZ compression method provided by OpenGauss.

4. The XLOG log compression method based on the OpenGauss platform according to claim 1, characterized in that, Setting the flag bit in the header information node as described in step S4 includes: S41. Use the highest bit of xl_tot_len in the XLogRecord structure of the XLOG log as the flag bit; S42. If the XLOG log data is not compressed, mark the highest bit of xl_tot_len as 0; S43. If the XLOG log data has been compressed, mark the highest bit of xl_tot_len as 1.

5. The XLOG log compression method based on the OpenGauss platform according to claim 1, characterized in that, Step S5 describes identifying the flag bits in the WAL record header information node before reading the XLOG log data, including: S51. Before reading XLOG log data, the XlogReadRecord function calls ReadPageInternal repeatedly to read XLOG page information into a block of memory; S52. Locate the XLOG log data in memory and generate an XLogRecord; S53. Identify and determine the flag bits in XLogRecord.

6. The XLOG log compression method based on the OpenGauss platform according to claim 1, characterized in that, The step S6, which involves decompressing the data and copying the decompressed data into memory to replace the original compressed data, refers to calling a decompression algorithm to decompress the compressed data of the actual data nodes in the XLOG log data WAL record, and then copying the decompressed data into memory to replace the original compressed data of the actual data nodes.

7. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the XLOG log compression method based on the OpenGauss platform as described in any one of claims 1-6.

Images