Data processing method and electronic device

Abstract

Embodiments of the present application provide a data processing method and electronic device. The method includes: in response to a first database operation instruction, determining a first buffer page group in a buffer pool of a database, the first buffer page group including multiple associated buffer pages; and compressing the data in the multiple buffer pages of the first buffer page group and writing the data to a disk, thereby improving the compression ratio and saving disk space.

Description

Technical Field

[0001] The present application relates to the field of databases, and more particularly, to a data processing method and an electronic device. Background Art

[0002] The database stores table tuples and index data on disk in pages, and data is read and written on a page-by-page basis. To improve access speed, the database uses an in-memory buffer pool to cache data. The buffer pool consists of multiple buffer pages, and the data cached in each buffer page corresponds to a page on disk.

[0003] The data cached in the buffer pool is eliminated to disk or loaded from disk in units of buffer pages. For each buffer page, when data is eliminated to disk, it can be compressed and stored on disk to save disk space. Correspondingly, when data is loaded from disk to the buffer page, the compressed data on disk is decompressed and then loaded into the buffer page. Since the buffer page size is a fixed value and the minimum physical space occupied on the disk is also fixed, the buffer page size directly affects the compression ratio. The set buffer page size may not meet the user's requirement for a high compression ratio. Summary of the Invention

[0004] The present application provides a data processing method and an electronic device to improve the compression ratio and save disk space.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] In response to a first database operation instruction, determining a first buffer page group in a buffer pool of the database, the first buffer page group including a plurality of associated buffer pages;

[0007] The data in the plurality of buffer pages of the first buffer page group is compressed and then written to a disk.

[0008] Optionally, before compressing the data in the plurality of buffer pages of the first buffer page group and writing the data to the disk, the method further includes:

[0009] adding input and output (IO) locks to the plurality of buffer pages in sequence according to the order of the pages corresponding to the plurality of buffer pages of the first buffer page group;

[0010] After compressing the data in the plurality of buffer pages of the first buffer page group and writing the data to the disk, the method further includes:

[0011] The IO locks of the plurality of buffer pages of the first buffer page group are released in sequence according to the reverse order of the pages to which the plurality of buffer pages correspond.

[0012] Optionally, also include:

[0013] According to the order of pages corresponding to the multiple buffer pages of the first buffer page group, for each buffer page, an exclusive lock is added to the buffer page in the buffer table of the buffer pool. If the exclusive lock is added successfully, a spin lock of the buffer description information is added to the buffer page. If the spin lock is added successfully, it is determined whether the buffer page is a dirty page. If none of the multiple buffer pages of the first buffer page group are dirty pages, the buffer description information of the multiple buffer pages of the first buffer page group is deleted, the spin locks of the multiple buffer pages of the first buffer page group are released, the records of the multiple buffer pages of the first buffer page group are deleted from the buffer table, and the exclusive locks of the multiple buffer pages of the first buffer page group are released.

[0014] Optionally, also include:

[0015] If adding an exclusive lock or a spin lock to any buffer page of the first buffer page group fails, determining a second buffer page group in the database buffer pool, where the second buffer page group includes multiple buffer pages;

[0016] The data in the plurality of buffer pages of the second buffer page group is compressed and then written to the disk.

[0017] Optionally, the first buffer page group includes a first buffer page and at least one second buffer page, and determining the first buffer page group in the database buffer pool includes:

[0018] Determining the first buffer page in the database buffer pool;

[0019] Based on the association relationship between the first buffer page and the second buffer page, the second buffer page is determined in the database buffer pool.

[0020] Optionally, also include:

[0021] In response to a second database operation instruction, determining a third buffer page group in the buffer pool, the third buffer page group including a plurality of associated buffer pages;

[0022] The data of the page corresponding to the second database operation instruction in the disk is decompressed and loaded into the multiple buffer pages of the third buffer page group.

[0023] Optionally, the third buffer page group includes a third buffer page corresponding to the second database operation instruction, and at least one fourth buffer page associated with the third buffer page;

[0024] Determining a third buffer page group in the buffer pool includes:

[0025] determining the third buffer page in the buffer pool;

[0026] Based on the association relationship between the third buffer page and the fourth buffer page, the fourth buffer page is determined in the buffer pool.

[0027] Optionally, also include:

[0028] IO locks are sequentially added to the multiple buffer pages of the third buffer page group according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group.

[0029] Optionally, sequentially adding IO locks to the multiple buffer pages of the third buffer page group according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group includes:

[0030] After determining the third buffer page in the buffer pool, adding an IO lock to the third buffer page;

[0031] If the page corresponding to the third buffer page is the first of the pages corresponding to the multiple buffer pages of the third buffer page group, after determining the fourth buffer page in the buffer pool, continue to add an IO lock to the fourth buffer page in the order of the pages corresponding to the fourth buffer page;

[0032] If the page corresponding to the third buffer page is not the first of the pages corresponding to the multiple buffer pages of the third buffer page group, the IO lock of the third buffer page is released, and after the fourth buffer page is determined in the buffer pool, IO locks are added to the multiple buffer pages of the third buffer page group in sequence according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group.

[0033] Optionally, after decompressing the data of the page corresponding to the second database operation instruction on the disk and loading the data into the multiple buffer pages of the third buffer page group, the method further includes:

[0034] The IO locks of the multiple buffer pages of the third buffer page group are released in sequence according to the reverse order of the pages to which the multiple buffer pages of the third buffer page group correspond.

[0035] Optionally, also include:

[0036] In response to a third database operation instruction, determining a fifth buffer page in the buffer pool;

[0037] The data in the fifth buffer page is compressed and then written to the disk.

[0038] Optionally, also include:

[0039] In response to a fourth database operation instruction, determining a sixth buffer page in the buffer pool;

[0040] The data of the page corresponding to the fourth database operation instruction in the disk is decompressed and loaded into the sixth buffer page.

[0041] In a second aspect, the present application provides an electronic device, comprising: a memory and a processor;

[0042] The memory is used to store computer programs;

[0043] The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor executes the method according to the first aspect.

[0044] In a third aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the method described in the first aspect.

[0045] In the data processing method and electronic device provided in the present application, multiple buffer pages can be bound to form a buffer page group, and the buffer page group can be used to expand the IO unit of the buffer pool. Multiple buffer pages in the buffer page group are loaded and eliminated together. When data is eliminated, the data in multiple buffer pages are compressed together and written to the disk. By compressing multiple buffer pages together, the compression rate is improved and disk space is saved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 It is a schematic diagram of disk data writing;

[0048] Figure 2 This is a schematic diagram of the principle of a buffer page group provided by an embodiment of the present application;

[0049] Figure 3 This is a flow chart of a data processing method provided in an embodiment of the present application;

[0050] Figure 4 This is a schematic diagram of data loading of a buffer page group provided by an embodiment of the present application;

[0051] Figure 5This is a schematic diagram of data elimination of a buffer page group provided by an embodiment of the present application;

[0052] Figure 6 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0053] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0054] First, the terms involved in the embodiments of this application are introduced.

[0055] Buffer pool: An area in memory used to cache database tables, indexes and other data.

[0056] Disk pages: The database is stored on disk in pages, storing data such as tables and indexes.

[0057] Buffer page: The buffer pool consists of multiple buffer pages. The data cached in each buffer page corresponds to a disk page. The size of the buffer page can be set. For example, the size of the buffer page is 8KB.

[0058] Transparent compression: Compresses data before writing it to the file system, leveraging the file hole mechanism provided by file systems such as ext4 and xfs to save disk space. Transparency means that upper-layer applications are unaware of the disk space savings; the file system handles the work, leaving upper-layer applications only to perform compression.

[0059] Table: Relational databases use the logical form of tables to store data.

[0060] Compression ratio: The ratio of the compressed file size to the uncompressed file size. For example, if a 100MB file is compressed to 90MB, the compression ratio is 90 / 100*100% = 90%. The lower the compression ratio, the higher the compression ratio.

[0061] File system block size: The minimum physical space occupied by the disk, usually 4KB. For example, the actual size of a file is 5KB, but it will occupy 8KB of disk space.

[0062] Let's first explain transparent compression in more detail. Transparent compression takes advantage of the fact that holes in mainstream file systems do not actually occupy disk space. Through compression, holes are formed at the end of the file, and the file system does not occupy disk storage for the hole part. Figure 1 As shown in the figure, without compression, writing 8KB of data from memory to the file system (the file system is responsible for persistence) will occupy 8KB of disk space. However, when the 8KB of data from memory is compressed and written to the file system, the 8KB of data is compressed to 2KB, and the remaining 6KB is empty. The empty part does not actually occupy disk space when written to the file system. However, since the file system block size is 4KB, at least 4KB is occupied at a time, so the compressed data ultimately occupies 4KB of disk space. This shows that using transparent compression can reduce disk space usage and lower storage costs.

[0063] For databases, operations such as adding, deleting, and modifying data are all performed in the buffer page. When writing data, the data is first written to the buffer page, and then, according to a certain cache eviction mechanism, the data in the buffer page is eliminated from the disk. When reading data from the disk, the compressed data is decompressed and loaded into the buffer page. When the buffer page is about to be eliminated and written to disk, combined with the aforementioned transparent compression function, the purpose of saving disk space can be easily achieved. When the page needs to be read from disk to the buffer page, the compressed data is obtained from the file system, then decompressed in memory, and the decompressed data is placed in the buffer page.

[0064] The buffer pages in the buffer pool are of uniform size, and the size of the buffer pages directly affects the compression ratio. For example, if the buffer page size is 8KB, and a compression algorithm can compress data to 25% of its original size, then the 8KB buffer page will be compressed to 2KB. However, since the file system block size is 4KB, the disk space occupied will still be 4KB, and the actual compression ratio is 50%. If the buffer page size is 16KB, and the same compression algorithm is used, then the 16KB buffer page will be compressed to 4KB. Given a 4KB file system block size, the disk space occupied will be 4KB, and the compression ratio is 25%. This is double the compression ratio when the buffer page size is 8KB.

[0065] As you can see, the larger the buffer page, the higher the compression ratio. To achieve a high compression ratio, you can set the buffer page size to a larger value. However, it is worth noting that the buffer page size also affects database performance. Larger buffer pages increase the input / output (IO) amplification caused by reading or modifying data, which can affect database performance. Furthermore, larger buffer pages affect more data when the entire page is locked. With a uniform buffer page size, it's impossible to achieve both high compression ratios for some tables and high performance for others.

[0066] To address this issue, related technologies have proposed a multi-buffer pool solution. During the operation of a database instance, multiple buffer pool instances are used. The buffer page size in each buffer pool is uniform, but different buffer pools have different buffer page sizes. When a page is needed, the buffer pool to which it belongs is determined based on the page information, and the corresponding buffer pool is searched and processed. However, this solution involves managing multiple buffer pools, increasing management overhead and processing complexity.

[0067] In view of this, an embodiment of the present application proposes a data processing method based on a single buffer pool. For tables that require a high compression rate, multiple buffer pages can be bound to form a buffer page group (group). The logical concept of the buffer page group is used to expand the IO unit of the buffer pool, providing different multiples of the IO capacity of a single buffer page, which is equivalent to increasing the size of the buffer page by multiple times. The multiple buffer pages in the buffer page group are loaded and eliminated together. When eliminating data, the data in the multiple buffer pages are transparently compressed together and written to the disk. The compression rate is improved by compressing the multiple buffer pages together. When loading data, the compressed data on the disk is decompressed and then written to the multiple buffer pages in the buffer page group.

[0068] like Figure 2As shown in the figure, backend 1 and backend 2 are processes that execute Structured Query Language (SQL) (corresponding to user SQL requests), and the buffer table is a hash table. The buffer table stores the correspondence between page identifiers (page ids) and buffer page positions, and is used to quickly find the position of the buffer page corresponding to the page in the buffer pool according to the page identifier. For example, s0, s1, and s5 in the figure represent slots in the bucket, which store page identifiers and buffer page positions. TagC1, TagC2, TagN1, and TagN3 are page identifiers, representing pages C1, C2, N1, and N3, respectively. ID=0, ID=3, ID=5, and ID=7 are the buffer page positions corresponding to pages C1, C2, N1, and N3, respectively. The buffer table can be divided into multiple areas. Figure 2 The dotted box is used to illustrate the page identifier and buffer page location. The page identifier and buffer page location are stored in the slot of the bucket in the corresponding area. When an exclusive lock is added to the buffer page in the buffer table, it is performed on a per-area basis. The exclusive lock is the cache mapping lock (BufMappingLock). The buffer page array is a specific implementation of the buffer pool. The free buffer pages are linked through the freelist. Multiple buffer pages of the buffer page group, for example Figure 2 The two buffer pages corresponding to pages C1 and C2 are linked by a doubly linked list. When the buffer page is read / written to disk, it may be compressed and decompressed depending on whether the table is compressed. The disk may contain compressed tables and ordinary tables. Ordinary tables are tables that are stored on disk without compression, for example Figure 2 The buffer page group corresponding to pages C1 and C2 is written to the disk after transparent compression, and the buffer page corresponding to page N1 is not compressed but written directly to the disk.

[0069] In an embodiment of the present application, the cache read and write strategy of each table is defined based on the table dimension. When creating a table, the user can specify how many pages the table requires as IO units. Specifying two or more pages as IO units is a multi-page read and write table, that is, multiple adjacent pages on the disk are grouped together. The group consisting of multiple pages can be called a page group, and the pages in the group are called brother pages. Multiple brother pages are loaded from the disk into the corresponding multiple buffer pages in the buffer pool together, and are eliminated from the multiple buffer pages in the buffer pool to the disk together. The multiple buffer pages corresponding to the page group on the disk are the buffer page group. Taking two pages as an example, the two pages are eliminated from the two buffer pages in the buffer pool to the disk together, and loaded from the disk to the corresponding two buffer pages in the buffer pool together. There is no situation where one page is in the buffer pool and the other page is not in the buffer pool. Single-page read and write tables still use one page as the IO unit for data loading or elimination. By binding multiple buffer pages, a high compression rate is achieved, while having no performance impact on tables that require high performance, achieving a balance between high compression rate and high performance.

[0070] The buffer pool uses a single buffer page as the basic management unit. Buffer page locking, lookup, eviction, and loading operations are performed on a single buffer page basis. However, when using a multi-page read-write table, a buffer page is linked to sibling pages within the same buffer page group through a doubly linked list. Loading or eviction of multiple buffer pages within a buffer page group is bundled. For example, a multi-page read-write table uses two pages as the I / O unit. In the file system, two consecutive pages form a page group, which are read and decompressed together and then decompressed into two corresponding buffer pages in the buffer pool. Buffer page allocation is performed using the same algorithm as for a single buffer page, requiring two requests, resulting in two buffer pages. Therefore, the two buffer pages in the buffer pool are not necessarily contiguous in memory. A doubly linked list is used to quickly locate sibling buffer pages. Locking and releasing buffer pages during data loading and eviction are also performed at the granularity of a single buffer page. A buffer page group is considered locked only after all buffer pages within it are locked, and is considered released only after all buffer pages within it are released. When data is loaded, the first page of the plurality of pages is used as a reference. If it is determined that the first page has been loaded completely, it is determined that all pages of the entire page group have been loaded completely.

[0071] The data processing method provided in the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0072] Figure 3 This is a flow chart of a data processing method provided by an embodiment of the present application. Figure 3 As shown, the method includes:

[0073] S301 : In response to a first database operation instruction, determine a first buffer page group in a buffer pool of a database, where the first buffer page group includes a plurality of associated buffer pages.

[0074] S302: Compress the data in the plurality of buffer pages of the first buffer page group and write the data to the disk.

[0075] The first database operation instruction can be a cache elimination instruction of the database background, or the first database operation instruction can be a user SQL request. Assuming that the user SQL request is for any operation such as addition, deletion, and modification, the page to be operated needs to be read into the buffer pool. If there is no free buffer page in the buffer pool, cache elimination is required. In the case where cache elimination is required, a buffer page is determined as an eliminated page in the buffer pool based on the elimination algorithm. If the eliminated page has an associated buffer page, that is, the eliminated page belongs to the first buffer page group, then the associated multiple buffer pages, that is, the first buffer page group, need to be eliminated. For example, the first buffer page group includes a first buffer page and at least one second buffer page. In response to the first database operation instruction, the first buffer page is determined in the database buffer pool; based on the association between the first buffer page and the second buffer page, the second buffer page is determined in the database buffer pool. Afterwards, the data in the multiple buffer pages of the first buffer page group is transparently compressed and written to the disk, thereby achieving a high compression rate.

[0076] In the above embodiment, when data is eliminated from a buffer page to a disk, multiple associated buffer pages are compressed together and then written to the disk. When data is loaded from the disk to the buffer page, a third buffer page group is determined in the buffer pool in response to a second database operation instruction. The third buffer page group includes multiple associated buffer pages. The data of the page corresponding to the second database operation instruction on the disk is decompressed and loaded into multiple buffer pages of the third buffer page group.

[0077] The second database operation instruction can be a data read instruction from the database backend, or the second database operation instruction can be a user SQL request. The page corresponding to the operation of the second database operation instruction is a page of a multi-page read table. Therefore, the buffer page determined in response to the second database operation instruction is a plurality of associated buffer pages included in the third buffer page group. For example, the third buffer page group includes a third buffer page corresponding to the second database operation instruction and at least one fourth buffer page associated with the third buffer page. In response to the second database operation instruction, the third buffer page is determined in the buffer pool; based on the association between the third buffer page and the fourth buffer page, the fourth buffer page is determined in the buffer pool. After determining the plurality of buffer pages, the data of the page corresponding to the second database operation instruction is decompressed and loaded into the plurality of buffer pages of the third buffer page group, thereby achieving binding of the plurality of buffer pages in the buffer page group during data loading.

[0078] The process of loading or eliminating data from a buffer page involves locking and releasing the buffer page. The following describes the data loading and elimination processes.

[0079] During the use of the database, multiple processes may access different brother pages of the same page group. Before loading the page from the disk to the buffer page, the buffer page needs to be allocated first. Here, the buffer page needs to be locked to prevent other processes from reading dirty data. In the embodiment of the present application, the locking order of the pages in the page group is controlled, and the pages are locked in the order of increasing pages to avoid deadlock. For example, for the third buffer page group in the aforementioned embodiment, before loading data into the multiple buffer pages of the third buffer page group, IO locks (Buffer IO lock) are added to the multiple buffer pages of the third buffer page group in sequence according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group.

[0080] Optionally, taking the example that the third buffer page group includes a third buffer page corresponding to the second database operation instruction and at least one fourth buffer page associated with the third buffer page, after the third buffer page is determined in the buffer pool, an IO lock is added to the third buffer page; if the page corresponding to the third buffer page is the first of the pages corresponding to multiple buffer pages of the third buffer page group, then after the fourth buffer page is determined in the buffer pool, the IO lock is continued to be added to the fourth buffer page in the order of the pages corresponding to the fourth buffer page; if the page corresponding to the third buffer page is not the first of the pages corresponding to multiple buffer pages of the third buffer page group, the IO lock of the third buffer page is released, and after the fourth buffer page is determined in the buffer pool, IO locks are added to the multiple buffer pages of the third buffer page group in sequence according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group.

[0081] Combine Figure 4 For explanation. Figure 4As shown, assume that page A and page B are two pages belonging to a page group, page A is the previous page, page B is the next page, and the two buffer pages corresponding to page A and page B constitute the third buffer page group. Assume that background 1 needs to read page A, and background 2 needs to read page B. Page A and page B do not have corresponding buffer pages in the buffer pool, so pages A and page B need to be loaded from disk into the buffer pages. Assume that according to the order of events, background 1 first wants to read page B, and finds that page B is not in the buffer pool, so it allocates the corresponding buffer page for page B and adds an IO lock to the buffer page. The IO lock is an exclusive lock. Due to the association between page B and page A, background 1 also needs to load page A into the buffer pool, so it needs to allocate a buffer page for page A. However, at this moment, backend 2 attempts to read page A and discovers that page A is not in the buffer pool. Backend 2 has preemptively allocated a corresponding buffer page for page A and added an IO lock to that buffer page. At the same time, due to the relationship between page A and page B, backend 2 also needs to load page B into the buffer pool, so it needs to add an IO lock to the buffer page corresponding to page B. At this point, backend 1 holds the IO lock for the buffer page corresponding to page B and applies for an IO lock for the buffer page corresponding to page A, while backend 2 holds the IO lock for the buffer page corresponding to page A and applies for an IO lock for the buffer page corresponding to page B, resulting in a deadlock between backends 1 and 2.

[0082] In order to avoid the occurrence of the above-mentioned deadlock situation, in an embodiment of the present application, when it is necessary to add an IO lock to the buffer page in the buffer page group, the IO lock is always added in the ascending order of the pages corresponding to the buffer page. In the above example, both background 1 and background 2 first add an IO lock to the buffer page corresponding to page A, and then add an IO lock to the buffer page corresponding to page B. For background 1, it first allocates the corresponding buffer page for page B and adds an IO lock to the buffer page. When applying to add an IO lock to the buffer page corresponding to page A, it will first release the IO lock of the buffer page corresponding to page B that it has already held, so as to lock it in the order of first page A and then page B. In this way, when background 1 releases the IO lock of the buffer page corresponding to page B, background 2 has the opportunity to add an IO lock to the buffer page corresponding to page B. In this way, background 2 can complete the IO lock of page A and page B, so that page A and page B can be loaded from the disk into the buffer page. After background 2 completes the loading and releases the IO lock, background 1 finds that page A and page B have been loaded and can directly read from the buffer page. This ensures that background 1 and background 2 will not deadlock on the IO path.

[0083] When loading data, in addition to locking and marking the status before reading the data from the disk to the buffer page, the lock must be released and the status must be marked after the data is read from the disk to the buffer page to complete the loading. Unlike adding IO locks, when releasing IO locks, the IO locks of the multiple buffer pages of the third buffer page group are released in reverse order of the pages corresponding to the multiple buffer pages of the third buffer page group. That is, the IO locks are released in the order of page B first and then page A, thereby ensuring atomic loading in the page group, that is, all pages in the same page group are loaded into the buffer pool together, ensuring that the locks and status marks of the pages in the same page group are legal.

[0084] Still refer to Figure 4 To illustrate. Assume that both backend 1 and backend 2 need to load page A and page B. Backend 1 first obtains the IO locks of the two buffer pages corresponding to page A and page B. Since the IO locks are added in the order of page A first and then page B, backend 2 is waiting for the IO lock of the buffer page corresponding to page A. After reading the file system, backend 1 puts the data of page A and page B from the disk into the corresponding buffer pages. If backend 1 releases the IO locks on the corresponding buffer pages and marks the corresponding status in the order of page A first and then page B, then after releasing the IO locks on the buffer pages corresponding to page A and marking the corresponding status, backend 2 wakes up from waiting. When it is determined that page A has been loaded, backend 2 will think that the entire page group has been loaded, and then return the buffer page address of page B to the upper layer. However, at this time, backend 1 may still be releasing the IO lock on the buffer page corresponding to page B and marking the status, which means that page B has not actually been loaded, resulting in an error.

[0085] To avoid the above situation, in an embodiment of the present application, the IO lock is released and the status is marked in the order of page B first and then page A. After the background 1 puts the data of page A and page B from the disk into the corresponding buffer pages, it first releases the IO lock on the buffer page corresponding to page B and marks the corresponding status. Then, it releases the IO lock on the buffer page corresponding to page A and marks the corresponding status. After the IO lock on the buffer page corresponding to page A is released, the background 2 wakes up from waiting. At this time, page B and page A have been completely loaded, and the background 2 returns the buffer page address of page B to the upper layer without causing an error.

[0086] In addition to data loading, data elimination also involves locking and releasing locks. Figure 3In the illustrated embodiment, taking the case of eliminating data from multiple buffer pages of a first buffer page group to disk as an example, before the data in the multiple buffer pages of the first buffer page group are compressed and written to disk, IO locks are sequentially added to the multiple buffer pages in the order of the pages corresponding to the multiple buffer pages of the first buffer page group; after the data in the multiple buffer pages of the first buffer page group are compressed and written to disk, the IO locks of the multiple buffer pages are sequentially released in the reverse order of the pages corresponding to the multiple buffer pages of the first buffer page group. The implementation principles of adding IO locks in the order of the pages corresponding to the multiple buffer pages and releasing IO locks in the reverse order of the pages corresponding to the multiple buffer pages are similar to those in the aforementioned embodiment and will not be repeated here.

[0087] After compressing the data in the multiple buffer pages of the first buffer page group and writing them to disk, and releasing the IO locks of the multiple buffer pages, an exclusive lock is added to the buffer page in the buffer table of the buffer pool in the order of the pages corresponding to the multiple buffer pages of the first buffer page group. If the exclusive lock is added successfully, a spin lock is added to the buffer description information of the buffer page. If the spin lock is added successfully, it is determined whether the buffer page is a dirty page. If none of the multiple buffer pages of the first buffer page group are dirty pages, the buffer description information of the multiple buffer pages of the first buffer page group is deleted, the spin locks of the multiple buffer pages of the first buffer page group are released, the records of the multiple buffer pages of the first buffer page group are deleted from the buffer table, and the exclusive locks of the multiple buffer pages of the first buffer page group are released. If the exclusive lock or spin lock fails to be added to any buffer page of the first buffer page group, a second buffer page group is determined in the database buffer pool, the second buffer page group including multiple buffer pages; the data in the multiple buffer pages of the second buffer page group is compressed and written to disk.

[0088] Combine Figure 5 For explanation. Assume that background 1 needs to read page A, but there is no buffer page corresponding to page A in the buffer pool, so background 1 needs to load page A from the disk into the buffer page. Assume that Figure 5As shown in , if all the buffer pages in the buffer pool are occupied, the data of the buffer pages needs to be eliminated from the buffer pool to the disk in order to allocate buffer pages for page A. Assuming that the buffer page corresponding to page C is determined to be the victim page based on the elimination algorithm, the buffer page corresponding to page C needs to be eliminated to the disk. Since page C and page D are sibling pages of the same page group, page C is the previous page and page D is the next page. The two buffer pages corresponding to page C and page D are the first buffer page group in the aforementioned embodiment. The two buffer pages corresponding to page C are the third buffer page and the buffer page corresponding to page D is the fourth buffer page. In addition to the buffer page corresponding to page C, the buffer page corresponding to page D also needs to be eliminated to the disk. The content lock and IO lock of the two buffer pages corresponding to page C and page D are added in the order of page C first and page D later. The data in the two buffer pages corresponding to page C and page D are compressed and written to the disk. The content lock and IO lock of the two buffer pages corresponding to page C and page D are released in the order of page D first and page C later. In the buffer table, an exclusive lock is added to the area where the buffer page corresponding to page A and the buffer page corresponding to page C are located, that is, a cache mapping lock. The area is indicated by a dotted box in the figure. The page identifier of the buffer page corresponding to page A and the record of the location of the buffer page are inserted into the buffer table. After that, a spin lock (BufferDesc Spinlock) of the buffer description information is added to the buffer page corresponding to page C. Based on the buffer description information, it is determined whether the buffer page corresponding to page C is a dirty page, that is, whether it is redirty. If the buffer page corresponding to page C is a dirty page, it is necessary to reselect the buffer page as the eliminated page based on the elimination algorithm. If the buffer page corresponding to page C is not a dirty page, then continue to tentatively add an exclusive lock to the buffer page corresponding to page D in the buffer table. If adding the exclusive lock fails, then it is necessary to reselect the buffer page as the eliminated page based on the elimination algorithm. If adding the exclusive lock succeeds, then continue to tentatively add a spin lock of the buffer description information to the buffer page corresponding to page D. If adding the spin lock fails, it is necessary to reselect the buffer page as the eliminated page based on the elimination algorithm. The elimination algorithm reselects the buffer page as the elimination page. If the spin lock is added successfully, it determines whether the buffer page corresponding to page D is a dirty page based on the buffer description information, that is, whether it is re-dirty. If the buffer page corresponding to page D is a dirty page, it is necessary to reselect the buffer page as the elimination page based on the elimination algorithm. If the buffer page corresponding to page D is not a dirty page, the buffer description information of the two buffer pages corresponding to page C and page D is deleted, the spin locks of the two buffer pages corresponding to page C and page D are released, the records of the two buffer pages corresponding to page C and page D are deleted from the buffer table, and the exclusive locks of the areas where the two buffer pages corresponding to page C and page D are located are released. In this embodiment, when the buffer page is reselected as the elimination page based on the elimination algorithm, if the elimination page belongs to the second buffer page group, it is necessary to eliminate multiple buffer pages of the second buffer page group to the disk. The process is similar to that of this embodiment.

[0089] This embodiment achieves atomic eviction of a buffer page group, meaning that multiple buffer pages belonging to the same buffer page group are collectively evicted from the buffer pool, including writing the data in the buffer pages to disk and clearing the buffer description information and related records in the buffer table. Furthermore, when multiple pages are evicted, it is necessary to simultaneously hold locks on multiple pages, potentially leading to lock waits. For performance reasons, the embodiment of this application employs a tentative locking strategy—a strategy of abandoning the lock if it fails to acquire the lock. This avoids deadlocks or prolonged lock holding, and prevents other processes from accessing the pages.

[0090] It should also be noted that, in the embodiments of the present application, for tables requiring high compression ratios, a buffer page group approach can be used to improve the compression ratio. For tables requiring high performance, a single buffer page processing strategy is still adopted. Optionally, in response to a third database operation instruction, a fifth buffer page is determined in the buffer pool; the data in the fifth buffer page is compressed and written to disk. Optionally, in response to a fourth database operation instruction, a sixth buffer page is determined in the buffer pool; the data in the page corresponding to the fourth database operation instruction on disk is decompressed and loaded into the sixth buffer page. Thus, using a single buffer pool, the compatibility of high compression ratio and high performance is achieved.

[0091] Figure 6 Schematic block diagram of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 600 may include at least one processor 601 for implementing the data processing method provided in the embodiment of the present application.

[0092] Optionally, the electronic device 600 further includes at least one memory 602 for storing program instructions and / or data. The memory 602 is coupled to the processor 601. Coupling in the embodiments of the present application is an indirect coupling or communication connection between devices, units, or modules, which can be electrical, mechanical, or other forms, and is used for information exchange between devices, units, or modules. The processor 601 may operate in conjunction with the memory 602. The processor 601 may execute program instructions stored in the memory 602. At least one of the at least one memory may be included in the processor.

[0093] Optionally, the electronic device 600 further includes a communication interface 603 for communicating with other devices via a transmission medium, thereby enabling the electronic device 600 to communicate with other devices. The communication interface 603 may be, for example, a transceiver, an interface, a bus, a circuit, or a device capable of implementing transceiver functions. The processor 601 may utilize the communication interface 603 to transmit and receive data and / or information, and to implement the methods provided in the embodiments of the present application. For details, please refer to the detailed description in the preceding embodiments, which will not be repeated here.

[0094] The specific connection medium between the processor 601, the memory 602 and the communication interface 603 is not limited in the embodiment of the present application. Figure 6 The processor 601, the memory 602 and the communication interface 603 are connected via a bus 604. The bus 604 is connected to the Figure 6 The connections between the other components are shown in bold lines, which are only for illustration and not intended to be limiting. The bus can be divided into address bus, data bus, control bus, etc. Figure 6 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0095] It should be understood that the processor in the embodiments of the present application can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above-mentioned method embodiment can be completed by hardware integrated logic circuits in the processor or by software instructions. The above-mentioned processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above-mentioned method.

[0096] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0097] The present application also provides a computer-readable storage medium storing a computer program (also referred to as code or instruction). When the computer program is executed, the computer executes the method in any of the aforementioned embodiments.

[0098] The terms "unit," "module," and the like used in this specification may be used to refer to a computer-related entity, hardware, firmware, a combination of hardware and software, software, or software in execution.

[0099] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. In the several embodiments provided in this application, it should be understood that the disclosed devices, equipment, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not performed. In addition, the coupling or direct coupling or communication connection shown or discussed can be through some interface, indirect coupling or communication connection of devices or units, and can be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of the solution of this embodiment according to actual needs.

[0101] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] In the above-described embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (program) are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state disk (SSD)).

[0103] If this function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

[0104] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: include: In response to a first database operation instruction, determining a first buffer page group in a buffer pool of the database, the first buffer page group including a plurality of associated buffer pages; adding input and output (IO) locks to the plurality of buffer pages in sequence according to the order of the pages corresponding to the plurality of buffer pages of the first buffer page group; compressing the data in the plurality of buffer pages of the first buffer page group and writing the data to the disk; releasing the IO locks of the plurality of buffer pages of the first buffer page group in sequence according to the reverse order of the pages corresponding to the plurality of buffer pages; Also includes: According to the order of pages corresponding to the multiple buffer pages of the first buffer page group, for each buffer page, an exclusive lock is added to the buffer page in the buffer table of the buffer pool. If the exclusive lock is added successfully, a spin lock of the buffer description information is added to the buffer page. If the spin lock is added successfully, it is determined whether the buffer page is a dirty page. If none of the multiple buffer pages of the first buffer page group are dirty pages, the buffer description information of the multiple buffer pages of the first buffer page group is deleted, the spin locks of the multiple buffer pages of the first buffer page group are released, the records of the multiple buffer pages of the first buffer page group are deleted from the buffer table, and the exclusive locks of the multiple buffer pages of the first buffer page group are released.

2. The method according to claim 1, characterized in that Also includes: If adding an exclusive lock or a spin lock to any buffer page of the first buffer page group fails, determining a second buffer page group in the database buffer pool, where the second buffer page group includes multiple buffer pages; The data in the plurality of buffer pages of the second buffer page group is compressed and then written to the disk.

3. The method according to claim 1 or 2, characterized in that The first buffer page group includes a first buffer page and at least one second buffer page, and determining the first buffer page group in the database buffer pool includes: Determining the first buffer page in the database buffer pool; Based on the association relationship between the first buffer page and the second buffer page, the second buffer page is determined in the database buffer pool.

4. The method according to claim 1, wherein Also includes: In response to a second database operation instruction, determining a third buffer page group in the buffer pool, the third buffer page group including a plurality of associated buffer pages; The data of the page corresponding to the second database operation instruction in the disk is decompressed and loaded into the multiple buffer pages of the third buffer page group.

5. The method according to claim 4, characterized in that The third buffer page group includes a third buffer page corresponding to the second database operation instruction and at least one fourth buffer page associated with the third buffer page; Determining a third buffer page group in the buffer pool includes: determining the third buffer page in the buffer pool; Based on the association relationship between the third buffer page and the fourth buffer page, the fourth buffer page is determined in the buffer pool.

6. The method according to claim 5, characterized in that Also includes: IO locks are sequentially added to the multiple buffer pages of the third buffer page group according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group.

7. The method according to claim 6, characterized in that The adding IO locks to the multiple buffer pages of the third buffer page group in sequence according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group includes: After determining the third buffer page in the buffer pool, adding an IO lock to the third buffer page; If the page corresponding to the third buffer page is the first of the pages corresponding to the multiple buffer pages of the third buffer page group, after determining the fourth buffer page in the buffer pool, continue to add an IO lock to the fourth buffer page in the order of the pages corresponding to the fourth buffer page; If the page corresponding to the third buffer page is not the first of the pages corresponding to the multiple buffer pages of the third buffer page group, the IO lock of the third buffer page is released, and after the fourth buffer page is determined in the buffer pool, IO locks are added to the multiple buffer pages of the third buffer page group in sequence according to the order of the pages corresponding to the multiple buffer pages of the third buffer page group.

8. The method according to claim 6, characterized in that After decompressing the data of the page corresponding to the second database operation instruction on the disk and loading the data into the plurality of buffer pages of the third buffer page group, the method further includes: The IO locks of the multiple buffer pages of the third buffer page group are released in sequence according to the reverse order of the pages to which the multiple buffer pages of the third buffer page group correspond.

9. The method according to any one of claims 4 to 8, characterized in that Also includes: In response to a third database operation instruction, determining a fifth buffer page in the buffer pool; The data in the fifth buffer page is compressed and then written to the disk.

10. The method according to claim 9, characterized in that Also includes: In response to a fourth database operation instruction, determining a sixth buffer page in the buffer pool; The data of the page corresponding to the fourth database operation instruction in the disk is decompressed and loaded into the sixth buffer page.

11. An electronic device, characterized in that: include: memory and processor; The memory is used to store computer programs; The processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, the processor executes the method according to any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 10.

Images