Method and device for quickly dropping write-intensive load data into disk

Abstract

The invention relates to the technical field of databases, and provides a method and a device for quickly dropping write-intensive load data into a disk. The method comprises the following steps: acquiring an actual storage position of dirty page data, and determining a processing sequence of the dirty page data; according to the processing sequence, writing the dirty page data into a virtual storage area from a data buffer area, determining a virtual storage position of the dirty page data, and associating the virtual storage position with the actual storage position; a calculation unit of the virtual storage area obtains the actual storage position, and writes the dirty page data into the actual storage position from the virtual storage position; wherein a storage medium of the virtual storage area is an intelligent solid state disk; the problems that in the prior art, due to the fact that random write operation cannot be avoided, and page splitting or merging operation of a B + tree cannot be avoided, write blocking is serious, and the performance of a database system is sharply reduced are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of databases, and particularly to a method and apparatus for quickly flushing write-intensive load data to disk. Background Art

[0002] Currently, many database systems use B+ trees as storage engines. When using a B+ tree as a storage engine, the data to be stored is sequentially stored in the leaf nodes of the B+ tree, and the non-leaf nodes of the B+ tree serve as indexes for the leaf nodes; therefore, using a B+ tree as a storage engine enables the database system to support efficient range queries and single-point queries, and has better performance in read-intensive scenarios. Among them, the read-intensive scenario refers to a scenario where the frequency of read operations (such as query operations) in the database system is much higher than that of write operations (such as insert operations, delete operations, and update operations).

[0003] When using a B+ tree as a storage engine, the leaf nodes of the B+ tree are used to store the dirty page data in the data buffer in memory; among them, the dirty page data refers to the data in the data page that has been modified in memory but has not been written to disk. Since when writing the dirty page data into a certain leaf node in the data buffer, the newly inserted dirty page data in the B+ tree is often allocated to different leaf nodes, resulting in the actual storage positions of the dirty page data on the disk being discontinuous and discrete.

[0004] To maintain balance, when an insert operation or a delete operation causes a leaf node to be full or empty, the B+ tree needs to perform page splitting or merging operations. Among them, the balance of the B+ tree means that all leaf nodes in the B+ tree are at the same level, that is, the path lengths from the root node to any leaf node are the same, to ensure that the time complexities of search operations, insert operations, and delete operations are all O(logn). However, page splitting and merging operations not only increase the computational overhead of the CPU but also may cause additional disk I / O operations. For example, page splitting may require reallocating a leaf node for storing the dirty page data in a certain leaf node and correspondingly changing the actual storage position of the dirty page data, which further increases the complexity and overhead of write operations.

[0005] When it is necessary to write the dirty page data in the data buffer to the disk (or other persistent storage media), since the actual storage locations of the dirty page data are not arranged sequentially, the process of writing to the disk is a random write operation. Since a large number of random write operations need to be processed and the balance of the B+ tree needs to be maintained simultaneously, the database system needs to frequently access the data buffer and perform complex calculations, resulting in an increase in memory usage and a heavier computational burden on the Central Processing Unit (CPU); especially in a high-concurrency write-intensive scenario, due to the inability to avoid random write operations in the prior art, write blocking is severe, causing the performance of the database system to drop sharply and seriously affecting the user experience.

[0006] In view of this, overcoming the defects of the prior art is an urgent problem to be solved in the technical field. Summary of the Invention

[0007] The technical problem to be solved by the present invention is to provide a method and device for quickly flushing write-intensive load data to disk. The purpose is to sequentially write the dirty page data from the data buffer to the virtual storage area, and then the computing unit writes it to the actual storage location from the virtual storage area. Since the dirty page data is sequentially written to the virtual storage area, the write throughput of the database system can be improved and write blocking can be alleviated; the computing unit writes the dirty page data to the actual storage location, avoiding the intervention of memory and CPU, controlling the overhead of memory and CPU to a level similar to sequential writing, reducing random write operations, and making the overall performance of the write-intensive scenario have less impact on the database system, solving the problem that in the prior art, due to the inability to avoid random write operations, write blocking is severe and the performance of the database system drops sharply.

[0008] The present invention adopts the following technical solutions:

[0009] In a first aspect, the present invention provides a method for quickly flushing write-intensive load data to disk, including:

[0010] Obtain the actual storage location of the dirty page data and determine the processing order of the dirty page data;

[0011] According to the processing order, write the dirty page data from the data buffer to the virtual storage area, determine the virtual storage location of the dirty page data, and associate the virtual storage location with the actual storage location;

[0012] The computing unit in the virtual storage area obtains the actual storage location and writes the dirty page data from the virtual storage location to the actual storage location; wherein, the storage medium of the virtual storage area is an intelligent solid-state drive.

[0013] Further, according to the processing sequence, writing the dirty page data from the data buffer into the virtual storage area, determining the virtual storage location of the dirty page data, and associating the virtual storage location with the actual storage location includes:

[0014] Initialize a mapping relation table in the memory in advance;

[0015] Obtain the dirty page data from the data buffer according to the processing sequence, write the dirty page data into the virtual storage area, and obtain the virtual storage location of the dirty page data in the virtual storage area;

[0016] Establish a location mapping relation between the actual storage location and the virtual storage location, and update the location mapping relation to the mapping relation table, so that when the virtual storage location of the dirty page data changes, the location mapping relation is updated in the mapping relation table.

[0017] Further, writing the dirty page data from the virtual storage area into the actual storage location includes:

[0018] When the storage medium of the virtual storage area is the same as that of the actual storage area, write the dirty page data from the virtual storage location into the actual storage location only when the load of the central processing unit of the database system is lower than the load threshold;

[0019] When the storage medium of the virtual storage area is an intelligent solid-state drive, use the computing node of the intelligent solid-state drive to write the dirty page data from the virtual storage location into the actual storage location and update the location mapping relation.

[0020] Further, before obtaining the actual storage location of the dirty page data and determining the processing sequence of the dirty page data, it further includes:

[0021] When a checkpoint operation is triggered, determine whether the database system enters a write-intensive load state;

[0022] When the database system enters a write-intensive load state, perform feature extraction on the amount of dirty page data in the virtual storage area and the system performance information of the database system to determine the expansion and contraction amount of the virtual storage area;

[0023] Update the capacity of the virtual storage area according to the expansion and contraction amount of the virtual storage area, so as to write the dirty page data into the updated virtual storage area.

[0024] Further, when the database system enters a write-intensive load state, performing feature extraction on the amount of dirty page data in the virtual storage area and the system performance information of the database system to determine the expansion and contraction amount of the virtual storage area includes:

[0025] Collect feature information for multiple time periods; determine the virtual storage area capacity under different feature information states; use the virtual storage area capacity as the label for the corresponding feature information to obtain a labeled training set;

[0026] Construct an original scaling model; use the original scaling model to obtain a predicted scaling amount; compare the actual scaling amount of the virtual storage area capacity with the predicted scaling amount to obtain a regression error value;

[0027] From the feature information, obtain system performance information for different time periods; calculate a system penalty value based on the difference between the system performance parameters in the system performance information and the corresponding thresholds; calculate a write operation penalty value based on the difference between the write operation parameters in the system performance information and the corresponding thresholds; calculate a resource usage penalty value based on the difference between the resource usage parameters in the system performance information and the corresponding thresholds; determine the sum of the system penalty value, the write operation penalty value, and the resource usage penalty value as the original penalty value;

[0028] Determine the product of the original penalty value and the first weight coefficient as the target penalty value, and determine the sum of the regression error value and the target penalty value as the loss function value; use the loss function value to iteratively optimize the parameters of the original scaling model to train the original scaling model to obtain a target scaling model;

[0029] When the database system enters a write-intensive load state, collect the amount of dirty page data in the virtual storage area during the current time period and the system performance information of the database system to obtain the feature information of the current time period; perform feature extraction based on the target scaling model to obtain the virtual storage area scaling amount.

[0030] Further, the determining the virtual storage area capacity under different feature information states includes:

[0031] Determine the occupancy rate stage coefficient of the central processing unit of the database system; wherein, the occupancy rate stage coefficient is used to represent that the occupancy rate of the central processing unit is in a light state, a medium state, and a heavy state;

[0032] Determine the first write throughput from the virtual storage area to the actual storage area; determine the second write throughput from the data buffer to the virtual storage area; determine the ratio of the first write throughput to the second write throughput as the write throughput coefficient;

[0033] Determine the total number of data that needs to be written from the data buffer to the actual storage area within the time interval from the first preset time before triggering the checkpoint operation to the first preset time after triggering the checkpoint operation as the data volume to be flushed to disk;

[0034] Determine the product of the occupancy rate stage coefficient, the write throughput coefficient, and the amount of data to be written to disk as the corresponding virtual storage area capacity.

[0035] Further, the calculation expression of the system penalty value is:

[0036]

[0037] where ReLu() represents the activation function, and α1, β1, and γ1 are all penalty weight coefficients, and T r represents the average response time of the database system within the second preset time after triggering the checkpoint operation, and T rmax represents the response time threshold of the database system within the second preset time after triggering the checkpoint operation, and T d represents the average processing time of processing transactions within the second preset time after triggering the checkpoint operation, and T dmax represents the processing time threshold of processing transactions within the second preset time after triggering the checkpoint operation, D u represents the storage space utilization rate of the database system, D umax represents the threshold of the storage space utilization rate;

[0038] The calculation expression of the write operation penalty value is:

[0039]

[0040] where α2, β2, and γ2 are all penalty weight coefficients, and D w2 represents the amount of dirty page data written within the third preset time before triggering the checkpoint operation, and D w2max represents the threshold of the amount of dirty page data written, P d2 represents the amount of dirty page data generated within the third preset time before triggering the checkpoint operation, P d2max represents the threshold of the amount of dirty page data generated, I o represents the number of input / output operations per second of the database system, I omax represents the threshold of the number of input / output operations per second;

[0041] The calculation expression of the resource usage penalty value is:

[0042]

[0043] where α3, β3, and γ3 are all penalty weight coefficients, and C s represents the occupancy rate of the central processing unit, C smax represents the threshold of the occupancy rate of the central processing unit, S bIndicates: the write rate at which the database system writes to the actual storage area, S bmax Indicates: the threshold of the write rate, C h Indicates: the buffer hit rate of the database system within a second preset time before triggering the checkpoint operation, C hmax Indicates: the threshold of the buffer hit rate.

[0044] Further, constructing the original scaling model; obtaining the predicted scaling amount using the original scaling model; comparing the actual scaling amount of the virtual storage area capacity with the predicted scaling amount to obtain a regression error value, including:

[0045] Calculating the actual scaling amount of the virtual storage area capacity between each time period according to the virtual storage area capacity;

[0046] Training the original scaling model using the labeled training set to obtain the predicted scaling amount;

[0047] Determining the difference between the actual scaling amount and the corresponding predicted scaling amount as the scaling deviation value;

[0048] Determining the sum of the squares of the scaling deviation values corresponding to each piece of the feature information as the deviation feature value;

[0049] Determining the ratio of the deviation feature value of each piece of the feature information to the total amount of the feature information of each piece of the feature information as the regression error value.

[0050] In a second aspect, the present invention further provides a device for quickly flushing write-intensive load data, including:

[0051] At least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor for executing the method for quickly flushing write-intensive load data described in the first aspect.

[0052] In a third aspect, the present invention further provides a non-volatile computer storage medium, and the computer storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors for completing the method for quickly flushing write-intensive load data described in the first aspect.

[0053] In a fourth aspect, a laser chip is provided, including: a processor and an interface, for calling and running a computer program stored in a memory from the memory, and executing the method for quickly flushing write-intensive load data as described in the first aspect.

[0054] Fifth aspect, a computer program product containing instructions is provided. When the instructions are run on a computer or a processor, the computer or the processor is caused to execute the method for quickly flushing write-intensive load data as described in the first aspect to the fourth aspect and any one of them.

[0055] Sixth aspect, the present invention further provides a system for quickly flushing write-intensive load data, including the device for quickly flushing write-intensive load data as described in the second aspect, and using the method for quickly flushing write-intensive load data as described in the first aspect to complete the interaction of the device for quickly flushing write-intensive load data in the second aspect.

[0056] Different from the prior art, the present invention has at least the following beneficial effects:

[0057] The present invention directly writes dirty page data into the virtual storage area in the processing order, greatly improving the write throughput of the database system, reducing the write blocking problem, and improving the performance of the database system. The dirty page data is directly written from the virtual storage location to the actual storage location through the computing unit, enabling the dirty page data to be automatically completed from the virtual storage area to the actual storage area without going through the memory and the CPU; only using very little CPU and memory overhead, associating the corresponding virtual storage location with the actual storage location before the dirty page data is written to the actual storage location to ensure data consistency, and then in the write-intensive scenario, controlling the CPU and memory overhead to a level similar to sequential writing, greatly reducing the memory and CPU overhead during the process of flushing dirty page data, reducing random write operations, making the impact of the write-intensive scenario on the overall performance of the database system smaller, and improving the performance of the database system based on the B+ tree storage engine in the write-intensive scenario. Description of the Drawings

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required to be used in the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and those of ordinary skill in the art can also obtain other drawings based on these drawings without creative efforts.

[0059] Figure 1 It is a schematic diagram of flushing dirty page data in a write-intensive scenario of a prior art provided by an embodiment of the present invention;

[0060] Figure 2 It is a schematic flowchart of a method for quickly flushing write-intensive load data provided by an embodiment of the present invention;

[0061] Figure 3 It is a schematic diagram of flushing dirty page data in a write-intensive scenario of an embodiment of the present invention provided by an embodiment of the present invention;

[0062] Figure 4 It is a schematic flowchart of step 20 provided by an embodiment of the present invention;

[0063] Figure 5 It is a schematic flowchart of another method for quickly flushing write-intensive load data to disk provided by an embodiment of the present invention;

[0064] Figure 6 It is a schematic flowchart of step 10 provided by an embodiment of the present invention;

[0065] Figure 7 It is a schematic flowchart of step 102 provided by an embodiment of the present invention;

[0066] Figure 8 It is a schematic overall flowchart of flushing dirty page data of a database system to disk provided by an embodiment of the present invention;

[0067] Figure 9 It is a schematic architecture diagram of a device for quickly flushing write-intensive load data to disk provided by an embodiment of the present invention. Detailed implementation manners

[0068] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0069] In order to make the objectives, technical solutions and advantages of the present invention clearer and more understandable, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0070] Unless otherwise required by the context, in the entire specification and claims, the term "comprising" is interpreted in an open and inclusive sense, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples", etc. are intended to indicate that specific features, structures, materials or characteristics related to the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representations of the above terms are not necessarily referring to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the above terms of the embodiment or example due to reasons such as the order and position of appearance, however, it is not limited that they can be carried by one embodiment or example in a combined manner.

[0071] In the description of the present invention, it should be understood that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc. is based on the orientation or positional relationship shown in the drawings. It is only for the convenience of describing the present disclosure and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation to the present disclosure.

[0072] In the description of the present invention, the terms "first" and "second" are only used for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include one or more of such features. In the description of the embodiments of the present disclosure, unless otherwise specified, the meaning of "a plurality" is two or more. In addition, for example, in the description, for the same type of nouns, the method of adding "A" and "B" at the end is used to describe them as two independent individuals. In this case, the features defined with "A" and "B" are only used for the purpose of distinguishing similar individuals and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features.

[0073] When describing some embodiments, the expressions "coupled", "coupled to" and "connected" and their derivatives may be used. For example, when describing some embodiments, the term "connected" may be used to indicate that two or more components have direct physical contact or electrical contact with each other. Another example is that when describing some embodiments, the term "coupled to" may be used to indicate that two or more components have direct physical contact or electrical contact. However, the term "connected" or "coupled" may also mean that two or more components do not have direct contact with each other, but still cooperate or interact with each other, such as "optical path coupling" and "wireless connection". The embodiments disclosed herein are not necessarily limited to the content of the present invention.

[0074] In the description of the present invention, the expression "A and / or B" (where A and B are used to formally represent specific feature contents) is involved, and the corresponding expression includes the following three combinations: only A, only B, and the combination of A and B.

[0075] As used in the present invention, "about", "substantially" or "approximately" includes the stated value and the average value within an acceptable deviation range of the specific value, where the acceptable deviation range is determined by those of ordinary skill in the art considering the measurement being discussed and the errors associated with the measurement of the specific quantity (i.e., the limitations of the measurement system).

[0076] Example 1:

[0077] Most database systems adopt B+ trees or Log-Structured Merge Trees (LSM trees for short) as storage engines. These two storage engines have diametrically opposite advantages and disadvantages in terms of read and write performance. In the storage engine based on B+ trees, data is stored sequentially in leaf nodes, so it supports efficient range queries and single-point queries, and is more suitable for read-intensive scenarios. However, in write-intensive scenarios, its write performance is poor due to frequent random write operations. In the storage engine based on LSM trees, due to its sequential write mechanism, high write performance can be achieved. However, due to the dispersion of data and the impact of Compaction operations, its read performance is poor.

[0078] Among many relational databases in the prior art, due to the high demand for read performance and the mature technology ecosystem of related technologies, the storage engine based on B+ trees is generally still adopted. However, with the development of Internet technology, the amount of data that database systems need to store has increased sharply, and the requirements of users for relational databases are also changing significantly. Relational databases not only need to provide high read performance, but also the demand for write performance is gradually increasing, especially for business scenarios with dynamic load changes. In some users' business scenarios, the frequent addition and update of data pose new challenges to relational databases based on B+ tree storage engines. Frequent write operations not only cause high write congestion, but also sharply reduce the performance of the database system, seriously affecting the user experience.

[0079] As Figure 1 shown is a specific example of an application scenario of dirty page data flushing in a database system. Among them, flushing means writing dirty page data to the disk (that is, as shown by the arrow direction in Figure 1 , synchronizing the dirty page data from the data buffer to the corresponding data block in the disk block). Specifically, the database system adopts B+ trees as the storage engine; there are a file storage area and a common memory pool in the memory, which are used to cache data pages and manage dirty pages; among them, the common memory pool is used to temporarily store data that needs to be frequently accessed, reducing disk input / output operations; in one embodiment, the common memory pool may include a database system data buffer, a dictionary buffer of the data dictionary, a log buffer, a Structured Query Language (SQL) buffer, a sorting area, a hash area, etc.

[0080] To improve query performance, a database system usually maintains a data buffer in the common memory pool to cache frequently accessed data pages and index pages. The database system loads some nodes of the B+ tree into the data buffer in memory; these cached data pages and index pages may contain the data stored in some non-leaf nodes of the B+ tree (i.e., index pages) and the data stored in leaf nodes (i.e., data pages); for example, Figure 1 and below Figure 3 Each data in the data buffer in Figure 3 can be understood as the data page stored in a leaf node of the B+ tree. After these data pages are modified in memory, they become dirty pages, and finally the corresponding dirty page data needs to be written to disk, that is, written to the actual storage location in the file storage area. It should be noted that the "write operation" in the following text refers to the operation of writing dirty page data to disk.

[0081] There has been some research work on the write optimization technology for the B+ tree storage engine in the prior art. It mainly reduces the write amplification problem of the B+ tree and reduces the number of disk input / outputs through data compression technology, so as to improve the performance of the B+ tree. At the same time, there are also some studies that reduce the splitting and merging of B+ tree leaf nodes by improving the index structure of the B+ tree, thereby improving the write performance of the database system. In addition, some studies set up a data buffer to achieve the priority and fast disk write of frequently updated data, reduce the memory pressure, and then improve the performance of the database system.

[0082] However, due to the inevitable random write operations of the B+ tree-based storage engine, and during the random write operations, the B+ tree has to split and merge leaf nodes in order to maintain its balanced characteristics, making the above methods unable to well improve the write performance of the database system during write-intensive operations. Especially in the cloud environment, the database load is more dynamic and changeable. When the load characteristic changes from read-intensive operations to write-intensive operations, the write performance will cause the performance of the database system to drop sharply.

[0083] For the operation of quickly flushing frequently updated data to disk first, there are two defects in the write-intensive load: (1) The size of the data buffer is fixed. Under different write-intensive loads (i.e., different amounts of dirty page data to be written), there is likely to be a situation where the cache is insufficient or cache resources are wasted. (2) In the prior art, there is a solution that only flushes the dirty page data with frequent updates to disk first. When using this solution, it is necessary to dynamically adjust the checkpoint scheduling strategy, which increases the scheduling complexity and will increase the overhead of the database system under high write-intensive loads, affecting the performance of the database system. For example, under different write-intensive loads, when the checkpoint scheduling strategy is to start writing the dirty page data to disk when the amount of generated dirty page data exceeds a certain threshold, due to the large difference in the amount of dirty page data to be written, it is necessary to dynamically adjust the size of this threshold through the CPU, resulting in waste of CPU scheduling resources and a decrease in the performance of the database system.

[0084] To solve the above problems, as Figure 2 shown, an embodiment of the present invention provides a method for quickly flushing write-intensive load data to disk, including:

[0085] Step 10: Obtain the actual storage location of the dirty page data and determine the processing order of the dirty page data.

[0086] Among them, the actual storage location is in the actual storage area, and the actual storage area is the file storage area.

[0087] Since the storage engine of the database system in the embodiment of the present invention is a B+ tree, after performing a database operation (such as an insert operation), the original data stored in the corresponding leaf node in the data buffer becomes dirty page data. During this process, page merging or splitting of the B+ tree may be triggered, and then the actual storage location of the dirty page data will be determined; after determining the actual storage location, it is necessary to flush the dirty page data to disk, that is, write the dirty page data to this actual storage location. In order to minimize the CPU and memory overhead caused by random writes during the process of writing the dirty page data to the actual storage location, the embodiment of the present invention first obtains the actual storage locations of the dirty page data to be processed in sequence according to the generation order of the dirty page data, and uses the acquisition order as the processing order.

[0088] In an optional embodiment, a write queue can be initialized. When the database system triggers a checkpoint operation, obtain the dirty page data that needs to be written to disk from the data buffer, add it to the write queue, and obtain the actual storage location of each dirty page.

[0089] Among them, the checkpoint operation is used for the database system to write the dirty page data in memory to disk at a specific time point and record the current database state, ensuring that the data modified by the committed transactions in memory can be written to disk in a timely manner, so that the database system can reach a data consistent state at any time; for example, when a committed transaction is rolled back, the corresponding data modifications in memory need to be cancelled, and the current database state can be restored to before the checkpoint corresponding to the committed transaction.

[0090] Step 20: Write the dirty page data from the data buffer to the virtual storage area according to the processing order, determine the virtual storage location of the dirty page data, and associate the virtual storage location with the actual storage location.

[0091] As Figure 1 shown is a schematic diagram of data flushing using a B+ tree storage engine in the prior art. When the original data in memory is modified but not yet written to disk, dirty page data is generated; the original data or dirty page data (i.e., Figure 1 "data" in the data buffer) in each leaf node of the B+ tree corresponds to different actual storage locations. The modification of the original data in memory is often triggered by the insert operation, update operation, and delete operation executed by the database system. The insert operation is used to insert new data into the B+ tree, the update operation is used to update the original data to new data, and the delete operation is used to delete the original data. When inserting new data causes the leaf node where the original data is located to be full, it triggers the B+ tree page split. When the data page corresponding to this leaf node splits, some data in this leaf node will be moved to a new leaf node, resulting in a change in the actual storage location of the data on disk; when deleting the original data causes the leaf node where the original data is located to be underloaded (i.e., the leaf node is empty), it triggers the B+ tree page merge. When the data page corresponding to this leaf node merges with an adjacent leaf node, the actual storage location of the data in this leaf node on disk will also change.

[0092] When synchronizing the dirty page data corresponding to the insert operation or delete operation to the corresponding actual storage location, since the actual storage locations of each dirty page data are likely to be scattered at various positions on the disk according to the processing order, the process of writing multiple dirty page data to the actual storage location in sequence is a random write operation. Therefore, when the database system is in a write-intensive scenario for a long time, the write throughput is low and the write blocking is serious, resulting in a decline in the performance of the database system; in addition, due to the high frequency of insert operations and delete operations, the probability of causing the leaf nodes to be full or empty is relatively large. In order to maintain its balance, the B+ tree must continuously perform page split or merge operations. During this process, the CPU and memory will frequently adjust the actual storage locations corresponding to each leaf node, thereby further increasing the memory and CPU overhead of the database system, resulting in a large amount of CPU and memory in the write-intensive scenario and a sharp decline in the database system.

[0093] As Figure 3 shown in the figure is a schematic diagram of data disk flushing using the B+ tree storage engine according to an embodiment of the present invention. In a write-intensive scenario, the dirty page data is first sequentially written into the data blocks in the virtual storage area (i.e., the dirty page data is directly flushed down to the virtual storage area). Since the dirty page data is not directly synchronized to its corresponding actual storage location, but multiple dirty page data are sequentially written into a continuous area in the virtual storage area according to the processing order, the random write process is avoided, greatly reducing the memory and CPU overhead.

[0094] After obtaining the virtual storage location of the dirty page data, in order to ensure that the dirty page data can be found in the virtual cache area before the dirty page data is disk-flushed, the embodiment of the present invention associates the virtual storage location with the actual storage location of the dirty page data. With less memory and CPU overhead, it can be ensured that the dirty page data can be quickly and accurately queried during the disk flushing process.

[0095] Step 30: The computing unit of the virtual storage area obtains the actual storage location and writes the dirty page data from the virtual storage location to the actual storage location; wherein, the storage medium of the virtual storage area is an intelligent solid-state drive.

[0096] Since the database system according to the embodiment of the present invention uses a B+ tree as the storage engine, the actual storage locations corresponding to each leaf node are not sequential in the disk, that is, according to the processing order, the actual storage locations of each dirty page data are scattered in the disk. Therefore, random write operations cannot be avoided during the final disk flushing. The embodiment of the present invention uses the computing unit of an intelligent solid-state drive (Smart Solid State Drive, abbreviated as SmartSSD) to complete the disk flushing process, and controls the dirty page data to be randomly written from the virtual storage area to the actual storage location through the intelligent solid-state drive. Since the intelligent solid-state drive integrates data processing functions, the implementation program for writing the dirty page data from the virtual storage location to the actual storage location is deployed to the computing node of the intelligent solid-state drive, and this process is completed by the corresponding computing node, avoiding the intervention of memory and CPU in this process and reducing the scheduling complexity of the system. At the same time, when the dirty page data is written into the virtual storage area and when it is written into the actual storage area, the mapping relationship table is maintained in real time.

[0097] In one embodiment, since the computing unit is built with a Field Programmable Gate Array (FPGA), the FPGA is used to directly process dirty page data inside the virtual storage area, and the dirty page data is transmitted to the corresponding actual storage location through a Peripheral Component Interconnect Express (PCIe) interface. This process does not require the intervention of the CPU or memory. Since the dirty page data can be found at the actual storage location after being written to disk, there is no need to record the association between the virtual storage location and the actual storage location anymore; for example, when using the method of establishing a location mapping relationship to associate the virtual storage location with the actual storage location, after writing to the actual storage location, the location mapping relationship can be deleted.

[0098] The present invention directly writes dirty page data into the virtual storage area in the processing order, greatly improving the write throughput of the database system, alleviating the write blocking problem, and enhancing the performance of the database system. The dirty page data is directly written from the virtual storage location to the actual storage location through the computing unit, enabling the dirty page data to be automatically transferred from the virtual storage area to the actual storage area without passing through the memory and CPU; only using very little CPU and memory overhead, before the dirty page data is written to the actual storage location, the corresponding virtual storage location and the actual storage location are associated to ensure data consistency. Furthermore, in a write-intensive scenario, the CPU and memory overhead are controlled to a level similar to sequential writing, greatly reducing the memory and CPU overhead during the process of the dirty page data being written to disk, reducing random write operations, making the impact of the write-intensive scenario on the overall performance of the database system smaller, and enhancing the performance of the database system based on the B+ tree storage engine in the write-intensive scenario.

[0099] To enable quick and accurate query of the dirty page data after it is written to the virtual storage area and ensure data accuracy, as Figure 4 shown, step 20 includes:

[0100] Step 201: Initialize a mapping relationship table in the memory in advance.

[0101] As Figure 3 shown, in one embodiment, a mapping relationship table can be maintained in the common memory pool.

[0102] Step 202: Obtain the dirty page data from the data buffer according to the processing order.

[0103] Step 203: Write the dirty page data into the virtual storage area.

[0104] Step 204: Obtain the virtual storage location of the dirty page data in the virtual storage area.

[0105] Step 205: Establish a location mapping relationship between the actual storage location and the virtual storage location, and update the location mapping relationship to the mapping relationship table.

[0106] In the embodiment of the present invention, the actual storage location and the virtual storage location of the dirty page data are obtained. When the dirty page data is stored in the virtual storage area or the corresponding virtual storage location changes, the location mapping relationship of the dirty page data is updated in real time in the mapping relationship table. Then, when querying the dirty page data and the dirty page data has not been written to its actual storage location, the data can be obtained quickly and accurately, ensuring data consistency and the accuracy of data query. When continuously iteratively updating the dirty page data subsequently, there is no need to change the actual storage location, and only the data needs to be updated and the corresponding location mapping relationship needs to be updated.

[0107] In an alternative embodiment, when the storage medium of the virtual storage area is not an intelligent solid-state drive, the dirty page data can also be written from the virtual storage location to the actual storage location through a direct interconnection technology to avoid the intervention of the CPU and memory. As Figure 5 shown, after the step 20, it further includes:

[0108] Step 301: Connect the virtual storage area and the actual storage area using a physical hardware interface; wherein, the actual storage location is in the actual storage area.

[0109] In the embodiment of the present invention, a high-speed communication protocol is used to directly connect the storage devices of the virtual storage area and the actual storage area through a hardware interface to achieve efficient data transmission and communication. Among them, the high-speed communication protocol and the corresponding hardware interface are specifically selected by those skilled in the art according to the actual usage scenario and experience. In an alternative embodiment, the storage medium of the virtual storage area can be the same as the storage medium of the actual storage area; the hardware interface can be a PCIe interface, and the high-speed communication protocol can be an NVLink protocol.

[0110] Step 302: When the load of the central processing unit of the database system is lower than the load threshold, write the dirty page data from the virtual storage location to the actual storage location through the physical hardware interface.

[0111] In the embodiment of the present invention, for the write-intensive scenario, after waiting for the CPU load of the database system to decrease, the dirty page data is written from the virtual storage area to the actual storage area, and then the location mapping relationship of the dirty page data is deleted from the mapping relationship table; and when the write-intensive scenario lasts for a long time and the virtual storage area has been filled with data and cannot expand the space, the existing technology is adopted to directly write the dirty page data from the data buffer to the actual storage area.

[0112] It should be noted that the storage medium of the actual storage area generally will not be an intelligent solid-state drive; and once the storage medium of the virtual storage area is an intelligent solid-state drive, in order to reduce the memory and CPU overhead, the computing node of the intelligent solid-state drive can be used to write the dirty page data to the actual storage location.

[0113] In the prior art, in the process of writing dirty page data to the actual storage location, no matter what optimization means are adopted, only the performance overhead of random writing can be alleviated to a certain extent, the performance limit of random writing cannot be broken through, and the optimization of memory and CPU overhead is limited.

[0114] The present invention directly converts random writing into sequential writing first, and controls the memory and CPU overhead to a level similar to that of sequential writing. Although the update of the mapping relationship table still requires the intervention of memory and CPU, the overall performance overhead is basically similar to that of sequential writing. And in the process of writing dirty page data from the intelligent solid-state drive to the actual storage location, since there is no need to use memory and CPU to control data transmission, it will not affect the overall performance of the database system. When the database system is in a write-intensive scenario, the present invention improves the write throughput of data falling to disk and improves the overall performance of the database system.

[0115] Example 2:

[0116] When using the method for quickly falling disk of write-intensive load data in Embodiment 1 of the present invention, the capacity of the virtual storage area will affect the performance of the database system. When the virtual storage area is too small, once the write-intensive scenario lasts for a long time, it may be necessary to directly write the dirty page data to the actual storage location using the prior art solution, and the database system cannot achieve the ideal performance; when the virtual storage area is set too large, since the database system will not always be in a write-intensive scenario, in a non-write-intensive scenario, it is very likely that a large number of storage areas in the virtual storage area will not be used for a long time, resulting in waste of storage resources.

[0117] This embodiment provides a scheme for adaptive change of the virtual storage area to avoid the situation of insufficient capacity of the virtual storage area or waste of storage resources. Specifically, as Figure 6 shown, before the step 10, it further includes:

[0118] Step 101: When a checkpoint operation is triggered, determine whether the database system enters a write-intensive load state.

[0119] This embodiment will take the operation of executing a checkpoint as an example to illustrate the process of adaptive change of the virtual storage area.

[0120] In one embodiment, when the database system triggers a checkpoint operation, the dirty page data that needs to be flushed to disk in the cache is extracted, added to the write queue, and the actual storage location of each piece of dirty page data is obtained.

[0121] Step 102: When the database system enters a write-intensive load state, feature extraction is performed on the amount of dirty page data in the virtual storage area and the system performance information of the database system to determine the amount of expansion and contraction of the virtual storage area.

[0122] Among them, in this embodiment, the amount of dirty page data in the virtual storage area in the current time period and the system performance information of the database system in the current time period are collected, feature extraction is performed on the collected data, and then the capacity of the virtual storage area required in this state is determined. By comparing the required capacity of the virtual storage area with the current capacity of the virtual storage area, the amount of expansion and contraction of the virtual storage area is obtained. In one embodiment, a machine learning method is used to train a scaling model to learn the relationship between the amount of dirty page data, system performance information, and the capacity of the virtual storage area; the trained scaling model is used to perform feature extraction on the collected data to obtain the capacity of the virtual storage area required in this state.

[0123] For ease of understanding, an embodiment of the present invention provides a specific example of system performance information as follows:

[0124] Flag Meaning <![CDATA[C s > CPU occupancy rate of the database system <![CDATA[S b > Write rate of the database system writing to the actual storage area <![CDATA[N t > Number of write threads of the database system <![CDATA[I O > Number of input / output operations per second of the database system <![CDATA[D u > Storage space utilization rate of the database system <![CDATA[D c > Storage space capacity of the database system <![CDATA[D w1 > Amount of dirty page writes in the first ten minutes of the database system <![CDATA[D w2 > Amount of dirty page writes in the first minute of the database system <![CDATA[D d > Disk usage distribution of the database system <![CDATA[P d1 > Amount of dirty page generation in the first ten minutes of the database system <![CDATA[P d2 > Amount of dirty page generation in the first minute of the database system <![CDATA[T r > Average response time in the first ten minutes of the database system <![CDATA[T d > Average processing time of transaction processing in the first ten minutes of the database system <![CDATA[C h > Cache hit rate in the first ten minutes of the database system

[0125] It should be noted that in the embodiments of the present invention, "before" and "after" are both based on the time node of triggering the checkpoint operation, that is, before triggering the checkpoint operation and after triggering the checkpoint operation.

[0126] Step 103: Update the capacity of the virtual storage area according to the amount of expansion and contraction of the virtual storage area, so as to write the dirty page data into the updated virtual storage area.

[0127] By adaptively changing the size of the virtual storage area, the situation of insufficient cache or wasted cache resources is avoided, and storage resources are utilized more reasonably.

[0128] In this embodiment, by collecting the amount of dirty page data in the current time period, the capacity of the virtual storage area is adaptively set to meet the requirements of different write-intensive scenarios with different amounts of dirty page data for the virtual storage area.

[0129] In one embodiment, the scaling model of this embodiment can be a neural network model. The scaling model is trained offline and used online to calculate the capacity of the virtual storage area required in the current time period, and then the amount of expansion and contraction of the virtual storage area is determined. Specifically, as Figure 7 shown, the step 102 includes:

[0130] Step 1021: Collect feature information for multiple time periods; determine the virtual storage area capacity under different feature information states; use the virtual storage area capacity as the label for the corresponding feature information to obtain a labeled training set.

[0131] Among them, the feature information includes the amount of dirty page data in the virtual storage area of the database system in each time period, the system performance information of the database system, and the virtual storage area capacity in the corresponding time period. Adding labels to the training set is an offline operation.

[0132] In order to reasonably plan the virtual storage area capacity according to the duration of the write-intensive scenario, this embodiment performs feature extraction on the feature information and learns the association between the virtual storage area capacities corresponding to different feature information, so that by inputting the feature information into the scaling model, the virtual storage area capacity corresponding to the feature information can be predicted. In one embodiment, the determining the virtual storage area capacity under different feature information states includes:

[0133] Determine the occupancy rate stage coefficient of the central processing unit of the database system; wherein, the occupancy rate stage coefficient is used to represent that the occupancy rate of the central processing unit is in a light state, a medium state, and a heavy state; in this embodiment, the occupancy rate of the central processing unit is divided into three stages, namely the light state, the medium state, and the heavy state; in an optional embodiment, the occupancy rate stage coefficient corresponding to the light state can be 1, the occupancy rate stage coefficient corresponding to the medium state can be 1.5, and the occupancy rate stage coefficient corresponding to the heavy state can be 2. Determine the write throughput from the virtual storage area to the actual storage area as the first write throughput; determine the write throughput from the data buffer to the virtual storage area as the second write throughput; determine the ratio of the first write throughput to the second write throughput as the write throughput coefficient; determine the total number of data that needs to be written from the data buffer to the actual storage area within the time interval from a first preset time before triggering the checkpoint operation to a first preset time after triggering the checkpoint operation as the data volume to be flushed to disk; wherein, the first preset time is selected by those skilled in the art according to the specific usage scenario; in an optional embodiment, the first preset time can be 5 minutes. Determine the product of the occupancy rate stage coefficient, the write throughput coefficient, and the data volume to be flushed to disk as the corresponding virtual storage area capacity. Specifically, the virtual storage area capacity under different feature information states can be calculated according to the following expression:

[0134]

[0135] where C nRepresents the virtual storage area capacity under different characteristic information states, X represents the occupancy rate stage coefficient, Thrv>a represents the first write throughput (i.e., the write throughput from the virtual storage area to the actual storage area), Thrc>v represents the second write throughput (i.e., the write throughput from the data buffer to the virtual storage area), and C represents the amount of data to be flushed to disk.

[0136] Step 1022: Construct an original scaling model; obtain a predicted scaling using the original scaling model; compare the actual scaling of the virtual storage area capacity with the predicted scaling to obtain a regression error value.

[0137] Initialize a neural network model as the original scaling model. In one embodiment, the input dimension of the original scaling model is the same as the feature dimension of the labeled training set; the original scaling model consists of an embedding layer, three hidden layers, a fully connected layer, a Dense layer, and an output layer in sequence, and the activation function uses the Rectified Linear Unit (ReLU) activation function.

[0138] For the write-intensive scenario, this embodiment provides a loss function for the scaling model, and the loss function consists of two parts, specifically as follows:

[0139] The first part of the loss function is the regression error, which is used to measure the deviation between the actual scaling of the virtual storage area capacity and the predicted scaling. Specifically, in one embodiment, for each characteristic information in the training set, the actual scaling of the virtual storage area capacity between each time period can be calculated according to the virtual storage area capacity. Use the labeled training set to train the original scaling model to obtain the predicted scaling. After obtaining the actual scaling and the corresponding predicted scaling, determine the difference between the actual scaling and the corresponding predicted scaling as the scaling deviation value; determine the sum of the squares of the scaling deviation values corresponding to each characteristic information as the deviation characteristic value; determine the ratio of the deviation characteristic value of each characteristic information to the total amount of characteristic information of each characteristic information as the regression error value; specifically, the regression error value can be calculated according to the following expression:

[0140]

[0141] Among them, Represents the regression error value, N represents the total number of characteristic information input into the original scaling model, y i Represents the actual scaling, Represents the predicted scaling.

[0142] Step 1023: Obtain the system performance information within different time periods from the feature information; calculate the system penalty value based on the difference between the system performance parameters and the corresponding thresholds in the system performance information; calculate the write operation penalty value based on the difference between the write operation parameters and the corresponding thresholds in the system performance information; calculate the resource usage penalty value based on the difference between the resource usage parameters and the corresponding thresholds in the system performance information; determine the sum of the system penalty value, the write operation penalty value, and the resource usage penalty value as the original penalty value.

[0143] Among them, the thresholds of the system performance parameters, the write operation parameters, and the resource usage parameters are selected by those skilled in the art according to specific usage scenarios and are not limited herein.

[0144] The second part of the loss function is the penalty term, which ensures that the scaling amount output by the scaling model meets the performance constraints of the database system; specifically, the penalty term includes three parts, namely the system penalty term the write operation penalty term and the resource usage penalty term The original penalty value of the overall penalty term is the sum of the three corresponding values, that is, the original penalty value

[0145] In one embodiment, the calculation expression of the system penalty value is:

[0146]

[0147] Among them, ReLu() represents the activation function, and α1, β1, and γ1 are all penalty weight coefficients, and T r represents: the average response time of the database system within the second preset time after triggering the checkpoint operation, and T rmax represents: the response time threshold of the database system within the second preset time after triggering the checkpoint operation, and T d represents: the average processing time of processing transactions within the second preset time after triggering the checkpoint operation, and T dmax represents: the processing time threshold of processing transactions within the second preset time after triggering the checkpoint operation, D u represents: the storage space utilization rate of the database system, D umax represents: the threshold of the storage space utilization rate; among them, the second preset time is selected by those skilled in the art according to specific usage scenarios; in an optional embodiment, the second preset time can be 10 minutes.

[0148] In one embodiment, the calculation expression of the write operation penalty value is:

[0149]

[0150] Among them, α2, β2, and γ2 are all penalty weight coefficients, D w2 represents the amount of dirty page data written within the third preset time before triggering the checkpoint operation, D w2max represents the threshold of the amount of dirty page data written, P d2 represents the amount of dirty page data generated within the third preset time before triggering the checkpoint operation, P d2max represents the threshold of the amount of dirty page data generated, I o represents the number of input / output operations per second of the database system, I omax represents the threshold of the number of input / output operations per second; among them, the third preset time is selected by those skilled in the art according to the specific usage scenario; in an optional embodiment, the third preset time can be 1 minute.

[0151] In one embodiment, the calculation expression of the resource usage penalty value is:

[0152]

[0153] Among them, α3, β3, and γ3 are all penalty weight coefficients, C s represents the occupancy rate of the central processing unit, C smax represents the threshold of the occupancy rate of the central processing unit, S b represents the write rate at which the database system writes to the actual storage area, S bmax represents the threshold of the write rate, C h represents the buffer hit rate of the database system within the second preset time before triggering the checkpoint operation, C hmax represents the threshold of the buffer hit rate.

[0154] Step 1024: Determine the product of the original penalty value and the first weight coefficient as the target penalty value, and determine the sum of the regression error value and the target penalty value as the loss function value; use the loss function value to iteratively optimize the parameters of the original scaling model to train the original scaling model to obtain the target scaling model.

[0155] The loss function of the original scaling model in this embodiment is the sum of two parts, that is:

[0156]

[0157] Among them, represents the loss function value, and λ represents the first weight coefficient.

[0158] Step 1025: When the database system enters the write-intensive load state, collect the amount of dirty page data in the virtual storage area and the system performance information of the database system during the current time period to obtain the characteristic information of the current time period; perform feature extraction based on the target scaling model to obtain the scaling amount of the virtual storage area.

[0159] Input the characteristic information in the labeled training set into the original scaling model to start training the original scaling model; during the training process, the original scaling model first performs forward propagation on the training set and calculates the predicted scaling amount according to the current weights. Then use the loss function of this embodiment to calculate the loss function value, and update the network weight parameters of the original scaling model in combination with the backpropagation algorithm. After the training is completed, the target scaling model is obtained. After obtaining the target scaling model, extract the trained network weight parameters from the target scaling model and save them to the parameter file; deploy the online scaling model using the network weight parameters in the parameter file. Input the characteristic information of the current time period collected in step 102 into this online scaling model to obtain the scaling amount of the virtual storage area, and use this scaling amount of the virtual storage area to dynamically adjust the capacity of the virtual storage area.

[0160] In an optional embodiment, the judgment accuracy of the online scaling model can also be collected in real time. When the judgment accuracy is lower than a certain value, retrain the target scaling model, save the new network weight parameters to the parameter file, and at the same time notify the online scaling model to update the network weight parameters.

[0161] In this embodiment, by training the target scaling model offline and placing the process with large time overhead in the offline process, the load overhead of the database system is reduced.

[0162] Example 4:

[0163] As Figure 9 shown, it is a schematic architecture diagram of the device for quickly flushing write-intensive load data according to an embodiment of the present invention. The device for quickly flushing write-intensive load data in this embodiment includes one or more processors 21 and a memory 22. Among them, Figure 9 One processor 21 is taken as an example.

[0164] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 9 Taking the connection through the bus as an example.

[0165] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs and non-volatile computer-executable programs, such as the method for quickly flushing write-intensive load data in this embodiment. The processor 21 executes the method for quickly flushing write-intensive load data by running the non-volatile software programs and instructions stored in the memory 22.

[0166] The memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 22 optionally includes a memory remotely located relative to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, execute the method for quickly flushing write-intensive load data in the above embodiments. For example, execute each step of the method for quickly flushing write-intensive load data in the embodiments of the present invention described above.

[0168] The embodiments of the present invention also provide a non-volatile computer storage medium, which stores computer-executable instructions, and these computer-executable instructions are executed by one or more processors, for example Figure 9 a processor 21, which enables the above one or more processors to execute the method for quickly flushing write-intensive load data in the specific embodiments of the present invention. For example, execute each step of the method for quickly flushing write-intensive load data in the embodiments of the present invention described above; it can also implement Figure 9 the respective modules and units described above; or execute the method for quickly flushing write-intensive load data in the specific embodiments of the present invention. For example, execute each step of the method for quickly flushing write-intensive load data in the embodiments of the present invention described above; it can also implement Figure 9 the respective modules and units described above.

[0169] It should be noted that the content such as information interaction and execution process between the modules and units in the above device and system, due to being based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention, and will not be elaborated here.

[0170] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium. The storage medium can include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disks or optical discs, etc.

[0171] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for quickly flushing write-intensive load data to disk, characterized in that, Including: Obtain the actual storage location of the dirty page data and determine the processing order of the dirty page data; Write the dirty page data from the data buffer to the virtual storage area according to the processing order, determine the virtual storage location of the dirty page data, and associate the virtual storage location with the actual storage location; The computing unit of the virtual storage area obtains the actual storage location and writes the dirty page data from the virtual storage location to the actual storage location; wherein, the storage medium of the virtual storage area is an intelligent solid-state drive.

2. The method for quickly flushing write-intensive load data according to claim 1, wherein The step of writing the dirty page data from the data buffer to the virtual storage area according to the processing order, determining the virtual storage location of the dirty page data, and associating the virtual storage location with the actual storage location includes: Initialize a mapping relationship table in the memory in advance; Obtain the dirty page data from the data buffer according to the processing order; Write the dirty page data to the virtual storage area; Obtain the virtual storage location of the dirty page data in the virtual storage area; Establish a location mapping relationship between the actual storage location and the virtual storage location, and update the location mapping relationship to the mapping relationship table.

3. The method for quickly flushing write-intensive load data to disk according to claim 2, wherein, After the step of writing the dirty page data from the data buffer to the virtual storage area according to the processing order, determining the virtual storage location of the dirty page data, and associating the virtual storage location with the actual storage location, it further includes: Connect the virtual storage area and the actual storage area using a physical hardware interface; wherein, the actual storage location is in the actual storage area; When the central processing unit load of the database system is lower than the load threshold, write the dirty page data from the virtual storage location to the actual storage location through the physical hardware interface.

4. The method for quickly flushing write-intensive load data to disk according to claim 1, characterized in that Before the step of obtaining the actual storage location of the dirty page data and determining the processing order of the dirty page data, it further includes: When a checkpoint operation is triggered, determine whether the database system enters a write-intensive load state; When the database system enters a write-intensive load state, extract features from the dirty page data volume of the virtual storage area and the system performance information of the database system to determine the expansion and contraction amount of the virtual storage area; Update the capacity of the virtual storage area according to the expansion and contraction amount of the virtual storage area, so as to facilitate writing the dirty page data to the updated virtual storage area.

5. The method for quickly flushing write-intensive load data according to claim 4, wherein The step of extracting features from the dirty page data volume of the virtual storage area and the system performance information of the database system to determine the expansion and contraction amount of the virtual storage area when the database system enters a write-intensive load state includes: Collect feature information for multiple time periods; determine the capacity of the virtual storage area in different feature information states; use the capacity of the virtual storage area as the label of the corresponding feature information to obtain a labeled training set; Construct an original expansion and contraction amount model; use the original expansion and contraction amount model to obtain a predicted expansion and contraction amount; compare the actual expansion and contraction amount of the virtual storage area capacity with the predicted expansion and contraction amount to obtain a regression error value; Obtain system performance information for different time periods from the feature information; calculate a system penalty value based on the difference between the system performance parameters in the system performance information and the corresponding thresholds; calculate a write operation penalty value based on the difference between the write operation parameters in the system performance information and the corresponding thresholds; calculate a resource usage penalty value based on the difference between the resource usage parameters in the system performance information and the corresponding thresholds; determine the sum of the system penalty value, the write operation penalty value, and the resource usage penalty value as the original penalty value; Determine the target penalty value as the product of the original penalty value and the first weight coefficient, and determine the loss function value as the sum of the regression error value and the target penalty value; use the loss function value to iteratively optimize the parameters of the original scaling model to train the original scaling model and obtain the target scaling model; When the database system enters a write-intensive load state, collect the amount of dirty page data in the virtual storage area and the system performance information of the database system during the current time period to obtain the feature information of the current time period; perform feature extraction based on the target scaling model to obtain the virtual storage area scaling amount.

6. The method for quickly flushing write-intensive load data according to claim 5, characterized in that The determination of the virtual storage area capacity in different feature information states includes: Determine the occupancy rate stage coefficient of the central processing unit of the database system; wherein, the occupancy rate stage coefficient is used to represent that the occupancy rate of the central processing unit is in a mild state, a moderate state, and a severe state; Determine the first write throughput as the write throughput from the virtual storage area to the actual storage area; determine the second write throughput as the write throughput from the data buffer to the virtual storage area; determine the ratio of the first write throughput to the second write throughput as the write throughput coefficient; Determine the total amount of data to be flushed to disk as the sum of the quantities that need to be written from the data buffer to the actual storage area within the time interval from the first preset time before triggering the checkpoint operation to the first preset time after triggering the checkpoint operation; Determine the corresponding virtual storage area capacity as the product of the occupancy rate stage coefficient, the write throughput coefficient, and the total amount of data to be flushed to disk.

7. The method for quickly flushing write-intensive load data according to claim 5, characterized in that, The calculation expression of the system penalty value is: Among them, ReLu() represents the activation function, and α1, β1, and γ1 are all penalty weight coefficients, T r represents the average response time of the database system within the second preset time after triggering the checkpoint operation, T rmax represents the response time threshold of the database system within the second preset time after triggering the checkpoint operation, T d represents the average processing time of processing transactions within the second preset time after triggering the checkpoint operation, T dmax represents the processing time threshold of processing transactions within the second preset time after triggering the checkpoint operation, D u represents the storage space utilization rate of the database system, D umax represents the threshold of the storage space utilization rate; The calculation expression of the write operation penalty value is: Among them, α2, β2, and γ2 are all penalty weight coefficients, D w2 represents the amount of dirty page data written within the third preset time before triggering the checkpoint operation, D w2max represents the threshold of the amount of dirty page data, P d2 represents the amount of dirty page data generated within the third preset time before triggering the checkpoint operation, P d2max represents the threshold of the amount of dirty page data generated, I o represents the number of input / output operations per second of the database system, I omax represents the threshold of the number of input / output operations per second; The calculation expression of the resource usage penalty value is: Among them, α3, β3, and γ3 are all penalty weight coefficients, C s represents: the occupancy rate of the central processing unit, C smax represents: the threshold of the occupancy rate of the central processing unit, S b represents: the write rate at which the database system writes to the actual storage area, S bmax represents: the threshold of the write rate, C h represents: the buffer hit rate of the database system within a second preset time before triggering the checkpoint operation, C hmax represents: the threshold of the buffer hit rate.

8. The method for quickly flushing write-intensive load data to disk according to claim 5, characterized in that, Construct the original scaling model; use the original scaling model to obtain the predicted scaling amount; Comparing the actual scaling amount of the virtual storage area capacity with the predicted scaling amount to obtain the regression error value includes: Calculate the actual scaling amount of the virtual storage area capacity between each time period according to the virtual storage area capacity; Train the original scaling model using the labeled training set to obtain the predicted scaling amount; Determine the difference between the actual scaling amount and the corresponding predicted scaling amount as the scaling amount deviation value; Determine the sum of the squares of the scaling amount deviation values corresponding to each feature information as the deviation feature value; Determine the ratio of the deviation feature value of each feature information to the total amount of feature information of each feature information as the regression error value.

9. A device for quickly flushing write-intensive load data, characterized in that, The device for quickly flushing write-intensive load data includes at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and after being executed by the processor, the instructions are used to implement the method for quickly flushing write-intensive load data according to any one of claims 1-8.

10. A non-volatile computer storage medium, characterized in that, The computer storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors to complete the method for quickly flushing write-intensive load data according to any one of claims 1-8.

Images