Data writing methods in databases, database software products, equipment, media and products

Abstract

This application provides a data writing method, database software product, medium, and product for a database. The method includes: allocating target data corresponding to a data write operation to a target storage space in the database; then, determining a target speed based on the I / O amplification factor of the disk layer in the target storage space. In response to writing the target data to the write-ahead log, the target data can be written to the memory layer of the target storage space at the target speed, i.e., performing foreground data writing; finally, the target data is written to the disk layer of the target storage space, i.e., performing background data processing. In this method, the target speed determines the foreground data write speed, and the target speed is obtained based on an indicator reflecting the background data processing speed, i.e., the I / O amplification factor of the disk layer. This ensures that the foreground data write speed matches the background processing speed, allowing data to be written to the database stably and improving write performance fluctuations during the data writing process.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data writing method, database software product, device, medium and product in a database. Background Technology

[0002] With the rapid development of information technology, the amount of data generated in various scenarios is growing exponentially. Databases are typically used to efficiently store this exponentially increasing data while ensuring data security. Data can be written to the database at a raw speed, but during this process, the database is prone to write stalls or even write stops, which significantly reduce the data writing speed. Once the write stall or write stop is resolved, the original writing speed can be restored. This fluctuation in database write performance can negatively impact the user experience.

[0003] Based on the above description, how to improve the write performance fluctuations in the database has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a data writing method, database software product, device, medium, and product for improving write performance fluctuations in the database.

[0005] This application provides a method for writing data to a database, the method comprising:

[0006] In response to a data write operation, the target data corresponding to the data write operation is allocated to the target storage space of the database, the target storage space including a memory layer and at least one disk layer;

[0007] The target speed is determined based on the I / O amplification factor of the disk layer in the target storage space;

[0008] In response to the target data being written to the write-ahead log, the target data is written to the memory layer in the target storage space at the target speed;

[0009] The data in the memory layer is written to at least one disk layer in the target storage space.

[0010] This application provides a database software product, including: a read / write controller and a statistics module; the database runs a target foreground write thread and a background processing thread;

[0011] The read / write controller is configured to, in response to a data write operation, allocate the target data corresponding to the data write operation to the target storage space of the database, wherein the target storage space includes a memory layer and at least one level of disk layer;

[0012] Control the target foreground write thread to write the target data into the write-ahead log;

[0013] Control the target foreground write thread to write the target data into the memory layer of the target storage space at the target speed;

[0014] Control the background processing thread to write the data in the memory layer to at least one disk layer in the target storage space;

[0015] The statistics module is used to calculate the I / O amplification factor of the disk layer in the target storage space; and to determine the target speed based on the I / O amplification factor of the disk layer.

[0016] This application provides an electronic device including a processor and a memory. The memory stores one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the data writing method in the database described above. The electronic device may also include a communication interface for communicating with other devices or communication networks.

[0017] This application provides a non-transitory machine-readable storage medium storing executable code. When the executable code is executed by a processor of an electronic device, the processor can at least implement the data writing method in the database as described above.

[0018] This application provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the data writing method in the database as described above.

[0019] The data writing method for a database provided in this application, in response to a data write operation, allocates the target data corresponding to the data write operation to the target storage space of the database. This target storage space may include a memory layer and at least one disk layer. Then, a target speed can be determined based on the I / O amplification factor of the disk layer in the target storage space. After the target data is written to the write-ahead log, the target data is written to the memory layer of the target storage space according to the target speed, thus achieving foreground data writing. Finally, the data in the memory layer is written to at least one disk layer of the target storage space, thus achieving background data processing.

[0020] In the above method, the target data is written to the foreground at the target speed, meaning the target speed determines the foreground write speed. Furthermore, the target speed is derived from an indicator reflecting the background processing speed, namely the disk-level I / O amplification factor. Therefore, using this method, the foreground write speed can match the background processing speed, significantly reducing the frequency of write stalls or write stops caused by excessively high foreground write speeds and insufficient background processing speeds. Data can be written to the database stably and continuously, thus improving write performance fluctuations during the data writing process. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1a This is a schematic diagram of the structure of a database software product provided in an embodiment of this application;

[0023] Figure 1b This is a schematic diagram of the structure of another database software product provided in an embodiment of this application;

[0024] Figure 2 This is a schematic diagram of the structure of another database software product provided in an embodiment of this application;

[0025] Figure 3 This is an application diagram of a cloud computing environment provided in an embodiment of this application;

[0026] Figure 4 A flowchart illustrating a data writing method in a database, as provided in this application embodiment;

[0027] Figure 5 A flowchart illustrating another data writing method in a database provided in this application embodiment;

[0028] Figure 6 A flowchart illustrating yet another data writing method in a database provided in this application embodiment;

[0029] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0032] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0033] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to identification.” Similarly, depending on the context, the phrases “if determination” or “if identification (of the condition or event of the statement)” can be interpreted as “when determination” or “in response to determination” or “when identification (of the condition or event of the statement)” or “in response to identification (of the condition or event of the statement).”

[0034] It should be noted that, in the cases involving user interaction operations or triggering operations in the embodiments of this application, the user interaction operations or triggering operations involved in the embodiments of this application include, but are not limited to, various interaction operations such as touch operations, gesture operations, voice operations, head movement operations, and eye movement operations; among them, touch operations include, but are not limited to, click operations, double click operations, long press operations, swipe operations, pinch operations, or mouse hover operations. Swipe operations include, but are not limited to, straight line swipes and curved line swipes.

[0035] It should be noted that, in the case of user information involved in the embodiments of this application, the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0036] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0037] Before describing the various embodiments provided in this application in detail, the relevant concepts involved in the various embodiments of this application may also be introduced.

[0038] Data storage system: The infrastructure used to store, manage, and retrieve data. A common application of data storage systems includes databases, which optionally support a Log-Structured data storage architecture. Databases can use Log-Structured Merge-Tree (LSM tree) or Log-Structured Merge-TreeGraph (LSM-Graph) as their core data structure. Data in a database can be organized in column families. In summary, data in a database can be organized in column families and stored in the form of Log-Structured Merge-Tree or Log-Structured Merge-Graph, with one column family corresponding to one LSM tree. Optionally, databases that organize and store data in the above manner can include RocksDB, LevelDB, etc.

[0039] LSM Tree: This structure divides data storage into a memory layer and at least one disk layer. Data can be stored in a memory table within the memory layer, and once the memory table is full, it can be further converted into an immutable memory table. This immutable memory table can then be written to the disk layer. The relationship between RocksDB, the LSM tree, the memory layer, and the disk layer is illustrated in Figure 1.

[0040] The flush thread is used to flush the data contained in the immutable memory table in the memory layer to the first-level disk layer corresponding to the column family.

[0041] The Compaction thread is used to merge and compress data from different disk layers until all data in the memory layer is compressed to the last level of the column family's disk layer.

[0042] Read / write amplification: During data processing by the refresh and merge threads, the total amount of data read and written is generally greater than the amount of data actually written to the disk layer. The degree of read / write amplification can be measured by the read / write amplification factor. The read / write amplification factor can be calculated as: "the total amount of data read and written during the data refresh and merge process" divided by "the amount actually written to the database".

[0043] Write amplification: During data processing by the flush and merge threads, the total amount of data written is generally greater than the amount of data actually written to the disk layer. Similarly, the degree of write amplification can be measured by the write amplification factor. Write amplification can be calculated as: "the total amount of data written during data flushing and merging by the flush and merge threads" divided by "the amount actually written to the database".

[0044] Based on the above description, some embodiments of this application will be described in detail below with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0045] Figure 1a This is a schematic diagram illustrating the structure of a database software product provided in an embodiment of this application. Figure 1aAs shown, the database may include a read / write controller and a statistics module. The database software product provides multiple storage spaces, each including a memory layer and at least one disk layer. Furthermore, the database software product can run target foreground write threads and background processing threads. For the sake of brevity in the following embodiments, the database software product can be simply referred to as a database. The target foreground write thread is called the target thread. Optionally, the background processing thread may specifically include a refresh thread and a merge thread.

[0046] Based on this, the process of writing data into the database can be described in detail below:

[0047] Users can act as data producers, triggering data write operations to the database. The target data corresponding to these write operations becomes the database's consumable data. In response to a user-triggered write operation, the read / write controller allocates the target data to a target thread running in the database. The controller also determines the target storage space in the database to receive the target data. Subsequently, the controller controls the target thread to first write the target data to the Write-Ahead Logging (WAL) log at a certain speed, and then write the target data to the memory layer of the target storage space, completing the write operation. The memory layer can include multiple memory tables. The process of the target thread writing the target data to the memory layer specifically involves the target thread writing the target data to the memory tables contained within the memory layer. Furthermore, when a memory table is full, it can be further converted into an immutable memory table. This process of writing target data to the memory layer can be called the front-end write process, and the amount of data written to the memory layer can be used to measure the data producer's productivity.

[0048] Optionally, such as Figure 1a As shown, the database can also run background processing threads as data consumers, namely refresh threads and merge threads. After the target data is written to the memory layer of the target storage space, the refresh threads and merge threads can perform the following processing on the target data:

[0049] When an immutable memory table exists in the memory layer or the memory layer is full, the flush thread can flush the data contained in the immutable memory table in the memory layer to the first-level disk layer of the target storage space. When the first-level disk layer is full, the merge thread can merge the data in the first-level disk layer into the second-level disk layer; similarly, after the second-level disk layer is full, the merge thread can merge the data in the second-level disk layer into the next lower-level disk layer, and so on, until the amount of data stored in each level of disk layer is less than a threshold, thus completing the consumption of the target data. The above process of flushing and merging data to the disk layer can be called the background data processing process. Furthermore, the data processing speed of the flush thread and the merge thread can be used to measure the consumption capacity of the data consumer.

[0050] Therefore, the data writing process in a database can be summarized as follows: first write to the write-ahead log—then write to memory—then write to the database disk. Figure 1a The database shown supports the Log-Structured data storage architecture. In the process described above, the target thread can perform foreground data writing at a certain speed, while the refresh and merge threads can perform background data processing at a certain speed. For clarity and conciseness in the following description, the speed at which data is written to the memory layer can be called the foreground data write speed, and the processing of data in the memory layer by the refresh and merge threads can be called the background data processing speed. It's easy to understand that if the foreground data write speed and the background processing speed do not match—for example, if the foreground write speed is greater than the background processing speed—the database (more accurately, the target storage space) can easily enter a write stagnation state or even a write suspension state. At this time, the database write performance will fluctuate, i.e., the database write performance will decrease, and users can clearly feel the decrease in database write performance, thus affecting the user experience.

[0051] To address the aforementioned issues, this embodiment also includes a statistics module in the database. This module can extract the consumption capacity of data consumers and use that capacity to determine the production capacity of data producers, thereby ensuring that the consumption capacity of data consumers matches the production capacity of data producers and reducing the frequency with which the database enters a write stagnation or write suspension state.

[0052] Specifically, the statistics module can calculate the I / O amplification factor of the disk layer in the target storage space and determine a suitable foreground write speed, i.e., the target speed, based on this I / O amplification factor. Optionally, the target speed can also be equal to the remaining I / O bandwidth of the disk layer in the target storage space, both measured in bits per second. Optionally, when the I / O amplification factor is specifically a read / write amplification factor, this remaining I / O bandwidth is the remaining read / write bandwidth; when the I / O amplification factor is specifically a write amplification factor, this remaining I / O bandwidth is the remaining write bandwidth. Finally, the read / write controller can allocate the target speed to the target thread, which can then write the target data into the memory layer of the target storage space at the target speed. Optionally, the target speed can be encapsulated in a token, so that after the target thread requests a token from the read / write controller, the controller allocates the token to the target thread.

[0053] As can be seen, the target speed determined by the statistics module controls the write speed of the target data, thus determining the foreground write speed. Furthermore, the statistics module uses an indicator that reflects the background processing speed of the data, namely the disk-level I / O amplification factor, when determining the target speed. This means that the speed allocation period and target speed that determine the foreground write speed are derived from the background processing speed. Therefore, the foreground write speed determined by the statistics module matches the background processing speed. For example, a larger I / O amplification factor corresponds to a smaller background processing speed.

[0054] Optionally, for different data write operations triggered by users at different times, the read / write controller can allocate the data corresponding to each different data write operation to the corresponding storage space and thread. Simultaneously, the statistics module can determine the corresponding speed for each data write operation. Furthermore, since the storage space changes after any data write operation, the statistics module can re-determine a speed when the next data write operation occurs.

[0055] In this embodiment, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target storage space of the database. This target storage space may include a memory layer and at least one disk layer. Then, a target speed can be determined based on the I / O amplification factor of the disk layer in the target storage space. After the target data is written to the write-ahead log, the target data is written to the memory layer of the target storage space according to the target speed, thus achieving foreground data writing. Finally, the data in the memory layer is written to at least one disk layer of the target storage space, thus achieving background data processing.

[0056] In the above method, the target data is written to the foreground at the target speed, meaning the target speed determines the foreground write speed. Furthermore, the target speed is derived from an indicator reflecting the background processing speed, namely the disk-level I / O amplification factor. Therefore, using this method, the foreground write speed can match the background processing speed, significantly reducing the frequency of write stalls or write stops caused by excessively high foreground write speeds and insufficient background processing speeds. Data can be written to the database stably and continuously, thus improving write performance fluctuations during the data writing process.

[0057] Furthermore, the beneficial effects of the database and data writing method provided in the various embodiments of this application can also be understood in conjunction with the following:

[0058] The embodiment shown in Figure 1 describes a data writing method in which, in response to a user-triggered data writing operation, the database background processing speed is not considered. Instead, data is written to the foreground at a certain speed (i.e., a preset speed), and then the data is processed in the background.

[0059] However, when writing data to the front end at a preset speed, since this preset speed does not take into account the background processing speed, a mismatch between the front end writing speed and the background processing speed can easily occur. For example, the preset speed may exceed the background processing speed, causing the database (more accurately, the storage space for the written data) to easily enter a write stagnation state. After entering a write stagnation state, the read / write controller can reduce the preset speed by a preset percentage and continue writing data to the front end at the reduced speed. Then, in one scenario, if using the reduced speed still causes write stagnation, the read / write controller can further reduce the front end writing speed. In another scenario, if using the reduced speed does not cause write stagnation, the preset speed can be restored for front end writing.

[0060] Optionally, the read / write controller can determine whether the storage space has entered a write stall or write hangup state based on at least one of the following pressure indicators of the storage space where data is written in the database: the number of immutable memory tables in the storage space, the number of files in the first-level disk layer of the storage space, and the number of bytes rewritten by the merge thread during the process of merging the data in the memory layer into the last-level disk layer.

[0061] Specifically, storage space can be determined to enter a write stall state when at least one of the following conditions is met: the number of immutable memory tables is greater than a first number, the number of files in the first-level disk layer is greater than a second number, and the number of rewritten bytes is greater than a third number. Storage space can be determined to enter a write suspended state when at least one of the following conditions is met: the number of immutable memory tables is greater than a third number, the number of files in the first-level disk layer is greater than a fourth number, and the number of rewritten bytes is greater than a sixth number. Where the first number is less than the fourth number, the second number is less than the fifth number, and the third number is less than the sixth number.

[0062] The above data writing process is prone to the following problems:

[0063] 1. Since there is no limit to the front-end write speed of data at the beginning, it is easy for the database to enter a write sluggish state or a write suspended state, which will cause significant fluctuations in database write performance and thus affect the user experience.

[0064] 2. When entering a write stagnation state, a preset ratio can be used to reduce the preset speed. However, since this preset ratio is set manually, setting it too small will not improve the write stagnation state and may even cause the database to enter a write suspension state. Setting it too large will easily reduce the foreground write speed of data and may also cause the database to enter a write suspension state.

[0065] Using the solutions provided in the embodiments of this application, after a user triggers a data write operation, the read / write controller can immediately allocate a foreground write speed to the data corresponding to the data write operation. Since the foreground write speed is a speed that matches the background processing speed, the frequency of the database entering a write lag or write suspension state can be reduced. The thread can continuously and stably perform foreground data writing at this speed, thereby ensuring the stability of the database write performance.

[0066] for Figure 1a The database shown supports the Log-Structured data storage architecture. Optionally, data can be organized in the storage space as column families, or stored as a log structure merge tree or log structure merge graph. The target storage space corresponds to the target column family in the database, and the log structure merge tree or log structure merge graph corresponding to the target column family can include a memory layer and at least one disk layer. In this case, Figure 1a The database shown can be transformed into Figure 1b .

[0067] like Figure 1bAs shown, the database storage space can be logically divided into multiple column families. Assuming data can be stored in column families in the form of an LSM tree, each column family can correspond to one LSM tree, and any LSM tree can include a memory layer and at least one disk layer. Optionally, any disk layer can include a first-level disk layer (level 0), a second-level disk layer (level 1), ..., and a final-level disk layer (level i). Optionally, data can be stored as files in both the memory layer and the disk layer. Optionally, both the memory layer and the disk layer have their own data volume thresholds, that is, each layer has its own maximum data storage capacity. For at least one disk layer, the data volume threshold of the disk layer increases exponentially; for example, the data volume threshold of the second-level disk layer is 10 times that of the first-level disk layer, the data volume threshold of the third-level disk layer is 10 times that of the second-level disk layer, and so on.

[0068] When the statistics module is not used, in response to user-triggered data write operations, the read / write controller can allocate the target data to the target thread running in the database. The read / write controller can also determine the target column family in the database to receive the target data. Subsequently, the read / write controller can control the target thread to first write the target data to the write-ahead log at a certain speed, and then write the target data to the memory table contained in the memory layer corresponding to the target column family. Furthermore, when a memory table is full, it can be further converted into an immutable memory table.

[0069] When an immutable memory table appears in the memory layer or the memory layer is full, the flush thread can flush the data contained in the immutable memory table in the memory layer to the first-level disk layer corresponding to the target column family. When the first-level disk layer is full, the merge thread can merge the data in the first-level disk layer into the second-level disk layer; similarly, after the second-level disk layer is full, the merge thread can merge the data in the second-level disk layer into the next level disk layer, and so on, until the amount of data stored in each level disk layer is less than the threshold, thus completing the consumption of the target data.

[0070] The same issue will occur during the data writing process described above. Figure 1a The problem mentioned in the illustrated embodiment is that the database (more precisely, the target column family) enters a write sluggish state or even a write suspended state, which leads to a decrease in database write performance and affects user experience.

[0071] To address the aforementioned issues, this embodiment may use... Figure 1bThe statistics module in the database can calculate the I / O amplification factor of the disk layer corresponding to the target column family and determine a suitable foreground write speed, i.e., the target speed, based on this I / O amplification factor. Optionally, the target speed can also be equal to the remaining I / O bandwidth of the disk layer corresponding to the target column family; both are measured in bits per second. Optionally, when the I / O amplification factor is specifically a read / write amplification factor, this remaining I / O bandwidth is the remaining read / write bandwidth; when the I / O amplification factor is specifically a write amplification factor, this remaining I / O bandwidth is the remaining write bandwidth. Finally, the read / write controller can allocate the target speed to the target thread, and the target thread can then write the target data into the memory layer of the target column family according to the target speed.

[0072] In this embodiment, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target column family of the database. The data is stored in the column family of the database in the form of a log structure merge tree, which includes a memory layer and at least one disk layer. Then, the target speed allocated to the target data is determined based on the I / O amplification factor of the disk layer corresponding to the target column family, and the target data is written to the memory layer of the target column family according to the target speed, thus realizing the foreground writing of the data.

[0073] In the above method, the target data is written to the front end at the target speed, meaning the target speed determines the front end write speed. Furthermore, the target speed is derived from an indicator reflecting the back end processing speed, namely the disk-level I / O amplification factor. Therefore, using this method, the front end write speed can match the back end processing speed, significantly reducing the frequency of write stalls or write stops caused by excessively high front end write speeds and insufficient back end processing speeds. Data can be written to the column family stably and continuously, thus improving write performance fluctuations during the data writing process.

[0074] comprehensive Figure 1a and Figure 1b As shown in the illustrated embodiment, for a database that supports the Log-Structured data storage architecture, the data writing process can be: first write to the write-ahead log—then write to memory—then write to the disk. Figure 1b This paper presents a specific implementation of a Log-Structured data storage architecture. Data is organized in column families within the database storage space and stored as a log structure merge tree or log structure merge graph. One storage space in the database can correspond to one column family, and the LSM tree or LSM-Graph corresponding to a column family can include a memory layer and at least one disk layer. Therefore, regardless of... Figure 1a still Figure 1bThe database shown can achieve continuous and stable data writing with the help of the statistics module.

[0075] The following embodiments can Figure 1b Using the database shown as an example, the process of determining the I / O amplification factor is introduced. It should be noted that by replacing the target column family with the target storage space in the following embodiments; replacing the memory layer corresponding to the target column family with the memory layer of the target storage space; and replacing the disk layer corresponding to the target column family with the disk layer of the target storage space, it can be understood that... Figure 1a The database shown illustrates how the statistics module determines the I / O amplification factor.

[0076] The following can be used as Figure 1b Taking the database shown as an example, the process of determining the I / O amplification factor of the disk layer corresponding to the target column family is introduced:

[0077] In the above embodiments, the target speed determined by the statistics module can optionally be contained in a token; correspondingly, the speed allocation period is also the token allocation period. In response to a user-triggered data write operation, the read / write controller can encapsulate the target speed calculated by the statistics module into a token when the token allocation period arrives, and allocate this token to the target thread. The target thread can then, in response to receiving the token, write the target data into the memory layer corresponding to the target column family according to the target speed contained in the token.

[0078] For the target speed calculation, one optional method is for the statistics module to determine the target speed based on the data volume of the target data and the I / O amplification factor of the disk layer of the target column family. Specifically, target speed = data volume of the target data * disk layer I / O amplification factor.

[0079] Alternatively, the statistics module can obtain weight parameters reflecting the write pressure of the target column family, and then determine the target speed based on the target data volume, the disk layer I / O amplification factor, and the preset weight parameters. Specifically, the target speed = target data volume * disk layer I / O amplification factor * preset weight parameters.

[0080] The disk layer I / O amplification factor can specifically include read-write amplification factor or write amplification factor. However, regardless of the type of factor, as introduced in the previous related concepts of the embodiment shown in Figure 1, the I / O amplification factor can only be calculated after the data is written. Therefore, the statistics module actually calculates the disk layer I / O amplification factor corresponding to the target column family based on the historical data written to the target column family before the target data. After receiving the target data, this I / O amplification factor is used to determine the target speed and speed allocation period of the target data, and finally the target data is written to the target column family in the database.

[0081] For the I / O amplification factor of the disk layer, one optional method is to write historical data generated before the target data from any disk layer corresponding to the target column family to the next lower disk layer. After merging, the statistics module can calculate the I / O amplification factor of the next lower disk layer. This next lower disk layer is also any disk layer of the target column family. Further, the statistics module can determine the I / O amplification factor of the disk layer corresponding to the target column family based on the I / O amplification factors of different disk layers. Optionally, the statistics module can directly determine the I / O amplification factor of the disk layer corresponding to the target column family by summing the I / O amplification factors of different disk layers. The above statistical method can also be expressed as the following formula:

[0082]

[0083] in, Here, n is the I / O amplification factor for the disk layer corresponding to the target column family, and n is the total number of disk layers corresponding to the target column family. is the I / O amplification factor for the i-th disk layer.

[0084] For the I / O amplification factor of the next-level disk layer, one optional method, taking the read-write amplification factor as an example, is to calculate it as: "the total amount of data read and written during the process of merging data into the next-level disk layer" divided by "the amount actually written to the next-level disk layer". The write amplification factor follows a similar algorithm, which will not be elaborated here.

[0085] Optionally, as described in the embodiment shown in Figure 1, data can be stored as files in the memory layer and disk layer corresponding to the column family. Therefore, merging historical data in the disk layer by the merge thread is equivalent to merging historical files in the disk layer. In practice, the merge thread can sequentially merge multiple files in any disk layer into the next level disk layer. During the process of merging any file in any disk layer corresponding to the column family, if the file size is very small, it will increase the I / O amplification factor of the next level disk layer, further affecting the write performance of the column family. Specifically, if the file size is less than a first preset threshold, then the file is considered a small file.

[0086] To improve the above situation, the statistics module may optionally determine the I / O amplification factor of the next-level disk layer corresponding to the target column family in the following way:

[0087] If the size of any file being merged by the merging thread is less than a first preset threshold, it indicates that the merging thread is merging small files. In this case, the statistics module can obtain reference files that were merged before this file. The reference files can be at least one file, and the total size of the reference files and any other file must be higher than a second preset threshold, and the second preset threshold must not be less than the product of the first preset threshold and a preset ratio. Then, based on the respective file sizes of the file and reference files, weight parameters inversely proportional to the file sizes are set for each file and reference file, and the I / O amplification factor for the next level of disk is determined based on these weight parameters. Optionally, a maximum number of reference files can be set, for example, to 5.

[0088] For example, if the first preset threshold is 64MB and the preset ratio is 80%, then the second preset threshold is 51.2MB. Assuming any file is 200KB, its size is less than 64MB, making it a small file. The statistics module can then obtain two reference files preceding this file: reference file 1 is 20MB and reference file 2 is 30MB. The weight parameter for any file is set to 0.2 / 50, the weight parameter for reference file 1 is set to 20 / 51, and the weight parameter for reference file 2 is set to 30 / 51. The statistics module can then perform a weighted summation of the file and its respective weight parameters, using the weighted sum as the size of the file, and calculate the I / O amplification factor for the next disk layer based on this file size.

[0089] In this embodiment, for small files that are prone to causing excessive I / O amplification factor, the statistics module can use at least one reference file with a larger size preceding the small file and its weight parameters to smooth the size of the small file, and calculate the I / O amplification factor of the disk layer based on the smoothed file size, so as to improve the excessive I / O amplification factor caused by the small file size.

[0090] For preset weights, similar to the I / O amplification factor mentioned above, they can also be determined by using historical data of the target column family written before the target data.

[0091] In one optional manner, during the process of the refresh thread and the merge thread writing historical data prior to the target data to the disk layer corresponding to the target column family, the statistics module can monitor the pressure indicators of the target column family in real time and determine the preset weight parameter based on the monitored pressure indicators. Optionally, if at least one of the pressure indicators does not meet the corresponding preset value, the preset weight parameter is determined to be a value less than 1; if at least one of the pressure indicators meets the corresponding preset value, the preset weight parameter is determined to be equal to 1.

[0092] The stress metrics for the target column family can include the number of immutable in-memory tables in the target column family, the number of files in the first-level disk layer corresponding to the target column family, and the number of bytes rewritten by the merge thread during the process of merging data from the in-memory layer to the last-level disk layer. The preset values ​​for different stress metrics can be at least one of the first, second, and third quantities mentioned above.

[0093] In this embodiment, since the I / O amplification factor of the disk layer affects the background processing speed of the data, when the preset weight parameter is determined to be equal to 1, the target speed = the amount of target data * the I / O amplification factor of the disk layer. This target speed is matched with the background processing speed of the data. At this time, it can be ensured that the target data is stably and continuously written to the target column family.

[0094] In this embodiment, when the determined preset weight parameter is less than 1, the calculated target speed is slightly higher than the database's background processing speed. For example, if the preset weight parameter is 2 / 3, the target speed is 150% of the background processing speed, meaning the target speed exceeds the background processing speed. Since the target column family experiences less pressure at this time, the slightly higher target speed will not cause the column family to enter a write-limited or write-suspended state. Furthermore, the slightly higher target speed can provide users with a good data writing experience. In practice, it has also been found that when the foreground write speed is slightly higher than the background processing speed, the column family's I / O amplification factor actually decreases, thereby improving the database's write performance.

[0095] Optionally, while the target thread is writing target data into the target column family of the database at the target speed determined according to preset weight parameters, the read / write controller can also monitor the pressure index of the target column family in real time. In one scenario, if the pressure index consistently fails to meet the preset value or consistently meets the preset value, the preset weight parameters will not change, and the target speed will not change either. In this case, the target thread can write the target data into the target column family at a stable speed.

[0096] In another scenario, when at least one pressure metric changes from meeting the corresponding preset value to not meeting the corresponding preset value, the preset weight parameter will change from equal to 1 to less than 1. In this case, the target thread can continue to write the target data into the target column family at a speed slightly exceeding the background processing speed.

[0097] In another scenario, when at least one pressure indicator changes from not meeting the corresponding preset value to meeting the corresponding preset value, the preset weight parameter will change from equal to 1 to less than 1. Then, the target thread can write the target data into the target column family at a speed that matches the background processing speed.

[0098] In all the above scenarios, regardless of whether the preset weight reference is less than 1 or equal to 1, the target speed is at least sufficient to meet the backend data processing speed, and the frontend data writing speed will not be restricted too much, thus ensuring the database write performance.

[0099] Optionally, in practice, the I / O amplification factor may be statistically too large. This excessively large I / O amplification factor may directly cause the column family to enter a write stagnation or write suspension state. In this case, the foreground write speed of data will be limited, thus preventing the column family from fully utilizing its actual write capabilities. However, in the embodiments of this application, by using a preset weight parameter less than 1, a higher foreground write speed can be obtained, thereby improving the problem of low foreground write speed caused by the statistically large I / O amplification factor.

[0100] For data corresponding to user-triggered write operations, after it is written to the memory layer corresponding to the column family, the database's merge thread can be used to merge the data in different disk layers corresponding to that column family. Optionally, the merge thread can also be used to merge other data in the database that is not generated by the user. The above-mentioned merging of data from different sources can be considered as different types of merge tasks executed by the merge thread. Among them, the task of merging user-written data can be called the target merge task, and the task of merging non-user-written data can be called other merge tasks.

[0101] Since the I / O amplification factor can be related to the data written by the user, in order to further improve the accuracy of the I / O amplification factor statistics, the statistics module can optionally obtain the task type of each of the different merging tasks executed by the merging thread; and select the target merging task from the different merging tasks according to the task type. Finally, the disk layer I / O amplification factor is determined according to the number of target rewritten bytes rewritten by the merging thread during the execution of the target merging task.

[0102] In this embodiment, the statistics module can filter out rewritten bytes that are unrelated to user-written data by the merging thread through the merging task. Finally, it can count the remaining rewritten bytes that are related to user-written data to obtain the disk layer I / O amplification factor corresponding to the column family.

[0103] Figure 2 This is a schematic diagram illustrating the structure of another database software product provided in an embodiment of this application. Figure 1b Based on the database shown, such as Figure 2 As shown, the database may also include management components.

[0104] In the above embodiments, the statistics module calculates the I / O amplification factor for the disk layer. Optionally, in one case, the I / O amplification factor calculated by the statistics module can be the read-write amplification factor; in another case, the I / O amplification factor calculated by the statistics module can be the write amplification factor.

[0105] The specific parameter that the statistics module counts can be controlled by the management component. That is, the management component is used to configure the statistics module to control whether the statistics module counts the read / write amplification factor or the write amplification factor.

[0106] Optionally, as described in the above embodiments, when data is written to the foreground, the target thread can request a token from the read / write controller. Similarly, optionally, data can also request a token from the management component when it is processed in the background. Specifically, the refresh thread and the merge thread can also request a token from the management component before processing the target data, and only after obtaining the token can they begin refreshing and merging the data written to the memory layer corresponding to the target column family.

[0107] In this embodiment, the management component can configure the statistics module to perform statistical analysis on relevant parameters, enabling the database to achieve stable data writing as described in the above embodiments and improving the fluctuation of database write performance. Simultaneously, the management component can also control the background processing speed of the database by allocating tokens to the refresh and merge threads, thereby ensuring database write performance.

[0108] Optionally, Figure 1a The database shown may include a management component for configuring the statistics module to perform statistics on the read / write amplification factor or write amplification factor of the disk layer in the target storage space.

[0109] Optionally, such as Figure 2 As shown, the database may include a first thread pool, and multiple foreground write threads in the thread pool can form a thread queue. In response to a data write operation, the read / write controller can allocate the target data corresponding to the write operation to the target thread in the thread queue. Then, when the target thread is at the head of the queue, the read / write controller can allocate a period based on the speed, controlling the target thread to write the target data to the memory layer corresponding to the target column family at the target speed. The target thread being at the head of the queue indicates that the threads preceding the target thread in the thread queue are in a sleep state. Optionally, at any given time, one thread in the first thread pool is in a running state, while the remaining threads are not in a running state; the remaining threads can be in any of the following states: waiting, blocked, or sleeping.

[0110] In this embodiment, the setting of the first thread pool enables data to be written to the corresponding column families of the database in the order in which data write operations are triggered, thus ensuring the orderly writing of data.

[0111] In practice, the target data corresponding to a data write operation can be written to multiple column families in the database; that is, in the above embodiment, the target column families are multiple column families. It's easy to understand that the target data is not written to multiple column families separately, but rather the target data can be divided into several parts, and different parts of the target data can be written to different column families within multiple column families. For example... Figure 2 As shown, optionally, the database may also include a second thread pool, which may include multiple refresh threads and multiple merge threads to form different groups of refresh threads and merge threads. Different column families can use different groups of refresh threads and merge threads in the second thread pool to refresh and merge data in the memory layer.

[0112] In this embodiment, the setting of the second thread pool allows data written to different column families to be processed in the background by different groups of refresh threads and merge threads, thereby improving the situation where data written to multiple column families is congested in one group of refresh threads and merge threads.

[0113] When there are multiple target column families, and the read / write controller determines that any column family has entered a write stall or write suspension state, the read / write controller can limit the foreground write speed of that column family. At the same time, it can also limit the foreground write speed of the remaining column families. In other words, the write stall or write suspension state of one column family will affect the foreground write speed of other column families.

[0114] To mitigate the mutual influence between the states of these column families, one optional approach is for the statistics module to determine the target speed for any given column family based on the I / O amplification factor of the disk layer corresponding to that column family. Specific details can be found in the descriptions of the aforementioned embodiments and will not be repeated here. Following this method, the statistics module can obtain the target speeds for each of the multiple column families. Subsequently, the target thread can write data to the memory layers of different column families according to their respective target speeds. Furthermore, the refresh threads and merge threads in the second thread pool of the database can respectively refresh and merge the data in the memory layers corresponding to different column families into the last-level disk layer corresponding to each column family.

[0115] In this embodiment, the read / write controller can control the data to be written to the corresponding column family at a corresponding speed, and the speed corresponding to each column family is in line with the background processing speed of that column family. Therefore, the data can be written to different column families at a stable speed, thereby reducing the probability of write stalls or write hangs in different column families, which also improves the mutual influence between the states of different column families.

[0116] Another alternative, and simplified, approach involves the statistics module determining the target speed for any column family from among multiple column families based on the I / O amplification factor of the corresponding disk layer. Following this method, the statistics module obtains the target speeds for each of the multiple column families. Then, the statistics module identifies the column family with the largest disk layer I / O amplification factor as the reference column family, and further determines the target speed of this reference column family as the reference speed. The target thread then writes data to the memory layers of different column families according to this reference speed. Similarly, different groups of refresh and merge threads running in the database can refresh and merge the data in the memory layers of different column families to the last disk layer corresponding to each column family.

[0117] In this embodiment, the reference column family with the largest disk layer I / O amplification factor indicates that most of the target data is written to this reference column family. Therefore, the reference speed corresponding to this reference column family can be applied to the remaining column families, thereby ensuring that data can be stably written to different column families and guaranteeing the write performance of the database.

[0118] Furthermore, the databases provided in the above embodiments of this application can be deployed on server-side devices in a cloud computing environment. Users can submit data writing tasks to the database using client devices. Optionally, the aforementioned server-side device can be a cloud server maintained by a cloud service provider—referred to as a node. Client devices can be laptops, tablets, PCs, robots, etc.

[0119] In such Figure 3 The cloud computing environment shown may include several distributed deployments. Figure 3 The diagram illustrates compute nodes (201-1, 201-2, ...), and cache nodes. Each node can possess processing resources such as computing and storage. In a cloud computing environment, multiple nodes can be organized to provide a certain service, such as the database service mentioned in the embodiments of this application. Of course, a single node can also provide one or more services, such as... Figure 3The diagram illustrates services A, B, C, and D. In a cloud computing environment, services can be provided through external service interfaces, which client devices can call to use the corresponding services. Service interfaces can take the form of Software Development Kits (SDKs) and Application Programming Interfaces (APIs).

[0120] The services described above are deployed using various virtualization technologies supported by cloud computing environments, such as virtual machine-based and container-based virtualization technologies. Taking container-based virtualization technology as an example, several containers corresponding to a service can be assembled into a container group (pod). For example... Figure 3 The illustrated service B can be configured with one or more pods, and each pod can include a proxy and one or more containers. The one or more containers in the pod are used to handle requests related to one or more corresponding functions of the service, and the proxy in the pod is used to control network functions related to the service, such as routing and load balancing.

[0121] During operation, executing requests from client devices may require invoking one or more services in the cloud computing environment, and executing one or more functions of one service may require invoking one or more functions of another service. For example... Figure 6 As shown, after receiving a request from a client device, service A can call service B, and service B can request service D to perform one or more functions.

[0122] Based on the above introduction to the structure and working process of the database, Figure 4 This is a flowchart illustrating a data writing method in a database, as provided in an embodiment of this application. This method can be executed by the database software product provided in the above embodiments of this application. This method embodiment is related to... Figure 1a The database shown corresponds to this. For example... Figure 4 As shown, the method may include the following steps:

[0123] S101, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target storage space of the database, the target storage space including a memory layer and at least one disk layer.

[0124] S102, determine the target speed based on the I / O amplification factor of the disk layer in the target storage space.

[0125] S103, in response to the target data being written to the write-ahead log, writes the target data to the memory layer in the target storage space at the target speed.

[0126] S104, write the data in the memory layer to at least one disk layer in the target storage space.

[0127] Users can act as data producers to trigger data write operations to the database. In response to these operations, the database can allocate the target data to a target thread running within the database and determine the target storage space to receive the data. Simultaneously, the database can calculate the I / O amplification factor of the disk layer in this target storage space and determine a suitable foreground write speed—the target speed—based on this factor. Finally, the target data can be written to the memory layer of the target storage space at the target speed.

[0128] Although the method provided in this embodiment uses the database as the main execution body, more specifically, step S101 can be executed by the read / write controller in the database, step S102 can be executed by the statistics module in the database, step S103 can be executed by the target thread in the database, and step S104 can be executed by the background processing thread in the database.

[0129] In this embodiment, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target storage space of the database. This target storage space may include a memory layer and at least one disk layer. Then, a target speed can be determined based on the I / O amplification factor of the disk layer in the target storage space. After the target data is written to the write-ahead log, the target data is written to the memory layer of the target storage space according to the target speed, thus achieving foreground data writing. Finally, the data in the memory layer is written to at least one disk layer of the target storage space, thus achieving background data processing.

[0130] In the above method, the target data is written to the foreground at the target speed, meaning the target speed determines the foreground write speed. Furthermore, the target speed is derived from an indicator reflecting the background processing speed of the data, namely the disk-level I / O amplification factor. Therefore, using this method, the foreground write speed can match the background processing speed, significantly reducing the frequency of write stalls or write stops caused by excessively high foreground write speeds and insufficient background processing speeds. Data can be written to the database stably and continuously, thus improving write performance fluctuations during the data writing process.

[0131] Figure 5 A flowchart illustrating another database data writing method provided in this application embodiment. This method can be executed by the database provided in the above embodiments of this application. This method embodiment and... Figure 1b The database shown corresponds to this. For example... Figure 5As shown, the method may include the following steps:

[0132] S201, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target column family of the database, wherein the data is stored in the column family of the database in the form of a log structure merge tree, the log structure merge tree including a memory layer and at least one disk layer.

[0133] S202, determine the target speed based on the I / O amplification factor of the disk layer corresponding to the target column family.

[0134] S203, in response to the target data being written to the write-ahead log, writes the target data to the memory layer corresponding to the target column family according to the target speed.

[0135] S204, write the data in the memory layer to at least one disk layer corresponding to the target column family.

[0136] Users can act as data producers to trigger data write operations to the database. In response to these operations, the database can allocate the target data to a target thread running within the database and determine the target column family to receive it. Simultaneously, the database can calculate the disk layer I / O amplification factor corresponding to the target column family and determine a suitable foreground write speed—the target speed—based on this factor. Finally, the database can write the target data into the memory layer of the target column family at the target speed.

[0137] Although the method provided in this embodiment uses the database as the main execution body, more specifically, step S201 can be executed by the read / write controller in the database, step S202 can be executed by the statistics module in the database, step S203 can be executed by the target thread in the database, and step S204 can be executed by the background processing thread in the database.

[0138] In this embodiment, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target column family of the database. Then, the target speed allocated to the target data is determined according to the I / O amplification factor of the disk layer corresponding to the target column family, and the target data is written to the memory layer of the target column family according to the target speed, thus realizing the foreground writing of data.

[0139] In the above method, the target data is written to the front end at the target speed, meaning the target speed determines the front end write speed. Furthermore, the target speed and speed allocation period are derived from an indicator reflecting the back end processing speed, namely the disk-level I / O amplification factor. Therefore, using this method, the front end write speed can match the back end processing speed, significantly reducing the frequency of write stalls or write stops caused by excessively high front end write speeds and insufficient back end processing speeds. Data can be written to the column family stably and continuously, thus improving write performance fluctuations during the data writing process.

[0140] Furthermore, the contents not described in detail in this embodiment and the technical effects that can be achieved can be found in the relevant descriptions in the above embodiments, and will not be repeated here.

[0141] Optionally, the database may include a thread queue. In response to a data write operation, the target data corresponding to the data write operation can be allocated to the target thread in the thread queue. When the target thread is at the head of the queue, a period is allocated according to the speed, so that the target thread writes the target data to the memory layer corresponding to the target column family at the target speed.

[0142] In this embodiment, the thread queue setting allows data to be written to the corresponding column families in the database in the order in which data write operations are triggered, thus ensuring the orderly writing of data.

[0143] Figure 6 This is a flowchart illustrating another database data writing method provided in an embodiment of this application. This method can be executed by the database provided in the above embodiments of this application. Figure 6 As shown, the method may include the following steps:

[0144] S301, in response to a data write operation, the target data corresponding to the data write operation is allocated to the target column family of the database, wherein the data is stored in the column family of the database in the form of a log structure merge tree, the log structure merge tree including a memory layer and at least one disk layer.

[0145] S302 determines the target speed based on the I / O amplification factor of the disk layer corresponding to the target column family.

[0146] For details on the implementation of steps S301 and S302, please refer to [link / reference]. Figure 4 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0147] S303, assigns a token containing the target speed to the target data.

[0148] S304, in response to the receipt of the token and the writing of the target data to the write-ahead log, writes the target data to the memory layer corresponding to the target column family according to the target speed.

[0149] S305, write the data in the memory layer to at least one disk layer corresponding to the target column family.

[0150] In the above embodiments, the target speed determined by the statistics module can optionally be encapsulated in a token, and the corresponding speed allocation period is also the token allocation period. When the token allocation period arrives, the statistically determined target speed can be encapsulated in the token. After the target thread obtains the token, it can write the target data into the memory layer corresponding to the target column family according to the target speed contained in the token.

[0151] Furthermore, any content not described in detail in this embodiment, as well as the technical effects that can be achieved, can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0152] Optionally, Figure 6 The illustrated embodiment shows that the target thread can request a token to write data into the column family's memory layer. Similarly, optionally, data can also request a token while being processed in the background. Specifically, the refresh thread and the merge thread begin refreshing and merging the data written into the corresponding memory layer of the target column family only after obtaining the token.

[0153] Optionally, the target thread can request a token from the reading controller in the database, and the refresh thread and merge thread can request a token from the management component in the database.

[0154] In this embodiment, the management component can control the background processing speed of the database by allocating tokens to the refresh thread and the merge thread, thereby ensuring the write performance of the database.

[0155] for Figure 6 The target speed mentioned in the illustrated embodiment can be determined using one optional statistical method: based on the amount of target data and the I / O amplification factor of the disk layer of the target column family. Alternatively, a weighted parameter reflecting the write pressure of the target column family can be obtained, and the target speed can be determined jointly based on the amount of target data, the I / O amplification factor of the disk layer, and the preset weighted parameter.

[0156] For determining the I / O amplification factor of the disk layer used in the process of calculating the target speed, one optional method is to calculate the I / O amplification factor of the next-level disk layer after data is merged from the memory layer to the next-level disk layer corresponding to any disk layer of the target column family. Furthermore, the I / O amplification factor of the disk layer corresponding to the target column family can be determined based on the I / O amplification factors of the different levels of disk layers corresponding to the target column family.

[0157] Alternatively, if the merging thread is merging any file whose size is smaller than a first preset threshold, indicating that the merging thread is merging small files, then a reference file that was merged before this file can be obtained. This reference file can be at least one file, and the total size of the reference file and any other file must be higher than a second preset threshold, and the second preset threshold must not be less than the product of the first preset threshold and a preset ratio. Then, based on the respective file sizes of any file and the reference file, weight parameters inversely proportional to the file sizes are set for each file and the reference file, and the I / O amplification factor for the next level of disk is determined based on these weight parameters.

[0158] In this embodiment, for small files that are prone to causing excessive I / O amplification factor, the size of the small file can be smoothed by using at least one reference file with a larger size before the small file and its weight parameters, and the I / O amplification factor of the disk layer can be calculated based on the smoothed file size to improve the excessive I / O amplification factor caused by the small file size.

[0159] To further improve the accuracy of I / O amplification factor statistics, optionally, the task types of different merging tasks executed by the merging thread can be obtained, and the target merging task can be selected from different merging tasks according to the task type. Finally, the disk layer I / O amplification factor can be determined according to the number of target rewritten bytes rewritten by the merging thread during the execution of the target merging task.

[0160] In this embodiment, by filtering the merge task, rewritten bytes that are unrelated to the user's written data can be filtered out by the merge thread. Finally, the remaining rewritten bytes that are related to the user's written data can be counted to obtain the disk layer I / O amplification factor corresponding to the column family.

[0161] The preset weights used in the process of calculating the target speed can also be determined by using historical data of the target column family written before the target data.

[0162] In one optional manner, during the process of the refresh thread and the merge thread writing historical data prior to the target data to the disk layer corresponding to the target column family, the pressure indicators of the target column family can be monitored in real time, and a preset weight parameter can be determined based on the monitored pressure indicators. Optionally, if at least one of the pressure indicators does not meet the corresponding preset value, the preset weight parameter is determined to be a value less than 1; if at least one of the pressure indicators meets the corresponding preset value, the preset weight parameter is determined to be equal to 1.

[0163] In this embodiment, since the I / O amplification factor of the disk layer affects the background processing speed of the data, when the preset weight parameter is determined to be equal to 1, the target speed = the amount of target data * the I / O amplification factor of the disk layer. This target speed is matched with the background processing speed of the data. At this time, it can be ensured that the target data is stably and continuously written to the target column family.

[0164] In this embodiment, when the determined preset weight parameter is less than 1, the calculated target speed is slightly higher than the database's background processing speed. For example, if the preset weight parameter is 2 / 3, the target speed is 150% of the background processing speed, meaning the target speed exceeds the background processing speed. Since the target column family experiences less pressure at this time, the slightly higher target speed will not cause the column family to enter a write-limited or write-suspended state. Furthermore, the slightly higher target speed can provide users with a good data writing experience. In practice, it has also been found that when the foreground write speed is slightly higher than the background processing speed, the column family's I / O amplification factor actually decreases, thereby improving the database's write performance.

[0165] Optionally, in the above method embodiments, there is a process for statistically analyzing the I / O amplification factor at the disk layer. Optionally, in one case, the statistically analyzed I / O amplification factor can be the read-write amplification factor; in another case, the statistically analyzed I / O amplification factor can be the write amplification factor. Which parameter is specifically analyzed can be controlled by the database management component; that is, the management component is used to configure the statistics module to control whether the statistics module analyzes the read-write amplification factor or the write amplification factor.

[0166] In this embodiment, the management component can configure the statistics module so that the statistics module can perform statistics on the corresponding parameters, so that the database can achieve stable data writing as described in the above embodiments and improve the fluctuation of database write performance.

[0167] In addition, the method for determining the I / O amplification factor of the disk layer corresponding to the target column family described in the above embodiments can also determine the I / O amplification factor of the disk layer in the target storage space. It is only necessary to replace the target column family with the target storage space in the above embodiments; replace the memory layer of the target storage space with the memory layer of the target storage space with the memory layer of the target column family; and replace the disk layer of the target column family with the disk layer of the target storage space.

[0168] The above embodiments focus on describing the process of how target data is stably written to target column families, but do not limit the number of target column families. One scenario is that there may be multiple target column families, and the states of these column families are interdependent. To mitigate the mutual influence between the states of multiple column families, an optional approach is to obtain the target speed corresponding to each of the multiple column families. Then, the target thread can write data to the memory layers of different column families according to the target speeds corresponding to those column families. Furthermore, different groups of refresh threads and merge threads in the database can refresh and merge the data in the memory layers corresponding to different column families into the final disk layer corresponding to each column family.

[0169] In this embodiment, data can be written to the corresponding column family at a corresponding speed, and the speed corresponding to each column family is consistent with the background processing speed of that column family. Therefore, data can be written to different column families at a stable speed, thereby reducing the probability of write stalls or write hangs in different column families, which also improves the mutual influence between the states of different column families.

[0170] Another alternative, and simplified, approach involves determining the target speed for each of multiple column families. Then, the column family corresponding to the maximum disk layer I / O amplification factor is identified as a reference column family, and its target speed is further defined as the reference speed. The target thread can then write data to the memory layers of different column families according to this reference speed. Similarly, different groups of refresh and merge threads running in the database can refresh and merge the data in the memory layers of different column families to the final disk layer corresponding to each column family.

[0171] In this embodiment, the reference column family with the largest disk layer I / O amplification factor indicates that most of the target data is written to this reference column family. Therefore, the reference speed corresponding to this reference column family can be applied to the remaining column families, thereby ensuring that data can be stably written to different column families and guaranteeing the write performance of the database.

[0172] It should be noted that, although the executing entity of each step is specified in the description of the steps provided in the above embodiments, this application does not limit the steps in the above method embodiments to the same device.

[0173] That is, each step in the method embodiment can be executed by the same device, or the method can be executed by different devices. For example, the execution subject of steps S101 to S103 can be device A; or the execution subject of steps S101 and S102 can be device A, and the execution subject of step S103 can be device B; and so on.

[0174] Furthermore, in some processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or they may be executed in parallel. The operation numbers, such as S101, S102, etc., are merely used to distinguish different operations and do not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first" and "second" in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types.

[0175] In one possible design, the database data writing methods provided in the above embodiments can be applied to an electronic device. For example... Figure 7 As shown, the electronic device may include a processor 21 and a memory 22. The memory 22 is used to store data that supports the network component in performing the above-described operations. Figures 4-6 The program for writing data to a database provided in the illustrated embodiment, wherein the processor 21 is coupled to the memory 22 and is configured to execute the program stored in the memory 22.

[0176] The program includes one or more computer instructions, wherein when the one or more computer instructions are executed by the processor 21, they can perform the following steps:

[0177] In response to a data write operation, the target data corresponding to the data write operation is allocated to the target storage space of the database, the target storage space including a memory layer and at least one disk layer;

[0178] The target speed is determined based on the I / O amplification factor of the disk layer in the target storage space;

[0179] In response to the target data being written to the write-ahead log, the target data is written to the memory layer corresponding to the target column family at the target speed;

[0180] The data in the memory layer is written to at least one disk layer in the target storage space.

[0181] Optionally, the target data is organized in the form of column families in the target storage space and stored in the form of a log structure merge tree or a log structure merge graph; the target storage space corresponds to the target column family in the database; the log structure merge tree or log structure merge graph corresponding to the target column family includes a memory layer and at least one disk layer.

[0182] Optionally, when the processor 21 allocates the target data corresponding to the data write operation to the target storage space of the database, it is specifically used to: allocate the target data to the target column family;

[0183] When processor 21 writes the target data into the memory layer of the target storage space at the target speed, it is specifically configured to: write the target data into the memory layer corresponding to the target column family at the target speed.

[0184] When processor 21 writes data from the memory layer to at least one disk layer in the target storage space, it specifically performs the following: writes the data from the memory layer to at least one disk layer corresponding to the target column family. Optionally, the target speed is included in the token.

[0185] The processor 21 writes the target data into the memory layer of the target storage space at the target speed, specifically for: allocating tokens to the target data;

[0186] In response to the receipt of the token, the target data is written into the memory layer corresponding to the target column family at the target speed.

[0187] Optionally, when determining the target speed based on the I / O amplification factor of the disk layer corresponding to the target column family, the processor 21 is specifically configured to: determine the target speed based on the data volume of the target data and the I / O amplification factor of the disk layer.

[0188] Optionally, when determining the target speed based on the I / O amplification factor of the disk layer corresponding to the target column family, the processor 21 is specifically used to: obtain a preset weight parameter reflecting the write pressure of the target storage space;

[0189] The target speed is determined based on the amount of target data, the I / O amplification factor of the disk layer, and the preset weight parameters.

[0190] Optionally, the database runs a merge thread.

[0191] Processor 21 is further configured to: determine the I / O amplification factor of the next level disk layer in response to the merging thread writing historical data prior to the target data from any level disk layer corresponding to the target storage space to the next level disk layer of the target storage space;

[0192] The I / O amplification factor of the disk layer in the target storage space is determined based on the I / O amplification factor of each disk layer in the target storage space.

[0193] Optionally, the processor 21 is further configured to: if the file size of any file in any level of disk layer is less than a first preset threshold, then obtain a reference file in the historical data, wherein the total file size of the reference file and any file is higher than a second preset threshold, and the second preset threshold is not less than the product of the first preset threshold and a preset ratio;

[0194] Based on the file size of each of the files and the reference file, set weight parameters for each of the files that are inversely proportional to their file sizes;

[0195] The I / O amplification factor of the next-level disk layer is determined based on the weight parameters.

[0196] Optionally, the database runs a refresh thread and a merge thread.

[0197] Processor 21 is also configured to: monitor the pressure index of the target column family during the process of the refresh thread and the merge thread writing historical data prior to the target data into the target storage space disk layer;

[0198] Based on the pressure index of the target storage space, a preset weight parameter is determined.

[0199] Optionally, when the processor 21 determines the preset weight parameter based on the pressure index of the target column family, it is specifically used to: if the pressure index does not meet the preset value, determine that the preset weight parameter is a value less than 1;

[0200] If the pressure index meets the preset value, then the preset weight parameter is determined to be equal to 1.

[0201] The stress indicators include at least one of the following: the amount of data in the immutable memory table in the memory layer of the target storage space, the number of files in the first-level disk layer of the target storage space, and the target rewrite bytes of the merge thread in the database, wherein the target rewrite bytes include the number of bytes rewritten by the merge thread during the process of merging the historical data from the first-level disk layer of the target storage space to the last-level disk layer of the target storage space.

[0202] Optionally, the database is deployed with a merge thread.

[0203] Processor 21 is further configured to: obtain the task type of each of the different merging tasks executed by the merging thread;

[0204] According to the task type, a target merging task is selected from the different merging tasks. The merging thread executes the target merging task so that when the historical data fills any level of the target storage space, the data in any level of the disk is merged into the next level of the disk, until the historical data is merged into the last level of the target storage space.

[0205] The disk layer I / O amplification factor is determined based on the number of target rewrite bytes rewritten by the merging thread during the execution of the target merging task.

[0206] Optionally, the processor 21 is further configured to: in response to the data write operation, determine the target thread contained in the thread queue in the database;

[0207] If the target thread is at the head of the queue, then the target data is written to the memory layer corresponding to the target storage space using the target thread at the target speed.

[0208] Optionally, the target column family includes multiple column families.

[0209] When determining the target speed based on the I / O amplification factor of the disk layer corresponding to the target column family, the processor 21 is specifically used to: for any column family among the plurality of column families, determine the target speed corresponding to the any column family based on the I / O amplification factor of the disk layer corresponding to the any column family;

[0210] The step of writing the target data into the memory layer corresponding to the target column family according to the target speed includes:

[0211] Data is written to the memory layer of each of the multiple column families according to their respective target speeds.

[0212] Optionally, the target column family includes multiple column families.

[0213] When determining the target speed based on the I / O amplification factor of the disk layer corresponding to the target column family, the processor 21 is specifically used to: for any column family among the plurality of column families, determine the target speed corresponding to the any column family based on the I / O amplification factor of the disk layer corresponding to the any column family;

[0214] The column family corresponding to the maximum disk layer I / O amplification factor among the plurality of column families is determined as the reference column family;

[0215] The step of writing the target data into the memory layer corresponding to the target column family according to the target speed includes:

[0216] Data is written to the memory layer of each of the multiple column families according to the target speed corresponding to the reference column family.

[0217] Optionally, the I / O amplification factor includes a read-write amplification factor or a write amplification factor.

[0218] Processor 21 is configured to acquire the read / write amplification factor of the disk layer in the target column family, or to set the write amplification factor of the disk layer in the target column family.

[0219] Optionally, such as Figure 7 As shown, the structure of the electronic device may also include other components such as a communication component 23, a display 24, a power supply component 25, and an audio component 26.

[0220] Figure 7 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 7 The components shown. Additionally... Figure 7 The components mentioned are optional, not mandatory, and their specific selection depends on the product form of the electronic device. The electronic device in this application embodiment can be a conventional server, cloud server, or server array, or other server-side equipment.

[0221] The processor 21 can be any hardware processing device capable of executing the above-described method logic. Optionally, the processor can be a CPU, a graphics processing unit (GPU), or a microcontroller unit (MCU); it can also be a field-programmable gate array (FPGA), a programmable array logic (PAL), a general array logic (GAL), a complex programmable logic device (CPLD), or other programmable devices; or it can be an advanced RISC machine (ARM) or a system on chip (SoC), etc., but is not limited to these.

[0222] The aforementioned memory 22 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0223] The aforementioned communication component 23 is configured to facilitate wired or wireless communication between the device containing the communication component and other devices. The device containing the communication component can access wireless networks based on communication standards, such as Wi-Fi, 2G (e.g., Global System for Mobile Communications (GSM)), 3G (e.g., Wideband Code Division Multiple Access (WCDMA), 4G (e.g., Long Term Evolution (LTE)), 4G+ (e.g., LTE-Advanced (LTE-A)), or 5G (5th Generation Mobile Communication Technology), or combinations thereof. In one exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), Infrared Data Association (IRDA) technology, Ultra Wide Band (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0224] The aforementioned display 24 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0225] The aforementioned power supply component 25 provides power to various components of the device in which it resides. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which it resides.

[0226] The audio component 26 described above can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0227] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), SRAM, dynamic random access memory (DRAM), other types of random-access memory (RAM), ROM, EEPROM, EPROM, PROM, flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium.

[0228] Accordingly, this application also provides a computer program product, which includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to implement the steps in the above method embodiments. These computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device, so that the processor of the general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device can be implemented as a means to implement the corresponding functions in the above method embodiments.

[0229] Furthermore, the specific implementation form of the computer program product is not limited in the embodiments of this application. In some embodiments, the computer program product may be implemented as an application (APP), a mini-program, a PC client, a program module, a plug-in, an installation package, a software development kit (SDK), an image file of an optical disc (such as an ISO file), a plug-in, or software in the form of Software as a Service (SaaS), etc., but is not limited to these.

[0230] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data write-in method in a database, characterized by, The method comprises: in response to a data write operation, assigning target data corresponding to the data write operation to a target column family corresponding to a target storage space in a database, the target column family corresponding to a memory layer and at least one disk layer; determining a target speed according to an I / O amplification factor of the disk layer corresponding to the target column family; in response to the target data being written into a pre-write log, writing the target data into the memory layer corresponding to the target column family at the target speed; writing the target data into the at least one disk layer corresponding to the target column family.

2. The method of claim 1, wherein, The target data is organized in the form of a column family in the target storage space and stored in the form of a log-structured merge tree or a log-structured merge graph; The log-structured merge tree or the log-structured merge graph corresponding to the target column family respectively comprises a memory layer and at least one disk layer.

3. The method of claim 1, wherein, The target speed is contained in a token; The writing of the target data into the memory layer corresponding to the target column family at the target speed comprises: assigning the token to the target data; in response to the reception of the token, writing the target data into the memory layer corresponding to the target column family at the target speed.

4. The method of claim 1, wherein, The determination of the target speed according to the I / O amplification factor of the disk layer corresponding to the target column family comprises: obtaining a preset weight parameter reflecting the write pressure of the target column family; determining the target speed according to the data amount of the target data, the I / O amplification factor of the disk layer, and the preset weight parameter.

5. The method of claim 4, wherein, The database runs a merge thread; the method further comprises: in response to the historical data before the target data being written into a next-level disk layer from any level disk layer corresponding to the target column family by the merge thread, determining the I / O amplification factor of the next-level disk layer; determining the I / O amplification factor of the disk layer corresponding to the target column family according to the I / O amplification factors of different levels of disk layers corresponding to the target column family.

6. The method of claim 5, wherein, The method further comprises: if the file size of any file in the any level disk layer is less than a first preset threshold, obtaining a reference file in the historical data, the file total size of the reference file being higher than a second preset threshold, the second preset threshold being not less than the product of the first preset threshold and a preset proportion; setting a weight parameter inversely proportional to the file size for the any file and the reference file respectively according to the file sizes of the any file and the reference file respectively; determining the I / O amplification factor of the next-level disk layer according to the weight parameter.

7. The method of claim 4, wherein, The database runs a flush thread and a merge thread; the method further comprises: monitoring the pressure index of the target column family in the process of the historical data before the target data being written into the disk layer corresponding to the target column family by the flush thread and the merge thread; determining a preset weight parameter according to the pressure index of the target column family.

8. The method of claim 7, wherein, The determination of the preset weight parameter according to the pressure index of the target column family comprises: if the pressure index does not satisfy a preset value, determining the preset weight parameter as a value less than 1. If the pressure indicator meets a preset value, it is determined that the preset weight parameter is equal to 1.

9. The method of claim 7, wherein, The pressure indicator includes at least one of the following: a data amount of an immutable memory table in a memory layer corresponding to the target column family, a number of files in a first-level disk layer corresponding to the target column family, and a target rewriting byte amount of a merge thread in the database, the target rewriting byte amount including a number of bytes rewritten by the merge thread in the process of merging the historical data from the first-level disk layer corresponding to the target column family to a last-level disk layer corresponding to the target column family.

10. The method of claim 9, wherein, The database is deployed with a merge thread; the method further includes: acquiring respective task types of different merge tasks executed by the merge thread; according to the task types, screening a target merge task from the different merge tasks, and executing the target merge task by the merge thread to merge data in any level disk layer corresponding to the target column family to a next level disk layer of the any level disk layer when the historical data is full in the any level disk layer, until the historical data is merged to a last-level disk layer corresponding to the target column family; determining the target rewriting byte amount of the merge thread in the process of executing the target merge task according to the target rewriting byte amount.

11. The method of claim 1, wherein, The method further includes: in response to the data write operation, determining a target thread included in a thread queue in the database; writing the target data into the target column family at the target speed includes: if the target thread is at a head of the queue, writing the target data into a memory layer corresponding to the target column family by the target thread at the target speed.

12. The method of claim 1, wherein, The target column family includes a plurality of column families; determining the target speed according to the I / O amplification factor of the disk layer corresponding to the target column family includes: for any column family in the plurality of column families, determining a target speed corresponding to the any column family according to the I / O amplification factor of the disk layer corresponding to the any column family; writing the target data into the memory layer corresponding to the target column family at the target speed includes: writing data into the memory layer of each of the plurality of column families at a target speed corresponding to each of the plurality of column families.

13. The method of claim 1, wherein, The target column family includes a plurality of column families; determining the target speed according to the I / O amplification factor of the disk layer corresponding to the target column family includes: for any column family in the plurality of column families, determining a target speed corresponding to the any column family according to the I / O amplification factor of the disk layer corresponding to the any column family; determining a reference column family from the plurality of column families corresponding to a maximum disk layer I / O amplification factor; writing the target data into the memory layer corresponding to the target column family at the target speed includes: writing data into the memory layer of each of the plurality of column families at a target speed corresponding to the reference column family.

14. The method of claim 1, wherein, The I / O amplification factor includes a read-write amplification factor or a write amplification factor; the method further includes: configuring to acquire a read-write amplification factor of the disk layer corresponding to the target column family, or setting to acquire a write amplification factor of the disk layer corresponding to the target column family.

15. A database software product, characterized in that ​ A read-write controller and a statistics module; a target foreground write thread and a background processing thread run in the database; The read-write controller is configured to assign target data corresponding to a data write operation to a target column family corresponding to a target storage space in the database in response to the data write operation, the target column family corresponding to a memory layer and at least one disk layer; The target foreground write thread is controlled to write the target data to a pre-write log; The target foreground write thread is controlled to write the target data to the memory layer corresponding to the target column family at a target speed; The background processing thread is controlled to write data in the memory layer to at least one disk layer corresponding to the target column family; The statistics module is configured to count an I / O amplification factor of the disk layer corresponding to the target column family, and determine the target speed according to the I / O amplification factor of the disk layer.

16. An electronic device comprising: A memory and a processor; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the data write method in the database according to any one of claims 1 to 14.

17. A non-transitory machine-readable storage medium, comprising: The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by the processor of the electronic device, the processor executes the data write method in the database according to any one of claims 1 to 14.

18. A computer program product, characterised in that, The computer program product includes a computer program or instructions, which enable the computer program or instructions to implement the data write method in the database according to any one of claims 1 to 14.

Images