Data writing method, database system, electronic device and storage medium

Abstract

The present application provides a data writing method, a database system, an electronic device, and a storage medium. According to an embodiment of the present application, when creating a redo log (Redo Log), physical storage space is pre-allocated for the redo log, initial data is configured, and a mapping relationship is established. After a mapping relationship is determined between the redo log and the physical storage space, when the redo log is subsequently modified and updated, on the one hand, the file size will not be frequently modified, and on the other hand, there is no need to frequently establish a mapping relationship, thereby reducing the time overhead when writing the redo log, improving the writing speed of the redo log, and further optimizing the overall performance of the database, thereby reducing the problem of slow data writing speed affecting the access experience of the database system.

Description

Technical Field

[0001] The present application relates to the field of cloud computing technology, and in particular to a data writing method, a database system, an electronic device, and a storage medium. Background Art

[0002] With the continuous popularization and development of cloud computing applications, the demand for migrating and deploying traditional data systems to the cloud has increased, the application scope of cloud databases has gradually expanded, the demand for building database systems based on cloud storage has also increased accordingly, and the demand for optimizing cloud storage write speeds has also increased accordingly.

[0003] In traditional database systems, data writing is achieved by writing redo logs, that is, writing files. This process involves a large number of I / O (Input / Output) operations with the disk, which has a significant impact on the data writing speed and further affects the overall performance of the database.

[0004] Cloud databases typically require larger amounts of data to be written, resulting in a more pronounced impact on database performance due to slow write speeds. Therefore, optimizing redo log data writing is necessary to achieve efficient data writing and further improve the overall performance of cloud databases. Summary of the Invention

[0005] Embodiments of the present application provide a data writing method, a database system, an electronic device, and a storage medium to achieve efficient database data writing.

[0006] In a first aspect, an embodiment of the present application provides a data writing method, the method comprising:

[0007] Monitor data change operations in the memory buffer pool;

[0008] Rewriting the content of an initial redo log in a persistent storage medium to record the data change operation; wherein the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation;

[0009] According to the target redo log after content rewriting, the changed data indicated by the data change information is written into the persistent storage medium.

[0010] In a second aspect, an embodiment of the present application provides a database system comprising a computing node and a storage node, wherein the storage node comprises a persistent storage medium; the computing node comprises: a data change monitoring module for monitoring data change operations in a memory buffer pool; a content rewriting module for rewriting the content of an initial redo log in the persistent storage medium of the storage node to record the data change operation; wherein the initial redo log is pre-configured with initial data, and the content rewriting comprises replacing the initial data with data change information corresponding to the data change operation; a data writing module for writing the changed data indicated by the data change information into the storage node according to the target redo log after content rewriting.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above-described methods when executing the computer program.

[0012] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any of the methods described above.

[0013] Compared with the prior art, this application has the following advantages:

[0014] According to an embodiment of the present application, physical storage space is allocated in advance for the initial redo log (Redo Log), and initial data is configured, and a mapping relationship between the initial data and the physical storage space is established, so that the initial redo log has a certain file size and mapping relationship. When a data change operation of the memory buffer pool is monitored, the content of the initial redo log is rewritten, that is, the initial data in the initial redo log is replaced with data change information corresponding to the data change operation. Since content rewriting does not involve changing the file size of the redo log or establishing a mapping relationship, the time overhead of this process is small and the execution speed is fast. After the content of the initial redo log is rewritten, the changed data indicated by the data change information can be written to a persistent storage medium according to the target redo log after the content is rewritten to ensure the persistence of data storage.

[0015] Furthermore, when creating the initial redo log, you can use the physical storage space allocation function (such as the fallocate function in the Linux system) to quickly allocate physical storage space for the initial redo log. Combined with configuring appropriate data write frequency parameters, configure the initial data for the initial redo log. Since configuring the initial data is a background operation, excessive operation frequency will cause the background bandwidth to be occupied accordingly, thereby affecting the foreground bandwidth, resulting in task accumulation, and bringing a poor user experience to client users. By configuring appropriate data write frequency parameters, you can avoid the situation where foreground tasks are affected while maintaining rapid configuration of initial data, thereby further optimizing the overall performance of the database and reducing the poor user experience of client users.

[0016] The above description is only an overview of the technical solution of this application. In order to more clearly understand the technical means of this application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of this application more obvious and easy to understand, the specific implementation methods of this application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0018] Figure 1A This is a schematic diagram of the scenario of the data writing solution provided by this application;

[0019] Figure 1B This is another scenario diagram of the data writing solution provided by this application;

[0020] Figure 2 is a flow chart of a data writing method according to an embodiment of the present application;

[0021] Figure 3 This is a flowchart of a database performance testing method according to an embodiment of the present application;

[0022] Figure 4 This is a structural block diagram of a data writing device according to an embodiment of the present application;

[0023] Figure 5 This is a structural block diagram of a database performance testing method and apparatus according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of a database system according to an embodiment of the present application; and

[0025] Figure 7is a block diagram of an electronic device used to implement the embodiments of the present application. DETAILED DESCRIPTION

[0026] Hereinafter, only certain exemplary embodiments are briefly described. As will be appreciated by those skilled in the art, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and description are to be regarded as illustrative in nature and not restrictive.

[0027] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application.

[0028] When writing data, a database needs to write the data to a persistent storage medium through a persistent redo log. Traditional databases generate and persist redo logs by creating an empty redo log when the redo log is first written. Subsequent data updates are added to the redo log when data needs to be written. Writing data requires modifying the redo log's metadata, which means changing relevant information such as the redo log's file size and mapping relationships. Constantly modifying and updating file sizes and establishing mapping relationships incurs significant time overhead, so writing redo logs in this manner can significantly impact the write speed and, consequently, the overall performance of the database.

[0029] In view of this, an embodiment of the present application provides a new data writing solution to fully or partially solve the above technical problems.

[0030] In order to more clearly demonstrate the data writing solution provided in the embodiments of the present application, a specific application example of the solution in the embodiments of the present application is given below. Figure 1A and Figure 1B The following are schematic diagrams of two exemplary application scenarios for implementing the method of the embodiments of the present application. Figure 1A and Figure 1B The data writing solution involves modifying the data pages in the memory buffer pool corresponding to the database, writing them to the redo log, and finally storing the changed data in the persistent medium. Figure 1AAs shown, when a data change operation is detected in the memory cache pool of a compute node, the data change information corresponding to the data change operation needs to be written to the initial redo log. If the existing redo log has no free space, an initial redo log can be created first. This creation method involves creating a new initial redo log, pre-allocating physical storage space for the initial redo log in the persistent storage medium of the storage node, configuring initial data (for example, configuring 0 as the initial data), and establishing a mapping relationship between the initial data and the physical storage space in the persistent storage medium. The initial redo log is then rewritten based on the data change operation. It is understood that if a reusable redo log exists, that is, if the existing redo log has free space, for example, if the redo log contains initial data that has not been rewritten, or if the redo log does not contain initial data but still has free space, there is no need to create an initial redo log. Instead, the content of the reusable redo log can be directly rewritten, or physical storage space can be pre-allocated to configure the initial data, and the corresponding mapping relationship can be established before rewriting the initial data.

[0031] like Figure 1B The figure shows a specific method for rewriting the redo log content, that is, replacing the initial data with the data change information corresponding to the above-mentioned data change operation. Because physical storage space has been allocated for the redo log before the content is rewritten, and the initial data stored in the redo log has established a corresponding mapping relationship with the persistent storage medium, when rewriting the content, only the content needs to be replaced, and there is no need to frequently change the file size of the redo log and the mapping relationship between it and the physical storage space. Therefore, compared with the traditional data writing solution, the data writing speed of this solution is faster, and the data system performance of the data system using this solution is better, thereby reducing the problem of slow data writing speed affecting the access experience of the database system.

[0032] The embodiments of the present application can be applied to traditional local database systems, such as the relational database system MySQL, the object-relational database system PostgreSQL, etc., and can also be applied to database systems built based on cloud storage, for example, a relational database system based on cloud storage, a non-relational Key-Value database system based on cloud storage, etc.

[0033] Various embodiments of the present application are introduced as follows.

[0034] The embodiment of the present application provides a data writing method 200, such as Figure 2 The flowchart of the data writing method according to one embodiment of the present application may include:

[0035] Step S201: monitoring data change operations in the memory buffer pool.

[0036] When writing data, the first step is to obtain the data to be written. When writing data according to user instructions, the database system first performs the corresponding data modification operations in the memory buffer pool (Redo Buffer). In other words, data modification operations correspond to client user instructions and indicate data changes. These user instructions may be user-entered instructions corresponding to Data Manipulation Language (DML), such as at least one of INSERT (for data insertion), UPDATE (for data update), and DELETE (for data deletion).

[0037] In one possible scenario, the embodiments of the present application can be applied to a MySQL database system. The data storage structure used by MySQL is BTree, which is a self-balancing generalized binary tree suitable for storage systems with large I / O operations. By using the BTree structure, MySQL can complete operations such as finding data, sequential access, inserting data, and deleting data in a short time. When writing data, for example, when a client user updates a piece of data in the database through an UPDATE instruction, MySQL does not directly modify the data stored in the BTree, but first copies the data page involved to the memory buffer pool, and changes the data on the data page in the memory buffer pool, that is, performs a data change operation.

[0038] Since the memory buffer pool is an area on the memory and the data in the memory is volatile, when a data change operation in the memory buffer pool is detected, subsequent steps S202 and S203 need to be performed to synchronize the content corresponding to the data change operation in the memory buffer pool to a persistent storage medium (such as a disk) to ensure the persistence of data storage.

[0039] After the data change operation in the memory buffer pool is monitored, the process of persisting the changed data includes recording the data change information corresponding to the data change operation by using the redo log in the persistent storage medium, and modifying the corresponding data in the database according to the content in the redo log. Since the data change operation on the in-memory buffer pool is also a memory write operation, writing the redo log is a file write operation, wherein the memory write operation is very fast, and the file write operation involves performing I / O operations with the persistent storage medium. The time overhead brought by this process is very large, which will affect the speed of data writing. Therefore, step S202 mainly achieves the effect of accelerating the data write speed by optimizing the method of writing the redo log. The following is an explanation of step S202.

[0040] Step S202 , rewriting the content of the initial redo log in the persistent storage medium to record the data change operation; wherein the initial redo log is pre-configured with initial data, and rewriting the content includes replacing the initial data with data change information corresponding to the data change operation.

[0041] A redo log is a file on persistent storage media that records data change information corresponding to data modification operations in the memory buffer pool. This data change information describes the changes to the data in the database corresponding to the data modification operations. For example, this data change information might include information such as the number of bytes modified at a certain offset on a data page in a tablespace in the database, as well as the modified value. Based on this data change information, the corresponding data stored in the database's persistent storage media can be modified.

[0042] The initial redo log is a file pre-configured with initial data after the redo log is generated. The data type of the initial data can be an integer, a string, or other data types, and this application does not impose any restrictions on this. The initial data itself may not have substantive meaning. For example, the initial data can be 0, and the number of 0s can be one or more, and the specific number can also be determined by a pre-configured method. The initial redo log pre-configured with initial data also has corresponding physical storage space configured on the persistent storage medium. When a data change operation in the memory buffer pool is detected, the data change operation is recorded by rewriting the initial redo log content. Rewriting the content means replacing the initial data with the data change information corresponding to the data change operation. By replacing the initial data in the initial redo log with substantive data change information, the data change operation is recorded in the redo log. Since the content rewriting process does not involve reallocating physical storage space for the redo log, it saves the time overhead of writing data while allocating physical storage space for the redo log on the persistent storage medium, thereby improving the data writing speed and optimizing the data writing method.

[0043] After recording the data change information in the initial redo log, step S203 is performed to write the changed data to the persistent storage medium, thereby completing data persistence. The following is an explanation of step S203.

[0044] Step S203 : writing the changed data indicated by the data change information into the persistent storage medium according to the target redo log after content rewriting.

[0045] The initial redo log after content rewriting is recorded as the target redo log. The target redo log records data change information corresponding to the data change operation in the memory cache pool. The changed data indicated by the data change information recorded in the target redo log is written to a persistent storage medium to achieve persistent storage of the changed data. The persistent storage medium is a storage medium corresponding to the database and storing the data in the database.

[0046] In an optional embodiment, before rewriting the content of the initial redo log in the persistent storage medium, when monitoring the data change operation of the memory buffer pool, the initial redo log can also be created in the persistent storage medium first, and the corresponding physical storage space can be allocated for the initial redo log in the persistent storage medium. In other words, the physical storage space required to store the initial redo log can be allocated for the initial redo log on the persistent storage medium (such as a disk) used by the database system. By pre-allocating physical storage space for the newly created initial redo log, the storage area occupied by the initial redo log on the persistent storage medium can also be made as continuous as possible, which can reduce the seek overhead for subsequent writing and / or reading the initial redo log; on the other hand, it can prevent the situation where the required physical storage space is insufficient when writing the redo log, thereby affecting the writing speed.

[0047] When the initial redo log is divided into data blocks, an initial redo log can be divided into one or more data blocks. After the initial redo log is created in the persistent storage medium and the corresponding physical storage space is allocated to the newly created redo log, initial data can be configured for at least some of the data blocks in the initial redo log. Then, a mapping relationship can be created between the data blocks with initial data and the physical storage space in the persistent storage medium. In other words, the initial data can be pre-configured for all the data blocks in the initial redo log, or it can be pre-configured for only some of the data blocks. Thus, at least some of the data blocks in the initial redo log are pre-configured with initial data, and a mapping relationship is established between the above-mentioned data blocks and the physical storage space.

[0048] It should be noted that the above-mentioned step of creating the initial redo log can be executed after the set conditions are triggered, and whether to trigger the generation of the initial redo log can be determined based on the task load status of the computing node. By setting the trigger conditions, the occupation of computing resources by premature generation of the initial redo log can be avoided. Among them, the task load status can be one or more data such as the bandwidth of the computing node, the CPU load ratio, etc. The trigger condition determined according to the task load status can be when one or more data of the task load status exceeds the corresponding threshold, or the data exceeding the corresponding threshold exceeds the set number, etc., which can be set according to actual needs.

[0049] The above-mentioned step of allocating physical storage space corresponding to the initial redo log can be performed immediately after the initial redo log is created, or it can be performed after a certain period of time. Similarly, the above-mentioned step of configuring initial data and generating mapping relationships for at least some data blocks in the initial redo log can be performed immediately after the step of creating the initial redo log and allocating physical storage space corresponding to the initial redo log, or it can be performed after a certain period of time. The design of performing the steps after a certain period of time can avoid the premature occupation of computing resources by configuring initial data and generating mapping relationships.

[0050] In actual processing, the task load status of the computing node can be monitored. If the current task load is heavy, or in order to avoid the impact of allocating physical storage space, or configuring initial data and generating mapping relationships on the task load status, the processing can be delayed for a period of time; if the current load is light, the corresponding physical storage space can be allocated to the initial redo log directly after the initial redo log is created, or the processing of configuring initial data and generating mapping relationships can be executed after the physical storage space is allocated.

[0051] Of course, other conditions can also be set to trigger the above processing, for example, the interval time that will not increase the load can be determined based on prior monitoring, or the interval time can be determined in advance based on actual experience. The interval time can also be adjusted according to the impact on the load during use. This application does not impose any restrictions on this.

[0052] The initial data in the aforementioned data block may not be of substantive significance. In subsequent steps, the initial data in the data block can be replaced with substantive data change information through content rewriting. A data block can store one or more pieces of data change information depending on the physical storage space available. For an initial redo log that pre-configures initial data for only some data blocks, data change information can be recorded by replacing the initial data, directly writing the data change information to the initial redo log, or a combination of the above two methods.

[0053] In an optional embodiment, allocating corresponding physical storage space for the initial redo log in the persistent storage medium may include calling a physical storage space allocation function through the operating system of the device where the memory buffer pool is located, and allocating processing storage space for the initial redo log in the persistent storage medium. The physical storage space allocation function may include a function implemented by the file system layer in the operating system of the device where the memory buffer pool is located. By calling the physical storage space allocation function, physical storage space can be quickly reserved for the file in the persistent storage medium corresponding to the operating system. Therefore, the physical storage space allocation function can be used to quickly allocate corresponding physical storage space for the initial redo log. For example, in a Linux system, the physical storage space function that can be used is the fallocate function. By calling the fallocate function, physical storage space on the disk can be allocated for the initial redo log in the form of reserved disk space.

[0054] In order to more quickly configure initial data for at least some of the log blocks in the initial redo log, in an optional embodiment, a data write frequency parameter can be set. When configuring initial data for at least some of the data blocks in the initial redo log, the initial data can be configured according to the data write frequency parameter. For example, when using a zero filling operation to configure zeros as initial data for at least some of the data blocks in the initial redo log, a data write frequency parameter can be set for the zero filling operation to control the frequency with which zeros are written during the zero filling operation. The data write frequency is the frequency with which initial data is written to the initial file. The higher the frequency, the more frequent the write operations within the same period of time, and the greater the file space occupied by the initial redo log due to the initial data written. Therefore, by setting the data write frequency parameter, the file space occupied by the initial redo log can also be controlled. However, since allocating initial data is a background task, setting an excessively high data write frequency parameter will correspondingly increase the bandwidth occupied by the background task, thereby causing a backlog in the foreground task and impacting the normal operation of the foreground task. Therefore, by setting a reasonable data write frequency parameter, while maintaining write efficiency, the problem of foreground task backlogs caused by excessive background task bandwidth, which in turn affects the access experience of the database system, can be reduced.

[0055] In an optional embodiment, the initial data pre-configured in the initial redo log corresponds to multiple data blocks. When rewriting the content of the initial redo log in the persistent storage medium, the target data block for storing the data change information can be determined first. The data block whose content is about to be rewritten can be recorded as the target data block. The target data block can be the data block with the initial data configured in the initial redo log, that is, after the initial redo log is created and the initial data is allocated to the data blocks in the initial redo log, the data block whose content has not been rewritten. When determining the target data block, the first data block in the initial redo log that has not been rewritten can be selected in spatial order. After determining the target data block, the initial data in the target data block can be replaced with the data change information corresponding to the data change operation to complete the rewriting of the content of the target data block.

[0056] In an optional embodiment, when rewriting the content of an initial redo log in a persistent storage medium, if the amount of data change information exceeds the amount of initial data, at least one new data block is allocated to store the excess change data. Rewriting the content of the initial redo log requires replacing the pre-allocated initial data in the data blocks of the initial redo log with the data change information. When performing this step, after rewriting the content of a data block, it is possible that the data change information has not yet been recorded in the initial redo log, i.e., the amount of data change information exceeds the amount of initial data. In this case, a new data block can be allocated for the data change information not yet recorded in the initial redo log to store the excess change data information. The new data block can be a data block in the initial redo log that has not yet been rewritten, a data block in the initial redo log that has not been allocated any data, a data block in the initial redo log and / or other redo logs that can be used to store data, or a combination of these data blocks, and this application is not limited thereto.

[0057] Corresponding metadata is configured for the redo log. The metadata may include a mapping relationship between the data change information in the redo log and the storage space in the persistent storage medium. The metadata may also include one or more information related to the redo log, such as the data structure and file size of the redo log. When the file size of the redo log changes, and / or the mapping relationship between the data blocks in the redo log and the storage space in the persistent storage medium changes, the metadata corresponding to the redo log may be changed accordingly. For some newly added data blocks, that is, data blocks in the redo log that have not been pre-allocated physical storage space and / or have not established a mapping relationship with the physical storage space, when they are used to store data change information, the file space of the initial redo log becomes larger accordingly, and a corresponding mapping relationship is also established between the newly added data blocks and the physical storage space in the persistent storage medium. Therefore, when the above situation occurs, the metadata corresponding to the initial redo log may be changed accordingly.

[0058] In one possible scenario, in order not to affect the foreground task, when pre-configuring the initial data for the initial redo log, a smaller data write frequency parameter is used, and the initial data is only configured for some data blocks of the initial redo log, resulting in a smaller amount of initial data. After monitoring the data change operation in the memory buffer pool, when rewriting the content of the initial redo log, all the existing initial data is replaced with the data change information, and a situation occurs where the data change information is not written, that is, the amount of data change information exceeds the amount of the initial data. At this time, a new data block can be added to the initial redo log, physical storage space in the persistent storage medium can be allocated for the new data block, and the remaining data change information can be written, and a mapping relationship between the physical storage space and the data change information written to the new data block can be established, and the metadata corresponding to the above-mentioned initial redo log can be changed.

[0059] In an optional embodiment, before creating the initial redo log in the persistent storage medium, it may be determined that there is no available free space in the existing redo log. That is, after monitoring the data change operation of the memory buffer pool, when it is necessary to write the data change information corresponding to the data change operation into the redo log, it may be determined that there is no available free space in the existing redo log, and then a new initial redo log may be created to complete the subsequent steps and write the data change information into the initial redo log. In the case that there is still available free space in at least one existing redo log, the data change information to be written may be written into the at least one existing redo log. The case where there is available free space in the redo log may include one or more of the following: initial data whose content has not been rewritten is stored in the data block in the redo log, and the data block in the redo log is not allocated initial data but is in an idle state.

[0060] In one possible scenario, in the operating mode of cloud gaming (a gaming method based on cloud computing), there may be multiple scenarios where the number of user visits increases suddenly, for example, users flock to the game during the promotion period, users come online in large numbers after the game is shut down for maintenance, and users return to the game in large numbers due to important activities. Usually in this scenario, since the performance of traditional databases cannot support the rapid writing of large amounts of data, the game will become stuck or even crash, resulting in a poor gaming experience. By applying the data writing method provided in the embodiment of the present application in the cloud storage database deployed corresponding to the cloud game, the database performance can be improved, and while speeding up the database background writing speed, the normal operation of the foreground can be ensured, thereby protecting the user's gaming quality.

[0061] After optimizing the database data writing method, it is necessary to use a corresponding performance testing method to test the performance of the database using the data writing method provided by this application to verify the effectiveness of the optimization method. Therefore, this application also provides a database performance testing method. The following is an introduction to this database performance testing method.

[0062] This embodiment of the application provides a database performance testing method 300. Figure 3 300 is a flowchart of a database performance testing method according to an embodiment of the present application. The method 300 may include:

[0063] Step S301: monitoring data change operations in the memory buffer pool of the database.

[0064] Step S302, rewriting the content of the initial redo log in the persistent storage medium to record the data change operation; wherein, the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation; the target redo log is used to write the changed data indicated by the data change information to the persistent storage medium.

[0065] It can be understood that steps S301 and S302 are relevant steps of the data writing method provided in the embodiment of the present application.

[0066] Step S303: Count the time consumed in rewriting the initial redo log, and use it as a basis for determining the performance statistical index of the database.

[0067] The size of the time overhead is the main factor affecting the performance quality. Therefore, when testing the quality of multiple methods, the time it takes for each method to complete the same transaction can be counted. By comparing the time used by each method, the time overhead is smaller, that is, the method that completes the same transaction more efficiently is the better method. When testing the performance of a database to which a new method is applied, it is necessary to count the time spent executing the new method as a basis for determining the performance statistics of the database. In an embodiment of the present application, the time spent on rewriting the initial redo log, that is, the time overhead of rewriting the initial redo log, is used as a basis for determining the performance statistics of the database to which the method 200 of the embodiment of the present application is applied.

[0068] In a possible scenario, when the amount of data written is the same, the time spent writing the redo log using the existing technology and the time spent rewriting the redo log using the embodiment of the present application can be counted separately. By comparing the lengths of the two times, the advantages and disadvantages of the method can be judged.

[0069] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a data writing device. Figure 4 FIG. 4 is a block diagram of a data writing device 400 according to an embodiment of the present application. The data writing device may include:

[0070] The data change monitoring module 401 is used to monitor data change operations in the memory buffer pool.

[0071] The content rewriting module 402 is used to rewrite the content of the initial redo log in the persistent storage medium to record the data change operation; wherein the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation.

[0072] The data writing module 403 is configured to write the changed data indicated by the data change information into the storage node according to the target redo log after content rewriting.

[0073] In an optional embodiment, the apparatus 400 may further include: a redo log creation module, configured to create an initial redo log in a persistent storage medium.

[0074] In an optional embodiment, the redo log creation module may include:

[0075] The space allocation submodule is used to allocate corresponding physical storage space for the initial redo log in the persistent storage medium.

[0076] The data configuration submodule is used to configure initial data for at least part of the data blocks in the log file and generate a mapping relationship between the data blocks with the initial data and the physical storage space in the persistent storage medium.

[0077] In an optional embodiment, the space allocation submodule is specifically configured to call a physical space allocation function through the operating system of the device where the memory buffer pool is located, and allocate corresponding physical storage space for the initial redo log in the persistent storage medium.

[0078] In an optional embodiment, the data configuration submodule is specifically configured to configure initial data for at least part of the data blocks in the log file according to a data writing frequency parameter.

[0079] In an optional embodiment, the content rewriting module 402 may include:

[0080] The target data block determination submodule is used to determine the target data block for storing the data change information.

[0081] The value replacement submodule is used to replace the value of the target data block with the value of the data change information corresponding to the data change operation.

[0082] In an optional embodiment, the device may further include:

[0083] A log creation trigger module is used to monitor the task load status of the computing node and determine the trigger to create the initial redo log according to the task load status

[0084] In an optional embodiment, the content rewriting module 402 may further include a data block allocation submodule for allocating at least one new data block for storing the excess changed data when the data size of the data change information exceeds the data size of the initial data.

[0085] In an optional embodiment, corresponding metadata is configured for the redo log, and the metadata includes a mapping relationship between the data change information of the redo log and the storage space in the persistent storage medium. The device 400 may also include a metadata change module for changing the mapping relationship between the data change information in the metadata and the storage space in the persistent storage medium while storing the excess data change information in the newly added data block.

[0086] In an optional embodiment, the apparatus 400 may further include a free space determination module configured to determine whether there is any available free space in the existing redo logs before creating the initial redo log in the persistent storage medium.

[0087] Corresponding to the application scenario and method of the method provided in the embodiment of the present application, the embodiment of the present application also provides a database performance testing method and device. Figure 5 The diagram shows a block diagram of a database performance testing method and apparatus 500 according to an embodiment of the present application, which may include:

[0088] The data change monitoring module 501 is used to monitor data change operations in the memory buffer pool of the database.

[0089] The content rewriting module 502 is used to rewrite the content of the initial redo log in the persistent storage medium to record the data change operation; wherein, the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation; the target redo log is used to write the changed data indicated by the data change information to the persistent storage medium.

[0090] The time statistics module 503 is used to count the time consumed in the initial redo log rewriting, which serves as a basis for determining the performance statistics index of the database.

[0091] See also Figure 6 The present embodiment further provides a database system 600, including a computing node 601 and a storage node 602, wherein the storage node includes a persistent storage medium. The computing node monitors data change operations in a memory buffer pool, a redo buffer, rewrites the contents of an initial redo log pre-created in the persistent storage medium of the storage node, and further writes the changed data corresponding to the data change operation to a data page of the storage node based on a target redo log obtained after the content rewrite.

[0092] According to the above execution process, the computing node can be used to monitor the data change operation of the memory buffer pool; rewrite the content of the initial redo log of the storage node to record the data change operation; wherein, the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation; and is used to write the changed data indicated by the data change information into the storage node according to the target redo log after the content is rewritten.

[0093] The functions of each module in each device in the embodiment of the present application can be referred to the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0094] Figure 7 FIG. 1 is a block diagram of an electronic device for implementing an embodiment of the present application. Figure 7As shown, the electronic device includes: a memory 701 and a processor 702. The memory 701 stores a computer program that can be run on the processor 702. When the processor 702 executes the computer program, the method in the above embodiment is implemented. The number of the memory 701 and the processor 702 can be one or more.

[0095] The electronic device also includes:

[0096] The communication interface 703 is used to communicate with external devices and perform data exchange transmission.

[0097] If the memory 701, processor 702, and communication interface 703 are implemented independently, the memory 701, processor 702, and communication interface 703 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0098] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0099] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0100] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided in the embodiment of the present application.

[0101] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory. The input interface, the output interface, the processor and the memory are connected through an internal connection path. The processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0102] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor that supports the Advanced RISC Machines (ARM) architecture.

[0103] Furthermore, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM) and direct memory bus random access memory (DR RAM).

[0104] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another.

[0105] In the description of this specification, the reference terms "one embodiment," "some embodiments," "example," "specific example," or "some examples" mean that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials, or characteristics described may be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification, as well as features of different embodiments or examples, unless they are mutually inconsistent.

[0106] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. Throughout the description of this application, "plurality" means two or more, unless otherwise specifically defined.

[0107] Any process or method described in the flowchart or otherwise described herein can be understood to represent a module, segment or portion of code comprising one or more executable instructions for implementing the steps of a specific logical function or process. The scope of the preferred embodiments of the present application includes other implementations in which the functions may be performed in a different order than shown or discussed, including performing the functions substantially simultaneously or in reverse order depending on the functions involved.

[0108] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, apparatus or device (such as a computer-based system, a system including a processor or other system that can fetch instructions from an instruction execution system, apparatus or device and execute instructions), or used in combination with such instruction execution systems, apparatuses or devices.

[0109] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0110] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing module, or each unit may exist physically separately, or two or more units may be integrated into a single module. The aforementioned integrated modules may be implemented in the form of hardware or in the form of software functional modules. If the aforementioned integrated modules are implemented in the form of software functional modules and sold or used as independent products, they may also be stored in a computer-readable storage medium. The storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc.

[0111] The above is merely an exemplary embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various modifications or substitutions within the technical scope described in this application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A data writing method, characterized in that: include: Monitor data change operations in the memory buffer pool; Determining that no available free space exists in an existing redo log, creating an initial redo log in a persistent storage medium, and allocating corresponding physical storage space for the initial redo log in the persistent storage medium; Configuring initial data for at least part of the data blocks in the initial redo log, and generating a mapping relationship between the data blocks with the initial data and the physical storage space in the persistent storage medium; Rewriting the content of an initial redo log in a persistent storage medium to record the data change operation; wherein the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation; According to the target redo log after content rewriting, the changed data indicated by the data change information is written into the persistent storage medium.

2. According to the method of claim 1, allocating corresponding physical storage space for the initial redo log in the persistent storage medium comprises: The operating system of the device where the memory buffer pool is located calls a physical storage space allocation function to allocate corresponding physical storage space for the initial redo log in the persistent storage medium.

3. The method according to claim 1, wherein configuring initial data for at least part of the data blocks in the initial redo log comprises: Initial data is configured for at least part of the data blocks in the initial redo log according to the data write frequency parameter.

4. The method according to claim 1, wherein The initial data corresponds to a plurality of data blocks, and rewriting the content of the initial redo log in the persistent storage medium includes: determining a target data block for storing the data change information; The value of the target data block is replaced by the value of the data change information corresponding to the data change operation.

5. The method according to claim 4, characterized in that The rewriting of the content of the initial redo log in the persistent storage medium further includes: If the data size of the data change information exceeds the data size of the initial data, at least one newly added data block is allocated to store the excess change data.

6. The method according to claim 1, characterized in that Corresponding metadata is configured for the redo log, and the metadata includes a mapping relationship between data change information of the redo log and storage space in the persistent storage medium. The method further includes: While storing the excess data change information in the newly added data block, the mapping relationship between the data change information in the metadata and the physical storage space in the persistent storage medium is changed.

7. The method according to claim 1, characterized in that The method is applied to a computing node, and further comprises: Monitor the task load status of the computing node, and determine triggering creation of an initial redo log based on the task load status.

8. A database system, characterized in that: Includes computing nodes and storage nodes; The computing node: Used to monitor data change operations in the memory buffer pool; for determining that no available free space exists in the existing redo logs, creating an initial redo log in a storage node, and allocating corresponding physical storage space for the initial redo log in the storage node; Configuring initial data for at least part of the data blocks in the initial redo log, and generating a mapping relationship between the data blocks with the initial data and the physical storage space in the storage node; rewriting the content of the initial redo log of the storage node to record the data change operation; wherein the initial redo log is pre-configured with initial data, and the content rewriting includes replacing the initial data with data change information corresponding to the data change operation; The method is used to write the changed data indicated by the data change information into the storage node according to the target redo log after content rewriting.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images