Data writing method and server
By configuring persistent memory to direct access mode and using low-latency nonvolatile storage, writing to write write-pre-logs directly, the problem of write-pre-logs is solved, and the reliability and performance of the system is improved.
Patent Information
- Application Number
- CN202411220713.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2025-01-10
AI Technical Summary
In the prior art, cache is used as buffer before writing write-preparation logs to persistent storage media, resulting in loss of write-preparation logs in abnormal situations, affecting the reliability of the system.
Configure persistent memory in direct access mode and enable direct access parameters, allowing applications to write write-pre-logs directly to persistent memory, skip kernel space, and store write-pre-logs using low-latency and non-volatile persistent memory. Establish a mapping relationship between the virtual address area and the physical address area, write to the write-pre-log in sequential writing, and ensure the atomicity and durability of transactions by redoing the log and revoking the log when an exception occurs.
It effectively avoids the loss of write-pre-log, improves the efficiency of writing write-pre-logs and the overall performance and reliability of the system, and ensures the atomicity and durability of data writing.
Smart Images
Figure CN119292515A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the technical field of computing devices, and in particular, to a data writing method and a server. Background Art
[0002] WAL (Write-Ahead Logging) technology is a common database transaction log technology used to ensure the persistence and recovery capability of the database. The core idea of WAL technology is to write the corresponding WAL log (pre-write log) into a persistent log file before making any changes to the database.
[0003] In the related art, in order to improve the performance of writing the write-ahead log, a cache is often used as a buffer before writing the write-ahead log to a persistent storage medium. However, when an exception occurs, the write-ahead log in the cache will be lost, affecting the reliability of the system. Summary of the invention
[0004] The embodiments of the present application provide a data writing method and a server, which can avoid the loss of pre-write logs and improve the reliability of the system.
[0005] In a first aspect, an embodiment of the present application provides a data writing method, comprising:
[0006] On the operating system of the server, configuring the persistent memory to a direct access mode; mounting the persistent memory to a file system of the server and enabling direct access parameters to support direct access to the persistent memory;
[0007] Obtain a data write request, and write a pre-write log into the persistent memory according to the data write request; wherein the data write request is used to request to write the data to be written;
[0008] In response to a data write request, the data to be written is written into the database.
[0009] In the above solution, the persistent memory is configured as direct access mode, and the direct access parameters of the file system on which the persistent memory is mounted are enabled, so that the server's application can directly read and write the persistent memory, thereby directly writing the write-ahead log to the persistent memory. After obtaining the data write request, on the one hand, the server's kernel space is skipped, and the server's application directly writes the write-ahead log to the server hardware. On the other hand, low-latency and non-volatile persistent memory is used to store the write-ahead log, which can effectively improve the efficiency of writing the write-ahead log, and the write-ahead log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server.
[0010] In a possible implementation, the method further includes:
[0011] Establishing a mapping relationship between a virtual address area in a user process address space of the server and a physical address area of the persistent memory;
[0012] The mode of writing the write-ahead log is set to directly write into the persistent memory; and the path of writing the write-ahead log is set to the virtual address area.
[0013] In the above scheme, a mapping relationship between the virtual address area in the user process address space and the physical address area of the persistent memory is established, so that the application can use the virtual address area as a write path to directly access the persistent memory.
[0014] In a possible implementation, the method further includes:
[0015] Pre-create a write-ahead log file in persistent memory, and set the path where the application writes the write-ahead log to the virtual address corresponding to the write-ahead log file in persistent memory;
[0016] The writing the write-ahead log into the persistent memory includes:
[0017] The write-ahead log is written to a write-ahead log file pre-created in the persistent memory.
[0018] In the above solution, the write-ahead log is written to a pre-created write-ahead log file in the persistent memory, which can write the write-ahead log quickly and conveniently.
[0019] In a possible implementation, the method further includes:
[0020] If the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created, and the path for writing the write-ahead log is set to a virtual address area corresponding to the new write-ahead log file.
[0021] In the above scheme, when the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created in the persistent memory, and the path for writing the write-ahead log is set to the virtual address corresponding to the new write-ahead log file, so that the application can write the write-ahead log to the write-ahead log file pre-created in the persistent memory, which can effectively avoid the write-ahead log file from being too large and is conducive to improving the flexibility of the write-ahead log file.
[0022] In a possible implementation, writing the write-ahead log into the persistent memory includes:
[0023] Writes the write-ahead log to persistent memory in a sequential manner.
[0024] In the above solution, the write-ahead log is written into the persistent memory in a sequential write manner, thereby improving the efficiency of the data writing method.
[0025] In a possible implementation manner, writing the to-be-written data into a database includes:
[0026] Initiate a data write transaction according to the data write request;
[0027] Writing the data to be written into the cache and submitting the data writing transaction;
[0028] The data in the cache is written into the database in batches.
[0029] In the above solution, whether the data to be written is written into the cache is recorded through the data writing transaction.
[0030] In a possible implementation, the write-ahead log includes a redo log and / or an undo log, the redo log includes change information corresponding to the data write request; the undo log includes rollback information corresponding to the data write request.
[0031] In the above solution, based on the redo log and the undo log, the atomicity of the data writing transaction can be effectively protected.
[0032] In a possible implementation, the rollback information includes a counter instruction corresponding to the data write request.
[0033] In the above solution, based on the counter-instruction corresponding to the data write request, the data write transaction can be accurately rolled back to the data state before the data write transaction when an error occurs, thereby ensuring the atomicity of the transaction.
[0034] In a possible implementation, writing the write-ahead log to the persistent memory according to the data write request includes:
[0035] According to the data write request, before writing the to-be-written data into the cache, writing an undo log into the persistent memory;
[0036] After the data to be written is written into the cache, a redo log is written into the persistent memory.
[0037] In the above scheme, before writing the data to be written into the cache, an undo log is written into the persistent memory, so that when the system fails, if the data write transaction corresponding to the data to be written is not completed, the unfinished data write transaction can be canceled according to the undo log; after writing the data to be written into the cache, a redo log is written into the persistent memory, so that when the system fails, if the data to be written cannot be written into the database and the corresponding data write transaction has been committed, the data write transaction can be redoed according to the redo log, effectively ensuring the atomicity and durability of the transaction.
[0038] In a possible implementation, the method further includes:
[0039] When an exception occurs, the committed data write transaction is obtained and added to the redo queue; and the started but uncommitted data write transaction is obtained and added to the undo queue;
[0040] According to the undo log, undo processing is performed on the transactions in the undo queue;
[0041] According to the redo log, redo operations are performed on the transactions in the redo queue.
[0042] In the above scheme, when an exception occurs, the started but uncommitted data write transaction is undone according to the undo log, and the committed data write transaction is redone according to the redo log, which can effectively ensure the atomicity and durability of the transaction, accurately recover data, and improve the reliability of the system.
[0043] In a second aspect, an embodiment of the present application provides a server, the server comprising: a configuration module, an application, a write module, a persistent memory and a database; wherein:
[0044] The configuration module is used to configure the persistent memory to a direct access mode on the operating system of the server; mount the persistent memory to a file system of the server, and enable direct access parameters to support direct access to the persistent memory;
[0045] The application is used to obtain a data write request and write a pre-write log into the persistent memory according to the data write request; wherein the data write request is used to request to write the data to be written;
[0046] The writing module is used to respond to a data writing request and write the data to be written into the database.
[0047] In the above solution, the persistent memory is configured as direct access mode, and the direct access parameters of the file system on which the persistent memory is mounted are enabled, so that the server's application can directly read and write the persistent memory, thereby directly writing the write-ahead log to the persistent memory. After obtaining the data write request, on the one hand, the server's kernel space is skipped, and the server's application directly writes the write-ahead log to the server hardware. On the other hand, low-latency and non-volatile persistent memory is used to store the write-ahead log, which can effectively improve the efficiency of writing the write-ahead log, and the write-ahead log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server.
[0048] In a possible implementation, the configuration module is further configured to:
[0049] Establishing a mapping relationship between a virtual address area in a user process address space of the server and a physical address area of the persistent memory;
[0050] The mode of writing the write-ahead log is set to directly write into the persistent memory; and the path of writing the write-ahead log is set to the virtual address area.
[0051] In the above scheme, a mapping relationship between the virtual address area in the user process address space and the physical address area of the persistent memory is established, so that the application can use the virtual address area as a write path to directly access the persistent memory.
[0052] In a possible implementation, the configuration module is further configured to:
[0053] Pre-create a write-ahead log file in persistent memory, and set the path where the application writes the write-ahead log to the virtual address corresponding to the write-ahead log file in persistent memory;
[0054] The writing module is specifically used for:
[0055] The write-ahead log is written to a write-ahead log file pre-created in the persistent memory.
[0056] In the above solution, the write-ahead log is written to a pre-created write-ahead log file in the persistent memory, which can write the write-ahead log quickly and conveniently.
[0057] In a possible implementation, the configuration module is further configured to:
[0058] If the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created, and the path for writing the write-ahead log is set to a virtual address area corresponding to the new write-ahead log file.
[0059] In the above scheme, when the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created in the persistent memory, and the path for writing the write-ahead log is set to the virtual address corresponding to the new write-ahead log file, so that the application can write the write-ahead log to the write-ahead log file pre-created in the persistent memory, which can effectively avoid the write-ahead log file from being too large and is conducive to improving the flexibility of the write-ahead log file.
[0060] In a possible implementation, the application is specifically used to:
[0061] Writes the write-ahead log to persistent memory in a sequential manner.
[0062] In the above solution, the write-ahead log is written into the persistent memory in a sequential write manner, thereby improving the efficiency of the data writing method.
[0063] In a possible implementation, the writing module is specifically used to:
[0064] Initiate a data write transaction according to the data write request;
[0065] Writing the data to be written into the cache and submitting the data writing transaction;
[0066] The data in the cache is written into the database in batches.
[0067] In the above solution, whether the data to be written is written into the cache is recorded through the data writing transaction.
[0068] In a possible implementation, the write-ahead log includes a redo log and / or an undo log, the redo log includes change information corresponding to the data write request; the undo log includes rollback information corresponding to the data write request.
[0069] In the above solution, based on the redo log and the undo log, the atomicity of the data writing transaction can be effectively protected.
[0070] In a possible implementation, the rollback information includes a counter instruction corresponding to the data write request.
[0071] In the above solution, based on the counter instruction corresponding to the data write request, the data write transaction can be accurately rolled back to the data state before the data write transaction when an error occurs, thereby ensuring the atomicity of the transaction.
[0072] In a possible implementation, the writing module is further specifically used for:
[0073] According to the data write request, before writing the to-be-written data into the cache, writing an undo log into the persistent memory;
[0074] After the data to be written is written into the cache, a redo log is written into the persistent memory.
[0075] In the above scheme, before writing the data to be written into the cache, an undo log is written into the persistent memory, so that when the system fails, if the data write transaction corresponding to the data to be written is not completed, the unfinished data write transaction can be canceled according to the undo log; after writing the data to be written into the cache, a redo log is written into the persistent memory, so that when the system fails, if the data to be written cannot be written into the database and the corresponding data write transaction has been committed, the data write transaction can be redoed according to the redo log, effectively ensuring the atomicity and durability of the transaction.
[0076] In a possible implementation, the writing module is further used to:
[0077] When an exception occurs, the committed data write transaction is obtained and added to the redo queue; and the started but uncommitted data write transaction is obtained and added to the undo queue;
[0078] According to the undo log, undo processing is performed on the transactions in the undo queue;
[0079] According to the redo log, redo operations are performed on the transactions in the redo queue.
[0080] In the above scheme, when an exception occurs, the started but uncommitted data write transaction is undone according to the undo log, and the committed data write transaction is redone according to the redo log, which can effectively ensure the atomicity and durability of the transaction, accurately recover data, and improve the reliability of the system.
[0081] In a third aspect, an embodiment of the present application provides a server, the server comprising: a processing chip, and a memory communicatively connected to the processor;
[0082] The memory stores computer-executable instructions;
[0083] The processing chip executes the computer-executable instructions stored in the memory to implement the method described above.
[0084] In the above solution, on the one hand, the server's kernel space is skipped, and the server's application writes the pre-write log directly to the server hardware. On the other hand, low-latency and non-volatile persistent memory is used to store the pre-write log, which can effectively improve the efficiency of writing the pre-write log. In addition, the pre-write log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server.
[0085] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to implement the method as described above when executed by a processor.
[0086] In the above solution, on the one hand, the server's kernel space is skipped, and the server's application writes the pre-write log directly to the server hardware. On the other hand, low-latency and non-volatile persistent memory is used to store the pre-write log, which can effectively improve the efficiency of writing the pre-write log. In addition, the pre-write log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server.
[0087] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the method described above when executed by a processing chip.
[0088] In the above solution, on the one hand, the server's kernel space is skipped, and the server's application writes the pre-write log directly to the server hardware. On the other hand, low-latency and non-volatile persistent memory is used to store the pre-write log, which can effectively improve the efficiency of writing the pre-write log. In addition, the pre-write log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server.
[0089] In the data writing method and server provided in the embodiments of the present application, the persistent memory is configured as a direct access mode, and the direct access parameters of the file system on which the persistent memory is mounted are enabled, so that the server's application can directly read and write the persistent memory, thereby directly writing the write-ahead log to the persistent memory; after obtaining the data write request, on the one hand, the server's kernel space is skipped, and the server's application directly writes the write-ahead log to the server hardware; on the other hand, low-latency and non-volatile persistent memory is used to store the write-ahead log, which can effectively improve the efficiency of writing the write-ahead log, and the write-ahead log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server. BRIEF DESCRIPTION OF THE DRAWINGS
[0090] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0091] Figure 1 A schematic diagram of a hierarchical structure of a memory provided in an embodiment of the present application;
[0092] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;
[0093] Figure 3 A schematic diagram of a data writing method provided in an embodiment of the present application;
[0094] Figure 4 A flow chart of a server configuration method provided in an embodiment of the present application;
[0095] Figure 5 A schematic diagram of the architecture of a server provided in an embodiment of the present application;
[0096] Figure 6 A flowchart of a write-ahead log writing method provided in an embodiment of the present application;
[0097] Figure 7A schematic diagram of the structure of a data writing device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0098] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.
[0099] To facilitate understanding, the terms involved in the embodiments of the present application are first explained.
[0100] Persistent Memory (PMem): is a memory device located on the memory bus, allowing programs to directly address data by byte; the content stored in persistent memory is non-volatile during power cycles, that is, data will not be lost due to power failure. Therefore, persistent memory has the characteristics of low latency and non-volatility.
[0101] Figure 1 A schematic diagram of the hierarchical structure of the memory provided in the embodiment of the present application is shown in FIG. Figure 1 As shown, the memory hierarchy may include CPU registers, CPU caches, dynamic random access memory (DRAM), persistent memory, NAND flash solid state drive (NAND SSD), hard disk drive (HHD), and tape, wherein the CPU cache may include multiple cache levels such as level 1 cache (L1), level 2 cache (L2), level 3 cache (L3), and level 4 cache (L4). From CPU registers, CPU caches, dynamic random access memory, persistent memory, flash solid state drive, hard disk drive to tape, their prices decrease, their capacities increase, and their data access delays increase. The data access delays of CPU registers, CPU caches, dynamic random access memory, persistent memory, flash solid state drive, hard disk drive, and tape are approximately 0.1ns, 1-10ns, 100ns, less than 1us, 10-100us, 10ms, and 100ms, respectively.
[0102] Dynamic random access memory is volatile, while persistent memory and NAND flash solid state drives are non-volatile; currently, the maximum memory capacity of a single persistent memory can reach 512GB, while a single dynamic random access memory generally has a maximum of 32 / 64GB; the unit price of persistent memory (e.g., price per GB) is about half of that of dynamic random access memory; the data transmission granularity between dynamic random access memory and persistent memory is cache line granularity, while the data transmission granularity of NAND flash solid state drives is block granularity; the data transmission granularity between dynamic random access memory and persistent memory supports Load / Store instructions, while NAND flash solid state drives support IO instructions; persistent memory has a latency performance advantage of 1-2 orders of magnitude compared to NAND flash solid state drives. In summary, from the perspective of cost, performance, and capacity, persistent memory is a layer of storage located between dynamic random access memory and NAND flash solid state drives, and persistent memory fills the performance gap between dynamic random access memory and NAND flash solid state drives.
[0103] Persistence: refers to the mechanism of storing data (such as objects in memory) in a permanent storage medium (such as disk, hard disk, tape, etc.) and being able to reload data when needed. The main application of persistence is to store objects in memory in a relational database. Of course, it can also be stored in a disk file or in an XML data file, etc.
[0104] Direct Access (DAX) is a mechanism that allows applications to access persistent media directly from the CPU (via loads and stores), bypassing the traditional I / O stack (page cache and block layer).
[0105] WAL technology is a common database transaction log technology used to ensure the persistence and recovery capability of the database. The core idea of WAL technology is to write the corresponding WAL log (write-ahead log) into a persistent log file before making any changes to the database.
[0106] For ease of understanding, the following Figure 2 , the application scenarios involved in the embodiments of the present application are described.
[0107] Figure 2 A schematic diagram of an application scenario provided by an embodiment of the present application. Figure 1 ,The server can be divided into user space, kernel space and hardware. The user space of the server can include applications, the kernel space of the server can include cache, and the hardware of the server can include persistent memory and local database.
[0108] The application may be used to issue a data write request and, according to the data write request, write the write-ahead log to the persistent memory. The data write request may be used to request that the data to be written be written to the local database.
[0109] In response to the data writing request, the server writes the data to be written into the local database.
[0110] The embodiment of the present application does not limit the specific type of the local database. For example, the local database can be a relational database, a distributed database, etc.
[0111] It should be noted that the server may first write the data to be written into the cache, and then write the data in the cache into the local database in batches, or directly write the data to be written into the local database. Figure 1 The example in which the server first writes the data to be written into the cache and then writes the data in the cache into the local database in batches is used for illustration only, and does not constitute a limitation on the technical solution provided in the embodiment of the present application.
[0112] Next, combine Figure 3 , the data writing method provided in the embodiment of the present application is described.
[0113] Figure 3 The following is a flow chart of a data writing method provided in an embodiment of the present application. The execution subject of the method is Figure 1 The server shown in the figure may also be a processing chip set in the server. The following description only takes the execution subject as an example. Figure 3 , the method may include:
[0114] S301. On the operating system of the server, configure the persistent memory to direct access mode; mount the persistent memory to the file system of the server, and enable direct access parameters to support direct access to the persistent memory.
[0115] The server can be divided into user space, kernel space and hardware. The user space of the server can include applications, the kernel space of the server can include cache and configuration modules, and the hardware of the server can include persistent memory and local database.
[0116] Exemplarily, the direct access mode of the persistent memory may be Fsdax (Filesystem-DAX) mode. In this embodiment, the configuration module may configure the persistent memory to Fsdax mode on the operating system of the server. In this case, the persistent memory is configured as a block device supporting direct access mode, in which mode, the user can directly access the persistent memory.
[0117] Optionally, the file system may be an xfs file system or an ext4 file system, or other file systems supporting direct access mode, which are not limited here.
[0118] Specifically, the configuration module mounts the persistent memory to the server's file system and turns on the direct access parameter, thereby configuring the file system to direct access mode so that applications can skip the kernel page cache and directly read and write data on the persistent memory.
[0119] S302: Obtain a data write request, and write a pre-write log into a persistent memory according to the data write request; wherein the data write request is used to request writing of data to be written.
[0120] Among them, the write-ahead log includes modifications to the database.
[0121] Applications can use this to issue data write requests.
[0122] Exemplarily, issuing a data write request by the application may include the application sending the data write request to a write module.
[0123] Correspondingly, the write module receives the data write request.
[0124] The data writing request may include the data to be written. The data writing request may be used to request the writing module to write the data to be written into the database.
[0125] In this embodiment, the application can obtain the data write request issued by the application, and according to the data write request, skip the kernel space of the server and directly write the write-ahead log to the persistent memory in the server hardware, which can effectively improve the performance of writing the write-ahead log.
[0126] S303: Respond to the data writing request and write the data to be written into the database.
[0127] In this embodiment, the writing module can obtain the data to be written according to the data writing request, and write the data to be written into the database.
[0128] Exemplarily, the database may be located locally on the server. Optionally, the database includes but is not limited to a relational database, a distributed database, and the like.
[0129] In a possible implementation, writing the data to be written into the database includes:
[0130] According to the data writing request, start the data writing transaction (begin transaction);
[0131] Write the data to be written into the cache and commit the data writing transaction;
[0132] Write the data in the cache to the database in batches.
[0133] Specifically, the write module can first write the data to be written into the cache, and then write the data in the cache into the local database in batches, so as to reduce the frequent operations on the local database, thereby improving the data writing performance of the server. When a system failure occurs, the data to be written corresponding to the submitted data write transaction may still remain in the cache and fail to be written to the database, and may be lost when the system fails; or the data to be written corresponding to the unfinished data write transaction may have been written to the database, which may cause inconsistency between the database and the data write transaction status.
[0134] The data write transaction may be recorded in the persistent memory. Based on the data write transaction, the execution status of the data write request may be determined.
[0135] In the data writing method provided by this embodiment, the application program issues a data writing request, and writes a write-ahead log into the persistent memory according to the data writing request; the writing module responds to the data writing request and writes the data to be written into the database. On the one hand, the kernel space of the server is skipped, and the write-ahead log is written directly to the server hardware by the application program of the server. On the other hand, the write-ahead log is stored in a persistent memory with low latency and non-volatile memory, which can effectively improve the efficiency of writing the write-ahead log, and the write-ahead log in the persistent memory will not be lost in the event of an abnormality, thereby improving the overall data writing performance and reliability of the server.
[0136] Data recovery can be performed based on the write-ahead log. In a possible implementation, the write-ahead log includes a redo log and / or an undo log (also called a rollback log). The redo log includes change information corresponding to a data write request; the undo log includes rollback information corresponding to a data write request. The redo log is used to protect the persistence of data write transactions, and the undo log is used to protect the atomicity of data write transactions.
[0137] Based on the above embodiment, in a possible implementation, writing a write-ahead log to a persistent memory according to a data write request includes:
[0138] Based on the data write request, before writing the data to be written into the cache, write the undo log into the persistent memory;
[0139] After writing the data to be written into the cache, the redo log is written into the persistent memory.
[0140] Specifically, after receiving a data write request, the corresponding reverse operation instruction is generated and the undo log is written into the persistent memory, and then the data to be written is written into the cache.
[0141] For example, if the current data write request includes an insert operation, a corresponding delete operation is generated and written to the undo log; if the current data write request includes an update operation, such as changing a from A1 to A2, an operation of changing a from A2 to A1 is generated and written to the undo log.
[0142] After writing the data to be written into the cache, the change information of the data file is written into the redo log to complete the writing of the write-ahead log.
[0143] In this embodiment, before writing the data to be written into the cache, an undo log is written into the persistent memory, so that when the system fails, if the data write transaction corresponding to the data to be written is not completed, the unfinished data write transaction can be canceled according to the undo log; after writing the data to be written into the cache, a redo log is written into the persistent memory, so that when the system fails, if the data to be written cannot be written into the database and the corresponding data write transaction has been committed, the data write transaction can be redoed according to the redo log, effectively ensuring the atomicity and durability of the transaction.
[0144] Furthermore, in a possible implementation, the method further includes:
[0145] When an exception occurs, the committed data write transaction is obtained and added to the redo queue; and the started but uncommitted data write transaction is obtained and added to the undo queue;
[0146] According to the undo log, undo processing is performed on the transactions in the undo queue;
[0147] Redo operations are performed on transactions in the redo queue according to the redo log.
[0148] Specifically, when an exception occurs, in order to achieve the persistence of data write transactions, all submitted data write transactions need to be redone; in order to achieve the atomicity of data write transactions, all unfinished data write transactions need to be canceled. Therefore, all data write transactions that have been started before the failure occurs are forward scanned, and submitted data write transactions are obtained and added to the redo queue, and data write transactions that have been started but not submitted are obtained and added to the undo queue. The redo queue is forward scanned, and each data write transaction in the redo queue is redone according to the redo log; and the undo queue is reversely scanned, and each data write transaction in the undo queue is undoed according to the undo log, and a checkpoint is generated. At the next system failure, data recovery only needs to be performed based on the data write transactions after the checkpoint.
[0149] In this embodiment, when an exception occurs, an undo process is performed on the data write transaction that has been started but not committed according to the undo log, and a redo process is performed on the data write transaction that has been committed according to the redo log, which can effectively ensure the atomicity and durability of the transaction, accurately perform data recovery, and improve the reliability of the system.
[0150] In order to support server applications to write write-ahead logs directly to persistent memory, the persistent memory and applications need to be configured accordingly in advance. Figure 4 A flow chart of a server configuration method provided in an embodiment of the present application. The execution subject of the method can be Figure 1 The server in the embodiment can also be a processing chip set in the server. The following is an example of the execution subject being the server. Figure 4 In a possible implementation, the method further includes:
[0151] S401. Configure the persistent memory to be in direct access mode.
[0152] In this embodiment, the server can be divided into user space, kernel space and hardware. The user space of the server can include applications, the kernel space of the server can include cache and configuration modules, and the hardware of the server can include persistent memory and local database.
[0153] In this embodiment, the configuration module can configure the persistent memory to direct access mode on the server's operating system. In this case, the persistent memory is configured as a block device that supports direct access mode. In this mode, users can directly access the persistent memory.
[0154] S402: Mount the persistent memory to the file system of the server and enable direct access parameters.
[0155] Specifically, the configuration module mounts the persistent memory to the server's file system and turns on the DAX parameter to configure the file system to direct access mode so that applications can skip the kernel page cache and directly read and write data on the persistent memory.
[0156] In this embodiment, the persistent memory is configured as direct access mode, and the direct access parameters of the file system on which the persistent memory is mounted are enabled, so that the server's application can directly read and write the persistent memory, thereby directly writing the write-ahead log to the persistent memory, which is beneficial to improving the performance of writing the write-ahead log.
[0157] S403: Establish a mapping relationship between the virtual address of the user space of the server and the physical address of the persistent memory.
[0158] In this embodiment, the configuration module can map the virtual address area in the user process address space of the server to the physical address area of the persistent memory, thereby establishing a mapping relationship between the virtual address area in the user process address space and the physical address area of the persistent memory.
[0159] Figure 5 This is a schematic diagram of the server architecture provided in the embodiment of the present application. Figure 5 As shown in the figure, the server can be divided into three levels: user space, kernel space and hardware. User space includes applications and glibc library; at the hardware level, external storage devices are divided into three categories: block devices, flash memory and NVDIMM (Nonn-Volatile DIMM, non-volatile memory) devices. Block devices mainly include mechanical hard disks and solid-state hard disks; flash memory is divided into NAND flash memory and NOR flash memory according to storage structure.
[0160] The kernel supports multiple file system types. In order to provide a unified file operation interface for user programs and enable different file system implementations to coexist, the kernel sets up an abstract layer called Virtual File System (VFS), where file systems are divided into block device file systems, flash file systems, memory file systems, pseudo file systems, and other file systems; the storage devices corresponding to the block device file system are block devices such as mechanical hard disks and solid-state drives; the storage devices corresponding to the flash file system are NAND flash memory and NOR flash memory; the files of the memory file system are in memory and will be lost after power failure; the pseudo file system is a fake file system, just for the programming interface of the virtual file system. The kernel space also includes: page cache, libnvdimm subsystem, block device layer, MTD layer, block cache, IO scheduler, NVDIMM device driver, block device driver, and flash driver.
[0161] Among them, the speed of accessing external storage devices is very slow. In order to avoid accessing external storage devices every time a file is read or written, the file system creates a cache in memory for each file, called a page cache; the kernel provides a unified block device layer for all block devices, and provides a unified MTD (Memory Technology Device) layer for all flash memories; in order to avoid accessing block devices every time a read or write is performed, the kernel creates a block cache in memory for each block device, and the block cache is implemented based on the page cache; the IO scheduler is used to determine the submission order of read and write requests, and sorts the read and write requests according to the sector number, reducing the time spent on moving the head to find tracks and sectors when accessing the mechanical hard disk, thereby improving throughput; a corresponding driver is set for each external storage device; the libnvdimm subsystem provides support for three types of NVDIMM devices: NVDIMM devices in persistent memory mode, NVDIMM devices in block device mode, and NVDIMM devices that support both persistent memory and block device access modes. The persistent memory access mode is to treat the NVDIMM device as memory, and the block device access mode is to treat the NVDIMM device as a block device. For NVDIMM devices using the persistent memory mode, the file system needs to implement the direct access mode, bypassing the page cache and block device layer, and directly mapping the memory in the NVDIMM device to the virtual address space of the process or kernel, so that the application can directly access the NVDIMM device using the persistent memory mode, thereby greatly improving the access speed.
[0162] S404: Set the mode of writing the write-ahead log to directly write into the persistent memory; and set the path of writing the write-ahead log to the virtual address corresponding to the persistent memory.
[0163] In this embodiment, the configuration module can set the method for writing the application to the write-ahead log in the configuration file of the application to not write to the cache but directly write to the persistent memory; and set the path for writing the application to the write-ahead log or the directory for saving the write-ahead log to the virtual address corresponding to the persistent memory.
[0164] The application can use the mmap() call request to write the write-ahead log to the persistent memory according to the virtual address corresponding to the persistent memory.
[0165] Figure 6 A flowchart of a write-ahead log writing method provided in an embodiment of the present application is provided. The execution subject of the method may be Figure 1 The server shown may also be a processing chip in the server. The following is only an example of the execution subject being a server. In one possible implementation, writing a write-ahead log to a persistent memory may include:
[0166] S601: Pre-create a write-ahead log file in the persistent memory, and set the path where the application writes the write-ahead log to the virtual address corresponding to the write-ahead log file in the persistent memory.
[0167] In this embodiment, the server can be divided into user space, kernel space and hardware. The user space of the server can include applications, the kernel space of the server can include cache and configuration modules, and the hardware of the server can include persistent memory and local database.
[0168] The configuration module can pre-create a write-ahead log file in the persistent memory, and in the configuration file of the application, set the path where the application writes the write-ahead log to the virtual address corresponding to the write-ahead log file in the persistent memory.
[0169] S602: If the amount of data in the write-ahead log in the write-ahead log file does not exceed a preset threshold, write the write-ahead log to the write-ahead log file pre-created in the persistent memory.
[0170] In this embodiment, the application program may write the write-ahead log to a write-ahead log file created in advance in the persistent memory when the amount of data in the write-ahead log in the write-ahead log file does not exceed a preset threshold.
[0171] In the write-ahead log writing method provided in this embodiment, when the amount of data of the write-ahead log in the write-ahead log file of the application does not exceed a preset threshold, the write-ahead log is written to the write-ahead log file pre-created in the persistent memory, so that the write-ahead log can be written conveniently and quickly.
[0172] S603: If the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created in the persistent memory, and the path for writing the write-ahead log is set to a virtual address corresponding to the new write-ahead log file.
[0173] In this embodiment, when the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, the configuration module can create a new write-ahead log file in the persistent memory, and set the path for writing the write-ahead log to the virtual address corresponding to the new write-ahead log file in the configuration file of the application.
[0174] S604: Write the write-ahead log to a new write-ahead log file in the persistent memory.
[0175] In this embodiment, when the amount of write-ahead log data in the write-ahead log file exceeds a preset threshold, the application program may write the write-ahead log to a newly created write-ahead log file in the persistent memory.
[0176] It is understood that the step numbers in the various embodiments of the present application do not constitute a limitation on the execution order, and the execution order of the steps in each embodiment should be determined according to its internal logic. The various embodiments described in this application can be implemented separately or independently, and this application does not make specific limitations on this.
[0177] In the write-ahead log writing method provided by the present embodiment, when the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, the configuration module creates a new write-ahead log file in the persistent memory, and sets the path for writing the write-ahead log to the virtual address corresponding to the new write-ahead log file. The application writes the write-ahead log to the write-ahead log file created in advance in the persistent memory, which can effectively avoid the write-ahead log file from being too large and is conducive to improving the flexibility of the write-ahead log file.
[0178] In actual applications, there are many ways to write the write-ahead log to the persistent memory. In one possible implementation, writing the write-ahead log to the persistent memory includes:
[0179] Writes the write-ahead log to persistent memory in a sequential manner.
[0180] Specifically, writing the write-ahead log to the persistent memory may include writing the write-ahead log to the persistent memory in a random write or sequential write manner. To further improve the efficiency of writing the write-ahead log to the persistent memory, the write-ahead log may be written to the persistent memory in a sequential write manner, thereby improving the efficiency of the data writing method.
[0181] The embodiment of the present application also provides a data writing device. The data writing device may be a server. Figure 7 This is a schematic diagram of the structure of a data writing device provided in an embodiment of the present application. Figure 7 As shown, the data writing device includes a user space, a kernel space and hardware. The user space of the data writing device includes a configuration module and an application. The kernel space of the data writing device includes a cache, a configuration module and a writing module. The hardware of the data writing device includes a persistent memory and a local database.
[0182] A configuration module, configured to configure the persistent memory to a direct access mode on the operating system of the server; mount the persistent memory to a file system of the server, and enable direct access parameters to support direct access to the persistent memory;
[0183] The application is used to obtain a data write request and write a pre-write log into the persistent memory according to the data write request; wherein the data write request is used to request to write the data to be written.
[0184] The writing module is used to respond to a data writing request and write the data to be written into the database.
[0185] In a possible implementation, in a possible implementation, the configuration module is further used to:
[0186] Establishing a mapping relationship between a virtual address area in a user process address space of the server and a physical address area of the persistent memory;
[0187] The mode of writing the write-ahead log is set to directly write into the persistent memory; and the path of writing the write-ahead log is set to the virtual address area.
[0188] In a possible implementation, the configuration module is further configured to:
[0189] Pre-create a write-ahead log file in persistent memory, and set the path where the application writes the write-ahead log to the virtual address corresponding to the write-ahead log file in persistent memory;
[0190] The writing module is specifically used for:
[0191] The write-ahead log is written to a write-ahead log file pre-created in the persistent memory.
[0192] In a possible implementation, the configuration module is further configured to:
[0193] If the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created, and the path for writing the write-ahead log is set to a virtual address area corresponding to the new write-ahead log file.
[0194] In a possible implementation, the application is specifically used to:
[0195] Writes the write-ahead log to persistent memory in a sequential manner.
[0196] In a possible implementation, the writing module is specifically used to:
[0197] Initiate a data write transaction according to the data write request;
[0198] Writing the data to be written into the cache and submitting the data writing transaction;
[0199] The data in the cache is written into the database in batches.
[0200] In a possible implementation, the write-ahead log includes a redo log and / or an undo log, the redo log includes change information corresponding to the data write request; the undo log includes rollback information corresponding to the data write request.
[0201] In a possible implementation, the rollback information includes a counter instruction corresponding to the data write request.
[0202] In a possible implementation, the writing module is further specifically used for:
[0203] According to the data write request, before writing the to-be-written data into the cache, writing an undo log into the persistent memory;
[0204] After the data to be written is written into the cache, a redo log is written into the persistent memory.
[0205] In a possible implementation, the writing module is further used to:
[0206] When an exception occurs, the committed data write transaction is obtained and added to the redo queue; and the started but uncommitted data write transaction is obtained and added to the undo queue;
[0207] According to the undo log, undo processing is performed on the transactions in the undo queue;
[0208] According to the redo log, redo operations are performed on the transactions in the redo queue.
[0209] The data writing device provided in this embodiment can be used to execute the write-ahead log writing method shown in the above method embodiment. Its implementation principle and technical effect are similar and will not be described in detail here.
[0210] An embodiment of the present application also provides a computer-readable storage medium, in which computer execution instructions are stored. When the computer execution instructions are executed by a processing chip, they can be used to execute the data writing method shown in the above method embodiment. The implementation principle and technical effect are similar and will not be elaborated here.
[0211] An embodiment of the present application also provides a computer program product, including a computer program, which implements the method provided in any embodiment of the present application when the computer program is executed by a processing chip.
[0212] All or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned memory (storage medium) includes: read-only memory (English: read-only memory, abbreviated: ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.
[0213] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0214] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0215] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0216] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the embodiments of the present application are also intended to include these modifications and variations.
[0217] In the embodiments of the present application, the terms "include", "comprise" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the element. The term "or" and its variations may refer to "and / or". In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish similar objects, and do not have to be used to describe a specific order or sequence. In the embodiments of the present application, "multiple" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, A and / or B, which may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.
[0218] After considering the invention disclosed in the specification and practice, those skilled in the art will easily think of other embodiments of the present application. The embodiments of the present application are intended to cover any variation, use or adaptive change of the embodiments of the present application, which follows the general principles of the embodiments of the present application and includes common knowledge or conventional technical means in the technical field that are not disclosed in the embodiments of the present application.
Claims
1. A data writing method, characterized in that: include: Configure persistent memory to direct access mode; Mounting the persistent memory to a file system of a server and enabling direct access parameters to support direct access to the persistent memory; Obtaining a data write request, and writing a pre-write log into the persistent memory according to the data write request; wherein the data write request is used to request writing of the data to be written; In response to a data write request, the data to be written is written into the database.
2. The method according to claim 1, characterized in that The method further comprises: Establishing a mapping relationship between a virtual address area in a user process address space of the server and a physical address area of the persistent memory; The mode of writing the write-ahead log is set to directly write into the persistent memory; and the path of writing the write-ahead log is set to the virtual address area.
3. The method according to claim 2, characterized in that The method further comprises: Pre-create a write-ahead log file in the persistent memory, and set the path where the application writes the write-ahead log to a virtual address corresponding to the write-ahead log file in the persistent memory; The writing the write-ahead log into the persistent memory includes: The write-ahead log is written to a write-ahead log file pre-created in the persistent memory.
4. The method according to claim 3, characterized in that The method further comprises: If the amount of data in the write-ahead log in the write-ahead log file exceeds a preset threshold, a new write-ahead log file is created in the persistent memory, and the path for writing the write-ahead log is set to a virtual address area corresponding to the new write-ahead log file.
5. The method according to claim 1, characterized in that Writing the data to be written into the database comprises: Initiate a data write transaction according to the data write request; Writing the data to be written into the cache and submitting the data writing transaction; The data in the cache is written into the database in batches.
6. The method according to claim 5, characterized in that The write-ahead log includes a redo log and / or an undo log, the redo log includes change information corresponding to the data write request; the undo log includes rollback information corresponding to the data write request.
7. The method according to claim 6, characterized in that The rollback information includes a counter instruction corresponding to the data write request.
8. The method according to claim 6, characterized in that Writing the write-ahead log to the persistent memory according to the data write request includes: According to the data write request, before writing the to-be-written data into the cache, writing an undo log into the persistent memory; After the data to be written is written into the cache, a redo log is written into the persistent memory.
9. The method according to claim 8, characterized in that The method further comprises: When an exception occurs, the committed data write transaction is obtained and added to the redo queue; and the started but uncommitted data write transaction is obtained and added to the undo queue; According to the undo log, undo processing is performed on the transactions in the undo queue; According to the redo log, redo operations are performed on the transactions in the redo queue.
10. A server, comprising: A processing chip, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processing chip executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.