Data writing and reading method and device, equipment and medium

Abstract

The embodiment of the invention provides a data writing and reading method and device, equipment and a medium, and relates to the technical field of data storage, the data writing and reading method and device are applied to electronic equipment, a WAL file in a memory of the electronic equipment comprises a message header area and a data area, and the method comprises the steps of obtaining to-be-written first data and a first message header corresponding to the first data; acquiring a first storage position of the first data in the data area, and acquiring a second storage position of the first message header in the message header area; writing the first data into a first storage position; adding indication information in the first message header to obtain a second message header, the indication information indicating the first storage position; and writing the second message header into the second storage position. According to the scheme, the utilization rate of the storage space occupied by the WAL file can be improved, so that the effective space of the memory is fully utilized.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular, to a data writing and reading method, device, equipment and medium. Background Art

[0002] In order to improve the performance of a database and ensure the atomicity and durability of data operations, an electronic device usually adopts a Write Ahead Log (WAL) mechanism for data storage, that is, first writes data and a message header corresponding to the data into a buffer, and then writes the data and the message header in the buffer into a WAL file of a memory.

[0003] The WAL file includes multiple log pages, and each log page consists of two parts: a message header and data, that is, the message header and data are stored in the WAL file in the form of log pages. In the prior art, the length of a message header is 12 bytes (Byte, B), but in the memory, a message header needs to occupy one sector, and the length of one sector is 4 kilobytes (KiloByte, KB) or 512 B, that is, the length of one sector far exceeds the length of a message header, resulting in low utilization rate of the storage space occupied by the WAL file, and further resulting in insufficient utilization of the effective space of the memory. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a data writing and reading method, device, equipment and medium to improve the utilization rate of the storage space occupied by the WAL file, so that the effective space of the memory can be fully utilized. The specific technical solutions are as follows:

[0005] In a first aspect, the embodiments of the present application provide a data writing method applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. The method includes:

[0006] Obtain first data to be written and a first message header corresponding to the first data;

[0007] Obtain a first storage location of the first data in the data area, and obtain a second storage location of the first message header in the message header area;

[0008] Write the first data into the first storage location;

[0009] Add indication information to the first message header to obtain a second message header, where the indication information indicates the first storage location;

[0010] Write the second message header into the second storage location.

[0011] In some embodiments, the step of adding indication information to the first message header to obtain a second message header includes:

[0012] Calculating a first length between the first storage location and the second storage location to obtain indication information;

[0013] Writing the indication information into the first message header to obtain a second message header.

[0014] In some embodiments, the step of obtaining a first storage location of the first data in the data area includes:

[0015] If no data is written in the data area, obtaining a starting position and a second length of the message header area, and offsetting backward from the starting position by the second length to obtain a first storage location;

[0016] If data has been written in the data area, obtaining a third storage location and a third length of the last written data, and offsetting backward from the third storage location by the third length to obtain a first storage location.

[0017] In some embodiments, the step of obtaining a second storage location of the first message header in the message header area includes:

[0018] Obtaining a first quantity of written message headers;

[0019] Calculating a product of the first quantity and a preset message header length to obtain a fourth length;

[0020] Offsetting backward from the starting position of the message header area by the fourth length to obtain a second storage location.

[0021] In some embodiments, before writing the first data to the first storage location, the method further includes:

[0022] Determining a fourth storage location after offsetting backward from the first storage location by a fifth length, where the fifth length is the maximum length of one data;

[0023] If the fourth storage location is within the data area, performing the step of writing the first data to the first storage location.

[0024] In some embodiments, the length of the sector occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0025] In a second aspect, an embodiment of the present application provides a data reading method applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. The method includes:

[0026] Obtain a first reading position of the message header area;

[0027] Read a third message header from the first reading position, where the third message header indicates a second reading position of second data corresponding to the third message header in the data area;

[0028] Determine the second reading position according to indication information included in the third message header;

[0029] Read the second data from the second reading position.

[0030] In some embodiments, the step of determining the second reading position according to the indication information included in the third message header includes:

[0031] Obtain indication information from the third message header to obtain a sixth length, where the sixth length is an offset between the second reading position and the first reading position;

[0032] Offset backward by the sixth length at the first reading position to obtain the second reading position.

[0033] In some embodiments, the step of obtaining the first reading position of the message header area includes:

[0034] Obtain a second quantity of the read data;

[0035] Calculate a product of the second quantity and a preset message header length to obtain a seventh length;

[0036] Offset backward by the seventh length at the starting position of the message header area to obtain the first reading position.

[0037] In some embodiments, after obtaining the first reading position, it further includes:

[0038] If there is a message header at the first reading position, execute the step of reading the third message header from the first reading position.

[0039] In some embodiments, the length of the sector occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0040] In a third aspect, an embodiment of the present application provides a data writing device, which is applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. The device includes:

[0041] A first obtaining module, configured to obtain first data to be written and a first message header corresponding to the first data;

[0042] A second acquisition module, configured to acquire a first storage location of the first data in the data area and acquire a second storage location of the first message header in the message header area;

[0043] A first writing module, configured to write the first data to the first storage location;

[0044] An adding module, configured to add indication information to the first message header to obtain a second message header, where the indication information indicates the first storage location;

[0045] A second writing module, configured to write the second message header to the second storage location.

[0046] In some embodiments, the adding module is specifically configured to:

[0047] Calculate a first length between the first storage location and the second storage location to obtain indication information;

[0048] Write the indication information to the first message header to obtain a second message header.

[0049] In some embodiments, the second acquisition module is specifically configured to:

[0050] If no data is written in the data area, acquire a start position and a second length of the message header area, and offset backward from the start position by the second length to obtain a first storage location;

[0051] If data has been written in the data area, acquire a third storage location and a third length of the last written data, and offset backward from the third storage location by the third length to obtain a first storage location.

[0052] In some embodiments, the second acquisition module is specifically configured to:

[0053] Acquire a first quantity of written message headers;

[0054] Calculate a product of the first quantity and a preset message header length to obtain a fourth length;

[0055] Offset backward from a start position of the message header area by the fourth length to obtain a second storage location.

[0056] In some embodiments, the first writing module is further configured to:

[0057] Before writing the first data to the first storage location, determine a fourth storage location that is offset backward by a fifth length from the first storage location, where the fifth length is the maximum length of one data; if the fourth storage location is within the data area, perform the step of writing the first data to the first storage location.

[0058] In some embodiments, the length of the sector occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0059] Fourthly, an embodiment of the present application provides a data reading device, which is applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. The device includes:

[0060] A third acquisition module, configured to acquire a first reading position of the message header area;

[0061] A first reading module, configured to read a third message header from the first reading position, where the third message header indicates a second reading position of the second data corresponding to the third message header in the data area;

[0062] A determination module, configured to determine the second reading position according to the indication information included in the third message header;

[0063] A second reading module, configured to read the second data from the second reading position.

[0064] In some embodiments, the determination module is specifically configured to:

[0065] Acquire indication information from the third message header to obtain a sixth length, where the sixth length is the offset between the second reading position and the first reading position;

[0066] Offset backward by the sixth length from the first reading position to obtain the second reading position.

[0067] In some embodiments, the third acquisition module is specifically configured to:

[0068] Acquire a second quantity of the read data;

[0069] Calculate the product of the second quantity and a preset message header length to obtain a seventh length;

[0070] Offset backward by the seventh length from the starting position of the message header area to obtain the first reading position.

[0071] In some embodiments, the first reading module is further configured to:

[0072] After obtaining the first reading position, if there is a message header at the first reading position, perform the step of reading the third message header from the first reading position.

[0073] In some embodiments, the length of the sector occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0074] In a fifth aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete communication with each other through the communication bus;

[0075] The memory is used to store a computer program. The WAL file in the memory includes a message header area and a data area;

[0076] The processor is used to implement any one of the above data writing or data reading methods when executing the program stored on the memory.

[0077] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it implements any one of the above data writing or data reading methods.

[0078] In a seventh aspect, an embodiment of the present application provides a computer program product containing instructions. When it runs on a computer, it causes the computer to execute any one of the above data writing or data reading methods.

[0079] Advantages of the embodiments of the present application:

[0080] In the technical solution provided by the embodiments of the present application, the WAL file is divided into a message header area and a data area. The message header area is used to write message headers, and the data area is used to write data. The electronic device writes data in the data area, adds indication information indicating the storage position of the data in the data area to the message header to obtain a new message header, and writes the new message header in the message header area. In this way, the message headers can be centrally stored in the message header area, and the storage position of the message header in the message header area and the storage position of the data corresponding to the message header in the data area can be accurately determined, reducing the number of sectors occupied by storing the message headers. Compared with storing by log pages, the utilization rate of the storage space occupied by the WAL file is improved, and thus the effective space of the memory is fully utilized.

[0081] Of course, when implementing any product or method of the present application, it is not necessarily required to achieve all the above advantages at the same time. Description of the Drawings

[0082] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other embodiments can also be obtained based on these drawings.

[0083] Figure 1 Schematic diagram for writing a message header and data into a memory;

[0084] Figure 2 The first flowchart of the data writing method provided by the embodiment of the present application;

[0085] Figure 3 A refined schematic diagram of step S24 provided by the embodiment of the present application;

[0086] Figure 4 The second flowchart of the data writing method provided by the embodiment of the present application;

[0087] Figure 5 A schematic diagram for writing a message header and data into a memory provided by the embodiment of the present application;

[0088] Figure 6 A flowchart of the data reading method provided by the embodiment of the present application;

[0089] Figure 7 A flowchart of the data writing provided by the embodiment of the present application;

[0090] Figure 8 A flowchart of the data reading provided by the embodiment of the present application;

[0091] Figure 9 A schematic structural diagram of the data writing device provided by the embodiment of the present application;

[0092] Figure 10 A schematic structural diagram of the data reading device provided by the embodiment of the present application;

[0093] Figure 11 A schematic structural diagram of the electronic device provided by the embodiment of the present application. Detailed implementation manners

[0094] The following will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application belong to the scope of protection of the present application.

[0095] WAL is a logging algorithm and mechanism widely used in databases and distributed storage systems to improve database performance and ensure the atomicity and durability of data operations. Therefore, in the storage field, electronic devices usually adopt the WAL mechanism for data storage. That is, first, the data (such as the data generated by operations like adding, deleting, and modifying file content) and the corresponding message headers are written into the buffer in the form of logs, and then the data and message headers in the buffer are written into the WAL files in the memory, so as to apply the log-formatted data and message headers to a persistent and stable memory (such as a hard disk drive (HDD)).

[0096] This mode reduces the number of times of writing data into the memory (also known as the database). Especially when dealing with a large number of transaction operations, compared with the way of real-time synchronizing the database, the method of using WAL for data storage has higher efficiency. In addition, WAL also plays a key role in the disaster recovery process and can restore the data and system state by replaying the operations in the WAL files after a system failure.

[0097] In the buffer, the message headers and data are stored in sequence. A message header and the corresponding data can be called a log record, which is the smallest unit of the transaction log. In the memory, the area for storing the message headers and data can be called the WAL area. The WAL area includes WAL logs, and the WAL logs include multiple WAL files. Each WAL file includes multiple log pages, and each log page consists of a message header and data, storing a log record. That is, the message headers and data are stored in the WAL files in the form of log pages.

[0098] As Figure 1 shown in the schematic diagram of writing the message headers and data into the memory, the electronic device writes the data (i.e., WAL data) and message headers included in the log records of the Lock Database File (LDB) part in the buffer into the WAL files in the data region of the memory (such as a hard disk) through the Binary Large Object (blob) File System (FS). The hard disk also includes a Super block and a blob metadata region (abbreviated as the metadata region).

[0099] In the prior art, the length of a message header is 12B, including a Cyclic Redundancy Check (CRC) field, a Length (L) field, and a magic number field, each field occupying 4B. However, in a memory, a message header needs to occupy a sector (such as an HDD sector), and the length of a sector is 4KB or 512B, that is, the length of a sector far exceeds the length of a message header. For example, Figure 1 in which each message header occupies a sector, and the WAL data corresponding to each message header is stored after that sector.

[0100] The above situation results in low utilization rate of the storage space occupied by the WAL file, and further leads to the ineffective use of the available space in the memory. In addition, the low utilization rate of the WAL area in the memory where the WAL file is stored means that the WAL file (i.e., the log file) will be filled up faster, resulting in more frequent WAL file switches, and further leading to additional memory Input / Output (I / O) operations. For example, operations such as updating the file system metadata, creating a new WAL file, and closing an old WAL file. These additional I / O operations will increase the burden on the memory, especially in an I / O-intensive environment. If the WAL file switches too frequently, it may lead to a longer recovery time. After the system crashes, the recovery process needs to process and apply all relevant WAL files, and frequent switches may result in more files to be processed, thus prolonging the recovery time.

[0101] To solve the above problems, an embodiment of the present application provides a data writing method, which is applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. In the embodiment of the present application, the electronic device may be a computer, a server, etc., which is not limited herein. The memory of the electronic device may be any storage device, such as an HDD, a Solid State Disk (SSD), etc., which is not limited herein. The WAL file may be any WAL file in the WAL area of the memory, and each WAL file can be divided into a message header area for storing the message header and a data area for storing the data. The WAL file in the embodiment of the present application may be the WAL file to which the electronic device currently writes data and the message header, which is not limited herein.

[0102] See Figure 2 , Figure 2 which is the first process schematic diagram of the data writing method provided by the embodiment of the present application, applied to an electronic device. The WAL file in the memory of the electronic device includes a header area and a data area. The above data writing message method includes the following steps.

[0103] Step S21: Obtain the first data to be written and the first message header corresponding to the first data.

[0104] Step S22: Obtain the first storage location of the first data in the data area, and obtain the second storage location of the first message header in the message header area.

[0105] Step S23: Write the first data to the first storage location.

[0106] Step S24: Add indication information to the first message header to obtain a second message header, where the indication information indicates the first storage location.

[0107] Step S25: Write the second message header to the second storage location.

[0108] In the technical solution provided by the embodiments of the present application, the WAL file is divided into a message header area and a data area. The message header area is used to write message headers, and the data area is used to write data. The electronic device writes data to the data area, adds indication information indicating the storage location of the data in the data area to the message header to obtain a new message header, and writes the new message header to the message header area. In this way, the message headers can be centrally stored in the message header area, and the storage location of the message header in the message header area and the storage location of the data corresponding to the message header in the data area can be accurately determined, reducing the number of sectors occupied by storing the message headers. Compared with storing by log pages, the utilization rate of the storage space occupied by the WAL file is improved, and thus the effective space of the memory is fully utilized.

[0109] In the above step S21, the first data is any data to be written to the WAL file, and the first message header is the message header corresponding to the first data to be written to the WAL file.

[0110] The electronic device can determine the next log record to be written in the buffer, read the data included in the log record from the buffer as the first data, and read the message header included in the log record as the first message header. Here, the manner in which the electronic device obtains the first data and the first message header is not limited.

[0111] In the above step S22, the first storage location is the starting position where the first data is to be written to the data area, and the second storage location is the starting position where the first message header is to be written to the message header area.

[0112] In the embodiments of the present application, the length of the message header area in the WAL file is the length of the sectors occupied by the message header area. For ease of description, the length of the message header area will be referred to as the second length (denoted as wal_header_size) hereinafter. To ensure that all message headers included in the WAL file can be written to the message header area, the second length can be greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0113] Before the electronic device writes data (i.e., before executing step S21), it can initialize the WAL file, and calculate the second length in the following manner during the initialization process.

[0114] Method 1: The electronic device can obtain a preset WAL file length, a preset log record length (which can also be referred to as a preset log page length), and a preset message header length. The preset WAL file length is the preset length of a WAL file, which can be expressed as wal_segment_size. The preset log record length is the preset maximum length of a log record, which can be expressed as wal_page_size. The preset message header length is the preset length of a message header. The message header will be described in detail later, and the preset message header length will not be elaborated here for the time being.

[0115] The preset WAL file length, the preset log record length, and the preset message header length can be defined in a configuration file. The electronic device can obtain the configuration file and extract each length from the configuration file. The method for the electronic device to obtain each length is not limited here.

[0116] The electronic device can calculate the ratio of the preset WAL file length to the preset log record length to obtain the maximum number of log records that a WAL file can store. This maximum number is also the maximum number of message headers included in a WAL file. The electronic device then calculates the product of the maximum number and the preset message header length to obtain the maximum total length of the message headers included in a WAL file.

[0117] For example, when the preset WAL file length is 16 MegaBytes (MB), the preset log record length is 8 KB, and the preset message header length is 16 B, the electronic device can calculate 16 MB / 8 KB to obtain the maximum number of message headers as 2000, and then calculate 2000 × 16 B to obtain the maximum total length of the message headers as 32 KB. The values of the above lengths are only examples and do not serve as limitations.

[0118] The electronic device can use the calculated maximum total length as the second length, or use any length greater than the maximum total length as the second length.

[0119] Method 2: After the electronic device calculates the maximum total length through Method 1, it can calculate the ratio of the maximum total length to the preset sector length (i.e., the fixed length of a sector in the memory), and calculate the ceiling of this ratio to obtain the number of sectors occupied by the message header area, and then calculate the product of the number of sectors occupied by the message header area and the preset sector length.

[0120] For example, when the preset sector length is 4 KB and the maximum total length of the message header is 32 KB, the electronic device can calculate 32 KB / 4 KB to obtain a ratio of 8, round up the ratio 8 to obtain the number of sectors occupied by the message header area as 8, and then calculate 8 × 4 KB to obtain 32 KB. The values of the above lengths are only examples and do not play a limiting role.

[0121] The electronic device can use the calculated product as the second length, or can use any length greater than the product as the second length. The method for calculating the second length is not limited herein.

[0122] By calculating the second length, the electronic device determines the length occupied by the message header area in the WAL file, and realizes the division of the message header area and the data area. For example, the storage space of the first second length in the WAL file is the message header area, and the remaining storage space in the WAL file is the data area.

[0123] In the embodiment of the present application, during the initialization process, the electronic device can also obtain the starting position of the message header area (denoted as position_header0), and combine the calculated second length and the preset WAL file length to determine the positions of the message header area and the data area. The starting position of the message header area is also the starting position of the current WAL file, and can be any idle position in the preset WAL area, which is not limited herein. The starting position of the message header area can also be defined in the configuration file, and the electronic device can extract the starting position of the message header area from the configuration file. The method for the electronic device to obtain the starting position of the message header area is not limited herein.

[0124] The electronic device can calculate the position after offsetting the starting position of the message header area by the second length backward to obtain the starting position of the data area (denoted as position_data0), that is, position_data0 = position_header0 + wal_header_size. The starting position of the data area is also the end position of the message header area. The electronic device can also calculate the position after offsetting the starting position of the message header area by the preset WAL file length backward to obtain the end position of the current WAL file (denoted as position_end), position_end = position_header0 + wal_segment_size.

[0125] In the embodiments of the present application, the electronic device may randomly determine an unwritten position from the positions indicating the data area (such as the position between the aforementioned position_data0 and position_end), and use this position as the first storage position; similarly, the electronic device may also randomly determine an unwritten position for the message header from the positions indicating the message header area (such as the position between the aforementioned position_header0 and position_data0), and use this position as the second storage position.

[0126] When a message header has been written in the message header area, the electronic device may determine the end position of the last written data (for ease of description, hereinafter this data will be simply referred to as the third data), and determine a position from the positions in the data area after the end position of the third data as the first storage position; similarly, the electronic device may also determine the end position of the last written message header (for ease of description, hereinafter this message header will be simply referred to as the fourth message header), and determine a position from the positions in the message header area after the end position of the fourth message header as the second storage position.

[0127] When no message header has been written in the message header area, the electronic device may randomly determine a position from the positions indicating the data area as the first storage position; similarly, the electronic device may also randomly determine a position from the positions indicating the message header area as the second storage position. The manner and order in which the electronic device determines the first storage position and the second storage position are not limited herein.

[0128] In the above step S23, after determining the first storage position, the electronic device may start from the first storage position and write the first data into the data area.

[0129] In the above step S24, the second message header is the message header corresponding to the first data and carrying indication information indicating the first storage position. The indication information may be the first storage position or the offset between the first storage position and the second storage position (for ease of description, hereinafter simply referred to as the first length), and this is not limited.

[0130] In the embodiments of the present application, after writing the first data, the electronic device may generate indication information and add the generated indication information to the first message header, and the message header after addition is the second message header.

[0131] In the above step S25, after determining the second storage position, the electronic device may start from the second storage position and write the second message header into the message header area.

[0132] In the embodiments of the present application, after writing the first data into the data area, the electronic device may also obtain a second storage location, and after obtaining the second storage location and generating a second message header, perform step S25 to write the second message header into the message header area. Here, the order of the electronic device writing the first data and obtaining the second storage location, and obtaining the second storage location and generating the second message header is not limited.

[0133] The electronic device may repeatedly execute the above steps S21 to S25 to obtain a log record to be written from the buffer, and write the data included in the obtained log record into the data area of the WAL file, and write the included message header into the message header area of the WAL file. By repeatedly executing the above steps S21 to S25 multiple times, the electronic device realizes writing multiple log records in the buffer into the memory.

[0134] In some embodiments, the electronic device may obtain the first storage location through the following steps: if no data is written in the data area, obtain the starting position and the second length of the message header area, and offset backward by the second length from the starting position to obtain the first storage location; if data has been written in the data area, obtain the third storage location and the third length of the last written data, and offset backward by the third length from the third storage location to obtain the first storage location.

[0135] In the embodiments of the present application, the electronic device determines whether data has been written in the data area to determine whether the first data is the first data to be written into the WAL file, and further determines the first storage location.

[0136] The electronic device may maintain a write counter (which may be denoted as wcounter) to count the number of data and message headers written into the WAL file. Each time the electronic device writes a data and the corresponding message header, the write counter is incremented by one.

[0137] The electronic device may determine whether data has been written in the data area according to the value of the write counter. When the value of the write counter is 0, the electronic device may determine that no data has been written in the data area; when the value of the write counter is greater than 0 (such as 1, 2, 3...), the electronic device may determine that data has been written in the data area. Here, the manner in which the electronic device determines whether data has been written in the data area is not limited.

[0138] If no data is written in the data area, the electronic device determines that the first data is the first data to be written into the WAL file, may obtain the preset starting position of the message header area and the pre-calculated second length, and calculate the position after offsetting backward by the second length from the starting position of the message header area, that is, calculate the starting position of the data area, and use the calculated position as the first storage location.

[0139] To improve the writing efficiency, the electronic device can also record the starting position of the calculated data area in the configuration file during the initialization process. In this way, the electronic device can directly obtain the starting position of the data area from the configuration file as the first storage position, and there is no limitation on this.

[0140] If data has been written in the data area, and the electronic device determines that the first data is not the first data to be written into the WAL file, it can obtain the storage position of the third data in the data area (i.e., the third storage position) determined when writing the third data, and obtain the length of the third data (i.e., the third length) from the message header corresponding to the third data, and calculate the position after offsetting the third length backward at the third storage position, that is, calculate the end position of the third data.

[0141] In the embodiments of the present application, the electronic device can directly use the calculated position as the first storage position, or can use any position after the calculated position as the first storage position, and there is no limitation on this.

[0142] In the technical solution provided by the embodiments of the present application, when data has been written in the data area, the electronic device determines the starting position of the data area as the first storage position, and when no data has been written in the data area, it determines the end position of the last written data as the first storage position, so as to make full use of the storage space and further improve the utilization rate of the WAL area.

[0143] In some embodiments, the electronic device can obtain the second storage position through the following steps: obtain the first quantity of the written message headers; calculate the product of the first quantity and the preset message header length to obtain the fourth length; offset the fourth length backward at the starting position of the message header area to obtain the second storage position.

[0144] In the embodiments of the present application, the electronic device counts the quantity of the message headers written into the WAL file (i.e., the first quantity), and obtains the first quantity. For example, the electronic device can obtain the value of the maintained write counter and determine this value as the first quantity.

[0145] After obtaining the first quantity, the electronic device can calculate the product of the first quantity and the preset message header length to obtain the total length of the written message headers, and this total length is the fourth length. The electronic device can calculate the position after offsetting the fourth length backward at the starting position of the message header area, that is, calculate the end position of the last written message header (i.e., the fourth message header).

[0146] The electronic device can directly use the calculated position as the second storage position, or can use any position after the calculated position as the second storage position, and there is no limitation on this.

[0147] In the embodiment of the present application, when the first quantity is 0, the fourth length is also 0. The position after offsetting the fourth length backward from the starting position of the message header area is still the starting position of the message header area. In this case, the second storage position is the starting position of the message header area. To simplify the calculation process, the electronic device can directly use the starting position of the message header area as the second storage position.

[0148] In the embodiment of the present application, the electronic device can also determine whether a message header has been written in the message header area according to the first quantity, and obtain the second storage position according to the judgment result.

[0149] When the value of the write counter is greater than 0, the first quantity is greater than 0. The electronic device determines that a message header has been written in the message header area, and can obtain the storage position of the fourth message header in the message header area (i.e., the fifth storage position) determined when writing the fourth message header, and calculate the position after offsetting the preset message header length backward from the fifth storage position, that is, calculate the end position of the fourth message header.

[0150] The electronic device can directly use the calculated position as the second storage position, or can use any position after the calculated position as the second storage position, and there is no limitation on this.

[0151] When the value of the write counter is 0, the first quantity is 0. The electronic device determines that no message header has been written in the message header area, and can directly use the starting position of the message header area as the second storage position.

[0152] In the technical solution provided by the embodiment of the present application, the electronic device calculates the storage position of the next message header according to the number of written message headers and the preset message header length, so as to make full use of the storage space and further improve the utilization rate of the WAL area.

[0153] In some embodiments, the indication information is the offset between the first storage position and the second storage position, that is, the first length (denoted as position). On this basis, see Figure 3 , Figure 3 is a refined schematic diagram of step S24 provided by the embodiment of the present application. The above step S24 may include the following steps.

[0154] Step S31, calculate the first length between the first storage position and the second storage position to obtain the indication information.

[0155] Step S32, write the indication information into the first message header to obtain the second message header.

[0156] In the technical solution provided by the embodiment of the present application, the electronic device uses the offset between the first storage position and the second storage position as the indication information, adds the offset to the first message header, and generates a new message header, which is convenient for determining the storage position where the first data is written.

[0157] In the above step S31, after the electronic device determines the first storage location and the second storage location, it can calculate the difference between the second storage location and the first storage location, and this difference is the first length. The electronic device uses the obtained first length as the indication information to be added.

[0158] In the embodiment of the present application, the electronic device can also determine whether a message header has been written in the message header area and calculate the first length according to the historical offset.

[0159] When it is determined that a message header has been written in the message header area, the electronic device can obtain the offset between the third storage location and the fifth storage location calculated when writing the third data and the fourth message header (for the sake of easy description, simply referred to as the eighth length), and calculate the sum value of the eighth length and the third length, as well as the difference between this sum value and the preset message header length, and use the calculated difference as the first length.

[0160] For example, the electronic device can calculate position + length (i.e., the third length) - 16B (i.e., the preset message header length) according to the first length position (referred to as the eighth length during this calculation) calculated when writing the message header and data last time, that is, calculate position = position + length - 16B, update position to position + length - 16B, and obtain the first length.

[0161] When it is determined that no message header has been written in the message header area, the offset between the start position of the data area and the start position of the message header area is the second length (i.e., the length of the message header area), and the electronic device can directly use the second length as the first length.

[0162] In the case of calculating the first length according to the historical offset, since the electronic device does not need to use the second storage location, the electronic device can first calculate the first length and then obtain the second storage location. For example, the electronic device can first obtain the first storage location, calculate the first length, and then obtain the second storage location, or it can first calculate the first length, obtain the first storage location, and then obtain the second storage location. The order in which the electronic device obtains the first storage location, calculates the first length, and obtains the second storage location is not limited herein.

[0163] In the above step S32, the electronic device can add indication information to the first message header, that is, add the first length to the first message header to generate a new message header, and this new message header is the second message header. For example, the electronic device can add the first length to the head or tail of the first message header, etc., and the comparison is not limited.

[0164] In the embodiments of the present application, the first message header may include a cyclic redundancy check field, a data length field, and a magic number field, and the lengths of these three fields are all 4B. The electronic device may add an offset field for carrying an offset in the first message header and write indication information in the offset field, that is, write the first length in the offset field to generate a second message header.

[0165] The length of the offset field may be 4B, 8B, etc. When the length of the offset field is 4B, the preset message header length may be 16B; when the length of the offset field is 8B, the preset message header length may be 20B. For ease of description, in the embodiments of the present application, the case where the offset field is 4B is taken as an example for illustration, and this is not limited.

[0166] In the embodiments of the present application, the offset field may be located at any position in the message header. For example, the offset field may be located after the magic number field, or may be located between the cyclic redundancy check field and the data length field. For ease of description, in the embodiments of the present application, the case where the offset field is located after the magic number field is taken as an example for illustration, and this is not limited.

[0167] In the technical solution provided by the embodiments of the present application, when the preset sector length is 4KB and the preset message header length is 16B, a sector can accommodate at most 4K / 16 = 250 message headers. That is, multiple message headers can be written in one sector. Compared with the prior art solution where one message header occupies one sector, the technical solution provided by the embodiments of the present application greatly improves the sector utilization rate.

[0168] In some embodiments, refer to Figure 4 , Figure 4 which is the second process schematic diagram of the data writing method provided by the embodiments of the present application. The above data writing message method may include the following steps.

[0169] Step S41, obtain the first data to be written and the first message header corresponding to the first data. This is the same as step S21 above.

[0170] Step S42, obtain the first storage position of the first data in the data area and obtain the second storage position of the first message header in the message header area. This is the same as step S22 above.

[0171] Step S43, determine the fourth storage position after offsetting backward by the fifth length at the first storage position, where the fifth length is the maximum length of one data. When the electronic device determines that the fourth storage position is within the data area, execute step S44.

[0172] Step S44, write the first data into the first storage position. This is the same as step S23 above.

[0173] Step S45: Add indication information to the first message header to obtain a second message header, where the indication information indicates the first storage location. This is the same as step S24 above.

[0174] Step S46: Write the second message header to the second storage location. This is the same as step S25 above.

[0175] In the technical solution provided by the embodiments of the present application, before writing data, the electronic device first determines whether the data can be written into the WAL file to ensure that the data can be completely written.

[0176] In the above step S43, the fourth storage location is the location after writing the first data. Before writing the first data, the electronic device can first calculate the maximum length of a piece of data, that is, the fifth length. The electronic device can obtain the preset log record length and the preset message header length, and calculate the difference between the preset log record length and the preset message header length, and this difference is the fifth length. For example, when the preset log record length is wal_page_size and the preset message header length is 16B, the fifth length can be wal_page_size - 16B.

[0177] The electronic device can also calculate the fifth length during the initialization process and record the fifth length in the configuration file. When executing step S43, the electronic device can directly extract the fifth length from the configuration file to improve the data writing efficiency.

[0178] The electronic device can calculate the position after offsetting the fifth length backward at the first storage location, that is, calculate the farthest end position after writing the first data, and the calculated position is the fourth storage location.

[0179] In the embodiments of the present application, the electronic device can also obtain the length of the first data (for the convenience of description, simply referred to as the ninth length) from the first message header, such as extracting the ninth length from the data length field in the first message header. The electronic device calculates the position after offsetting the ninth length backward at the first storage location as the fourth storage location, and this is not limited.

[0180] After obtaining the fourth storage location, the electronic device can determine whether the fourth storage location is within the data area to determine whether the first data can be written into the WAL file.

[0181] In the embodiments of the present application, the electronic device can use the positional relationship between the fourth storage location and the end position of the WAL file to determine whether the fourth storage location is within the data area.

[0182] The electronic device can calculate the position after offsetting a preset WAL file length backward from the starting position of the message header area to obtain the end position of the WAL file. To improve the writing efficiency, the electronic device can also record the end position of the WAL file obtained during the initialization process in the configuration file. In this way, the electronic device can directly obtain the end position of the WAL file from the configuration file, and this is not limited hereby.

[0183] The electronic device can determine whether the fourth storage position is less than or equal to the end position of the WAL file. When the fourth storage position is less than or equal to the end position of the WAL file, it is determined that the fourth storage position is within the data area; when the fourth storage position is greater than the end position of the WAL file, it is determined that the fourth storage position is outside the data area.

[0184] When the fourth storage position is within the data area, the first data can be written into the WAL file, and then the electronic device can execute step S44 to write the first data into the first storage position.

[0185] When the fourth storage position is outside the data area, the first data cannot be written into the WAL file. Then the electronic device can update the file system metadata, create a new WAL file, close the current WAL file, etc., use the new WAL file as the WAL file for currently writing data and message headers, and re - execute step S42 to obtain the first storage position and the second storage position of the first data and the first message header in the new WAL file.

[0186] In the embodiment of the present application, if the electronic device has obtained the first length before executing step S43, the electronic device can also use the first length to determine whether the first data can be written into the WAL file.

[0187] The electronic device can calculate the product of the first quantity and the preset message header length to obtain the total length of the written message headers (i.e., the first total length, such as wcounter×16B), calculate the sum value of the first total length and the first length to obtain the total length of the message header area and the written data (i.e., the second total length, such as position + wcounter×16B), then calculate the sum value of the second total length and the fifth length to obtain the total length of the message header area and the data after writing the first data (i.e., the third total length, such as position + wcounter×16B+wal_page_size - 16B), and determine whether the third total length is less than or equal to the preset WAL file length.

[0188] When the third total length is less than or equal to the preset WAL file length, the first data can be written into the WAL file; when the third total length is greater than the preset WAL file length, the first data cannot be written into the WAL file. The method for the electronic device to determine whether the first data can be written into the WAL file is not limited hereby.

[0189] As Figure 5 shown in the schematic diagram of writing the message header and data into the memory, the electronic device writes the data (i.e., WAL data) and the message header included in the log record of the LDB part in the buffer into the WAL file in the data partition of the memory through blobFS. The WAL file includes a message header area and a data area. The message header area stores multiple message headers, and each message header includes CRC, Len, magic number, and offset (i.e., displacement). The data area stores the corresponding multiple data. Figure 5 The lengths of the message header area, the lengths of each field, and the length of the data area in

[0190] Applying the technical solution provided by the embodiment of the present application, the structure of the message header is modified, and an offset field is added on the basis of the original message header to indicate the storage position of the data in the data area. In addition, by writing multiple message headers into the message header area centrally, the utilization rate of the WAL area can be effectively improved, so that multiple message headers can be written into one sector. In this way, the WAL area can accommodate more log records, reduce the switching frequency of the WAL file, and further reduce the performance overhead caused by the WAL file switching, thereby improving the writing efficiency of the system. At the same time, more historical log records are allowed to be retained in the WAL area, which can provide more historical log records in the scenarios where long-term log record analysis or recovery is required, facilitating the device to perform analysis.

[0191] Corresponding to the above data writing method, the embodiment of the present application also provides a data reading method, which is applied to an electronic device. The electronic device for data writing and the electronic device for data reading can be the same electronic device or different electronic devices. The WAL file in the memory of the electronic device includes a message header area and a data area. In the embodiment of the present application, the WAL file can be obtained by using the above Figures 2 to 4 shown data writing method, and it is the WAL file for the electronic device to currently read data and message headers. For specific details, please refer to the relevant description in the above data writing part.

[0192] Refer to Figure 6 , Figure 6 which is a schematic flowchart of the data reading method provided by the embodiment of the present application, applied to an electronic device. The WAL file in the memory of the electronic device includes a header area and a data area. The above data reading message method includes the following steps.

[0193] Step S61, obtain the first reading position of the message header area.

[0194] Step S62, read the third message header from the first reading position. The third message header indicates the second reading position of the second data corresponding to the third message header in the data area.

[0195] Step S63: Determine a second reading position according to the indication information included in the third message header.

[0196] Step S64: Read second data from the second reading position.

[0197] In the technical solution provided by the embodiments of the present application, the WAL file is divided into a message header area and a data area. The message header area is used to write message headers, and the data area is used to write data. The electronic device reads the message header in the message header area, and reads the data according to the indication information indicating the storage position of the data in the data area in the message header. In this way, the message headers can be centrally stored in the message header area, and the storage position of the message headers in the message header area and the storage position of the data corresponding to the message headers in the data area can be accurately determined, reducing the number of sectors occupied by storing the message headers. Compared with storing by log pages, the utilization rate of the storage space occupied by the WAL file is improved, and thus the effective space of the memory is fully utilized.

[0198] In the above step S61, the first reading position is the starting position of an unread message header in the message header area, and the message header at the first reading position is the third message header. The electronic device obtains the starting position of an unread message header in the message header area, and this position is the first reading position. In the embodiments of the present application, the reading position and the storage position are only distinguished in name, and actually both refer to an actual position in the memory.

[0199] In the above step S62, the data corresponding to the third message header is the second data, and the starting position where the second data is stored in the data area is the second reading position.

[0200] The electronic device can start from the first reading position and read data with a length of a preset message header length, and the read data is the third message header.

[0201] In the above step S63, the third message header includes indication information indicating the second reading position. The indication information can be the second reading position, or can be an offset between the second reading position and the first reading position (for the convenience of description, hereinafter simply referred to as the sixth length), and this is not limited. The manner of adding the indication information in the third message header can be specifically referred to the process of the electronic device adding the indication information in the second message header above, and will not be elaborated here.

[0202] The electronic device can read the indication information from the third message header, and determine the position indicated by the indication information as the starting position where the second data is stored in the data area, to obtain the second reading position.

[0203] In the above step S64, the electronic device may determine the length of the second data (abbreviated as the tenth length for ease of description) according to other information included in the third message header, and starting from the second reading position, read the data of the tenth length. The read data is the second data. In the embodiments of the present application, the third message header may include a data length field, and the electronic device may extract data from the data length field of the third message header to obtain the tenth length. The method for determining the tenth length is not limited herein.

[0204] In the embodiments of the present application, to ensure that all message headers included in the WAL file can be written to the message header area, the length of the sector occupied by the message header area (i.e., the second length) may be greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0205] In the embodiments of the present application, the electronic device may execute the relevant steps in the above data reading part for post-fault data recovery, data verification, etc. The conditions for the electronic device to perform data reading are not limited herein.

[0206] In the embodiments of the present application, the third message header may further include a cyclic redundancy check field and a magic number field. After the electronic device reads the second data, it may read the cyclic redundancy check code and the magic number from the cyclic redundancy check field and the magic number field respectively, determine whether the magic numbers match, and determine whether the cyclic redundancy check code passes the verification to determine whether the second data is successfully read. When the second data is not successfully read, the electronic device may end the reading of the current WAL file and return a prompt message indicating a reading failure. When the second data is successfully read, the electronic device may execute the above steps S61 to S64 and continue to read the message header and the corresponding data from the current WAL file.

[0207] The electronic device realizes reading multiple message headers and the corresponding data from the memory by repeatedly executing the above steps S61 to S64. In the embodiments of the present application, when the electronic device has read all the message headers and the corresponding data included in the WAL file, it ends the reading of the current WAL file and returns a prompt message indicating that the reading is completed. The electronic device may use the next WAL file as the current WAL file for continued reading, which is not limited herein.

[0208] In some embodiments, during the process of writing data, when the electronic device determines the second storage position by using the first quantity and the preset message header length, the above step S61 may be implemented through the following steps: obtaining the second quantity of the read data; calculating the product of the second quantity and the preset message header length to obtain the seventh length; and offsetting backward by the seventh length from the starting position of the message header area to obtain the first reading position.

[0209] The electronic device counts the number of read data (i.e., the second quantity) and obtains the second quantity. In the embodiments of the present application, the number of read message headers is the same as the number of read data, which is also the second quantity. The electronic device can maintain a read counter (which can be denoted as rcounter) to count the number of read data and message headers. After the electronic device reads each message header and the corresponding data, the read counter is incremented by one. The electronic device can obtain the value of the maintained read counter and determine this value as the second quantity.

[0210] After obtaining the second quantity, the electronic device can calculate the product of the second quantity and the preset message header length to obtain the total length of the read message headers, and this total length is the seventh length. The electronic device can calculate the position after offsetting the starting position of the message header area by the seventh length, that is, calculate the end position of the last read message header (for ease of description, simply referred to as the fifth message header), and use the calculated position as the first read position.

[0211] In the embodiments of the present application, the electronic device can also determine a position with a message header from the positions after the calculated position and use this position as the first read position, and this is not limited herein.

[0212] Applying the technical solution provided by the embodiments of the present application, the electronic device calculates the storage position of the next message header to be read according to the number of read data and the preset message header length, so as to make full use of the storage space and further improve the utilization rate of the WAL area.

[0213] In some embodiments, the indication information is the offset between the second read position and the first read position, that is, the sixth length. The above step S63 can be implemented through the following steps: Obtain the indication information from the third message header to get the sixth length, and the sixth length is the offset between the second read position and the first read position; offset the first read position by the sixth length backward to obtain the second read position.

[0214] The electronic device can extract the indication information from the third message header, and the extracted indication information is the sixth length. In the embodiments of the present application, the third message header may include an offset field, and the electronic device can extract the indication information from the offset field of the third message header. The method for obtaining the indication information is not limited herein.

[0215] After obtaining the sixth length, the electronic device can calculate the storage position of the second data in the data area after offsetting the first read position by the sixth length to obtain the second read position.

[0216] In the technical solution provided by the embodiments of the present application, the message header includes the offset between the second read position and the first read position. After determining the read position of the message header, the electronic device can directly determine the read position of the data according to the offset, improving the data read efficiency.

[0217] In some embodiments, after the electronic device executes step S61 above and obtains the first read position, it can determine whether there is a message header at the first read position to avoid data unreadable. In the embodiments of the present application, the electronic device can determine whether there is a message header at the first read position by determining whether there is data at the first read position.

[0218] When there is data at the first read position, there is a message header at the first read position, then the electronic device can execute step S62 above and read the third message header from the first read position; when there is no data at the first read position, there is no message header at the first read position, then the electronic device can determine the next unread position as the first read position, or determine that the current WAL file reading is completed, return the reading completion result or continue to read the next WAL file, which is not limited herein.

[0219] The following Figures 7 to 8 describes in detail the data writing and data reading methods provided by the embodiments of the present application.

[0220] Figure 7 FIG. is a schematic flowchart of a data writing provided by an embodiment of the present application, including the following steps.

[0221] Step S71, initialize the WAL file.

[0222] In the embodiments of the present application, the electronic device obtains the length of a preset WAL file (i.e., the preset WAL file length) wal_segment_size, the length of a log page (i.e., the preset log record length) wal_page_size, and the length of the message header area in the WAL file (i.e., the second length) wal_header_size from the configuration file.

[0223] Step S72, initialize the write counter.

[0224] In the embodiments of the present application, the electronic device initializes the write counter wcounter = 0.

[0225] Step S73, obtain the initial write position of the data and the write position of the message header.

[0226] In the embodiment of the present application, the electronic device obtains the initial writing position of the message header from the configuration file (i.e., the starting position of the message header area, which is also the starting position of the current WAL file) position_header0, and calculates the initial writing position of the data area (i.e., the starting position of the data area) position_data0 = position_header0 + wal_header_size.

[0227] Step S74, write data.

[0228] In the embodiment of the present application, the electronic device writes the data obtained from the buffer to the initial writing position position_data0.

[0229] Step S75, calculate the offset and update the write counter.

[0230] In the embodiment of the present application, the electronic device calculates the offset position = wal_header_size, updates the write counter wcounter = 1, and adds the offset to the initial message header obtained from the buffer to obtain the message header after adding the offset.

[0231] Step S76, write the message header.

[0232] In the embodiment of the present application, the electronic device writes the message header after adding the offset to the current writing position position_header. When writing for the first time, position_header = position_header0.

[0233] Step S77, calculate the writing position and offset of the next data.

[0234] In the embodiment of the present application, the electronic device calculates the writing position of the next data position_data = position_data (i.e., the writing position of the last data) + length (i.e., the length of the last written data, which is also the third length), and the offset of the next data position = position (i.e., the offset of the last data) + length - 16B (i.e., the preset message header length).

[0235] Step S78, determine whether the occupied storage space after writing the data is less than the length of the WAL file. If so, execute step S79; if not, execute step S712.

[0236] In an embodiment of the present application, the electronic device calculates position + wcounter × 16B + wal_page_size - 16B, and determines whether this value is less than wal_segment_size. If so, it is determined that the storage space occupied after writing the data is less than the length of the WAL file, and step S79 is executed; if not, it is determined that the storage space occupied after writing the data is greater than or equal to the length of the WAL file, and step S712 is executed.

[0237] Step S79: Write the data.

[0238] In an embodiment of the present application, the electronic device writes the data to the current writing position position_data.

[0239] Step S710: Calculate the writing position of the next message header.

[0240] In an embodiment of the present application, the electronic device calculates the writing position of the next message header position_header = position_header0 + wcounter × 16B, and adds an offset to the initial message header obtained from the buffer to obtain the message header after adding the offset.

[0241] Step S711: Update the write counter, and return to execute step S76.

[0242] In an embodiment of the present application, the electronic device updates the write counter wcounter = wcounter + 1.

[0243] Step S712: Determine that the WAL file has been written completely.

[0244] In an embodiment of the present application, the electronic device determines that the current WAL file has been written, and ends this writing. The electronic device can also create a new WAL file, and re - execute steps S71 to S712 to continue writing log records to the new WAL file.

[0245] Figure 8 It is a schematic flowchart of a data reading process provided by an embodiment of the present application, including the following steps.

[0246] Step S81: Read the disk data of the blob corresponding to the WAL file.

[0247] In an embodiment of the present application, the electronic device reads the WAL file in the memory.

[0248] Step S82: Initialize the read counter.

[0249] In an embodiment of the present application, the electronic device initializes the read counter rcounter = 0.

[0250] Step S83, read the message header.

[0251] In the embodiment of the present application, the electronic device obtains the reading position position_header of the message header and reads the message header. When reading for the first time, position_header = position_header0 (i.e., the starting position of the currently read WAL file, which is also the starting position of the message header area).

[0252] Step S84, locate the data position.

[0253] In the embodiment of the present application, the electronic device extracts the offset position from the read message header and calculates the data position (i.e., the second reading position) position_header + position.

[0254] Step S85, parse the data to obtain the magic number.

[0255] In the embodiment of the present application, the electronic device reads the data from the data position and obtains the magic number from the magic number field of the message header.

[0256] Step S86, determine whether the magic numbers match. If so, execute step S87; if not, end the reading.

[0257] In the embodiment of the present application, the electronic device determines whether the magic numbers match according to the read data and the magic number in the message header.

[0258] Step S87, obtain the check data CRC.

[0259] In the embodiment of the present application, the electronic device obtains the CRC from the cyclic redundancy check field of the message header.

[0260] Step S88, determine whether the CRC check is successful. If so, execute step S89; if not, end the reading.

[0261] In the embodiment of the present application, the electronic device determines whether the CRC check is successful according to the read data and the CRC in the message header.

[0262] Step S89, determine that the data reading is completed and increment the read counter by one.

[0263] In the embodiment of the present application, the electronic device calculates the read counter rcounter = rcounter + 1.

[0264] Step S810, locate the next message header position.

[0265] In the embodiment of the present application, the electronic device calculates the reading position position_header of the next message header as position_header = position_header0 + rcounter × 16B.

[0266] Step S811, determine whether there are more headers. If yes, return to execute step S83; if no, end the reading.

[0267] In the embodiment of the present application, the electronic device determines whether there is a message header at the next message header position. If yes, return to execute step S83; if no, determine that the current WAL file reading is completed and end the reading.

[0268] In the technical solution provided by the embodiment of the present application, by centrally writing multiple message headers of the WAL file in the distributed storage system, the utilization rate of the WAL area can be effectively improved, which allows more log data to be recorded in the same WAL file, thereby reducing the frequency of switching to a new WAL file and reducing the short-term performance fluctuations that may be caused by the switching of the WAL file. And improving the utilization rate of the WAL area means that the switching occurs less frequently, so a smoother and more stable writing performance can be provided. Moreover, a higher utilization rate of the WAL area means that the system can retain more historical log data, which is very useful for scenarios that require long-term log analysis or recovery.

[0269] Corresponding to the above data writing method, the embodiment of the present application provides a data writing device, see Figure 9 , which is applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. The device includes:

[0270] The first acquisition module 91 is used to acquire the first data to be written and the first message header corresponding to the first data;

[0271] The second acquisition module 92 is used to acquire the first storage position of the first data in the data area and acquire the second storage position of the first message header in the message header area;

[0272] The first writing module 93 is used to write the first data to the first storage position;

[0273] The adding module 94 is used to add indication information to the first message header to obtain a second message header, and the indication information indicates the first storage position;

[0274] The second writing module 95 is used to write the second message header to the second storage position.

[0275] In the technical solution provided by the embodiment of the present application, the WAL file is divided into a message header area and a data area. The message header area is used to write the message header, and the data area is used to write data. The electronic device writes data in the data area, adds indication information indicating the storage position of the data in the data area to the message header to obtain a new message header, and writes the new message header in the message header area. In this way, the message headers can be centrally stored in the message header area, and the storage position of the message header in the message header area and the storage position of the data corresponding to the message header in the data area can be accurately determined, reducing the number of sectors occupied by storing the message headers. Compared with storing by log pages, the utilization rate of the storage space occupied by the WAL file is improved, and thus the effective space of the memory is fully utilized.

[0276] In some embodiments, the adding module 94 is specifically configured to:

[0277] Calculate a first length between the first storage position and the second storage position to obtain the indication information;

[0278] Write the indication information into the first message header to obtain a second message header.

[0279] In some embodiments, the second obtaining module 92 is specifically configured to:

[0280] If no data is written in the data area, obtain the starting position and a second length of the message header area, and offset backward by the second length from the starting position to obtain the first storage position;

[0281] If data has been written in the data area, obtain the third storage position and a third length of the last written data, and offset backward by the third length from the third storage position to obtain the first storage position.

[0282] In some embodiments, the second obtaining module 92 is specifically configured to:

[0283] Obtain a first quantity of the written message headers;

[0284] Calculate the product of the first quantity and a preset message header length to obtain a fourth length;

[0285] Offset backward by the fourth length from the starting position of the message header area to obtain the second storage position.

[0286] In some embodiments, the first writing module 93 is further configured to:

[0287] Determine a fourth storage position after offsetting backward by a fifth length from the first storage position, where the fifth length is the maximum length of a piece of data; if the fourth storage position is within the data area, perform the step of writing the first data into the first storage position.

[0288] In some embodiments, the length of the sector occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0289] Corresponding to the above data reading method, an embodiment of the present application provides a data reading device. Refer to Figure 10 , which is applied to an electronic device. The WAL file in the memory of the electronic device includes a message header area and a data area. The device includes:

[0290] A third acquisition module 101, configured to acquire a first reading position of the message header area;

[0291] A first reading module 102, configured to read a third message header from the first reading position. The third message header indicates a second reading position of the second data corresponding to the third message header in the data area;

[0292] A determination module 103, configured to determine the second reading position according to the indication information included in the third message header;

[0293] A second reading module 104, configured to read the second data from the second reading position.

[0294] In the technical solution provided by the embodiment of the present application, the WAL file is divided into a message header area and a data area. The message header area is used to write message headers, and the data area is used to write data. The electronic device reads the message header in the message header area and reads the data according to the indication information indicating the storage position of the data in the data area in the message header. In this way, the message headers can be centrally stored in the message header area, and the storage position of the message headers in the message header area and the storage position of the data corresponding to the message headers in the data area can be accurately determined, reducing the number of sectors occupied by storing the message headers. Compared with storing by log pages, the utilization rate of the storage space occupied by the WAL file is improved, and thus the effective space of the memory is fully utilized.

[0295] In some embodiments, the determination module 103 is specifically configured to:

[0296] Obtain the indication information from the third message header to obtain a sixth length, where the sixth length is the offset between the second reading position and the first reading position;

[0297] Offset backward by the sixth length at the first reading position to obtain the second reading position.

[0298] In some embodiments, the third acquisition module 101 is specifically configured to:

[0299] Obtain a second quantity of the read data;

[0300] Calculate the product of the second quantity and the preset message header length to obtain a seventh length;

[0301] Offset backward by a seventh length from the starting position of the message header area to obtain a first reading position.

[0302] In some embodiments, the first reading module 102 is further configured to:

[0303] If there is a message header at the first reading position, perform the step of reading a third message header from the first reading position.

[0304] In some embodiments, the length of the sector occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

[0305] The embodiments of the present application further provide an electronic device, as Figure 11 shown, including a processor 111, a communication interface 112, a memory 113, and a communication bus 114. Among them, the processor 111, the communication interface 112, and the memory 113 complete mutual communication through the communication bus 114;

[0306] The memory 113 is used to store a computer program. The WAL file in the memory 113 includes a message header area and a data area;

[0307] When the processor 111 is used to execute the program stored on the memory 113, the steps of any of the above data writing methods or data reading methods are implemented.

[0308] The communication bus mentioned in the above electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience of representation, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus.

[0309] The communication interface is used for communication between the above electronic device and other devices.

[0310] The memory may include a Random Access Memory (RAM), or may also include a Non-Volatile Memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0311] The above-mentioned processor may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0312] In another embodiment provided by the present application, a computer-readable storage medium is further provided. A computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the above-mentioned any data writing method or data reading method is implemented.

[0313] In another embodiment provided by the present application, a computer program product containing instructions is further provided. When it runs on a computer, the computer is caused to execute the steps of any data writing method or data reading method in the above-mentioned embodiments.

[0314] In the above-mentioned embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer, or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a Solid State Disk (SSD)), etc.

[0315] It should be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprising", "including" or any other variant thereof are intended to cover non-exclusive inclusion, such that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or device comprising the said element.

[0316] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, electronic device, computer storage medium, and computer program product, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.

[0317] The above are only the preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application are included within the protection scope of the present application.

Claims

1. A data writing method, characterized in that: Applied to an electronic device, the WAL file in the memory of the electronic device includes a message header area and a data area, and the method includes: Acquire first data to be written and a first message header corresponding to the first data; Acquire a first storage location of the first data in the data area, and acquire a second storage location of the first message header in the message header area; writing the first data into the first storage location; Add indication information to the first message header to obtain a second message header, wherein the indication information indicates the first storage location; The second message header is written to the second storage location.

2. The method according to claim 1, characterized in that The step of adding indication information to the first message header to obtain the second message header includes: Calculating a first length between the first storage position and the second storage position to obtain indication information; The indication information is written into the first message header to obtain a second message header.

3. The method according to claim 1, characterized in that The step of obtaining a first storage location of the first data in the data area comprises: If no data is written in the data area, obtaining the starting position and the second length of the message header area, and shifting the second length backward at the starting position to obtain the first storage position; If data has been written into the data area, the third storage position and the third length of the data written last time are obtained, and the third length is shifted backward at the third storage position to obtain the first storage position.

4. The method according to claim 1, characterized in that: The step of obtaining the second storage location of the first message header in the message header area includes: Get the first number of message headers written; Calculate the product of the first number and a preset message header length to obtain a fourth length; The fourth length is shifted backward at the start position of the message header area to obtain a second storage position.

5. The method according to claim 1, characterized in that Before writing the first data into the first storage location, the method further includes: Determine a fourth storage position that is offset backward by a fifth length at the first storage position, wherein the fifth length is a maximum length of a data; If the fourth storage location is located in the data area, the step of writing the first data into the first storage location is performed.

6. The method according to any one of claims 1 to 5, characterized in that: The length of the sectors occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

7. A data reading method, characterized in that: Applied to an electronic device, the WAL file in the memory of the electronic device includes a message header area and a data area, and the method includes: Obtaining a first reading position of the message header area; Reading a third message header from the first reading position, the third message header indicating that second data corresponding to the third message header is at a second reading position in the data area; Determining the second reading position according to the indication information included in the third message header; The second data is read from the second read position.

8. The method according to claim 7, characterized in that The step of determining the second reading position according to the indication information included in the third message header comprises: Acquire indication information from the third message header to obtain a sixth length, where the sixth length is an offset between the second reading position and the first reading position; The first reading position is shifted backward by the sixth length to obtain the second reading position.

9. The method according to claim 7, characterized in that: The step of obtaining the first reading position of the message header area includes: Obtaining a second amount of read data; Calculate the product of the second number and a preset message header length to obtain a seventh length; The first reading position is obtained by shifting the seventh length backward at the start position of the message header area.

10. The method according to claim 7, characterized in that After obtaining the first read position, further comprising: If a message header exists at the first reading position, the step of reading the third message header from the first reading position is performed.

11. The method according to any one of claims 7 to 10, characterized in that: The length of the sectors occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

12. A data writing device, characterized in that: Applied to an electronic device, the WAL file in the memory of the electronic device includes a message header area and a data area, and the device includes: A first acquisition module, used to acquire first data to be written and a first message header corresponding to the first data; A second acquisition module, used to acquire a first storage position of the first data in the data area, and acquire a second storage position of the first message header in the message header area; A first writing module, used for writing the first data into the first storage location; an adding module, configured to add indication information to the first message header to obtain a second message header, wherein the indication information indicates the first storage location; The second writing module is used to write the second message header into the second storage location.

13. The device according to claim 12, characterized in that The adding module is specifically used for: Calculating a first length between the first storage position and the second storage position to obtain indication information; The indication information is written into the first message header to obtain a second message header.

14. The device according to claim 12, characterized in that The second acquisition module is specifically used for: If no data is written in the data area, obtaining the starting position and the second length of the message header area, and shifting the second length backward at the starting position to obtain the first storage position; If data has been written into the data area, the third storage position and the third length of the data written last time are obtained, and the third length is shifted backward at the third storage position to obtain the first storage position.

15. The device according to claim 12, characterized in that The second acquisition module is specifically used for: Get the first number of message headers written; Calculate the product of the first number and a preset message header length to obtain a fourth length; The fourth length is shifted backward at the start position of the message header area to obtain a second storage position.

16. The device according to claim 12, characterized in that The first writing module is further used for: Before writing the first data into the first storage position, determining a fourth storage position that is offset backward by a fifth length from the first storage position, wherein the fifth length is a maximum length of one data; If the fourth storage location is located in the data area, the step of writing the first data into the first storage location is performed.

17. The device according to any one of claims 12 to 16, characterized in that: The length of the sectors occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

18. A data reading device, characterized in that: Applied to an electronic device, the WAL file in the memory of the electronic device includes a message header area and a data area, and the device includes: A third acquisition module, used to acquire a first reading position of the message header area; A first reading module is used to read a third message header from the first reading position, wherein the third message header indicates that second data corresponding to the third message header is located at a second reading position in the data area; A determination module, configured to determine the second reading position according to the indication information included in the third message header; The second reading module is used to read the second data from the second reading position.

19. The device according to claim 18, characterized in that The determining module is specifically used for: Acquire indication information from the third message header to obtain a sixth length, where the sixth length is an offset between the second reading position and the first reading position; The first reading position is shifted backward by the sixth length to obtain the second reading position.

20. The device according to claim 18, characterized in that The third acquisition module is specifically used for: Obtaining a second amount of read data; Calculate the product of the second number and a preset message header length to obtain a seventh length; The first reading position is obtained by shifting the seventh length backward at the start position of the message header area.

21. The device according to claim 18, characterized in that The first reading module is further used for: After obtaining the first reading position, if a message header exists at the first reading position, the step of reading the third message header from the first reading position is performed.

22. The device according to any one of claims 18 to 21, characterized in that The length of the sectors occupied by the message header area is greater than or equal to the length of the maximum number of message headers included in the WAL file.

23. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; The memory is used to store computer programs. The WAL file in the memory includes a message header area and a data area; A processor, for implementing any of the methods described in claims 1-6 or 7-11 when executing a program stored in a memory.

24. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of claims 1-6 or 7-11 is implemented.

Images