A WAL processing method, apparatus, electronic device, and storage medium

Abstract

This application relates to the field of data processing technology, and more particularly to a Write-Ahead Log (WAL) processing method, apparatus, electronic device, and storage medium. The method involves receiving data segments to be processed and adding them to a preset data block. When the total number of data segments to be processed contained in the data block meets a preset quantity condition, the data block is transferred from the buffer to a memory-mapped area. A preset memory mapping function is called to map the data block from the memory-mapped area to the Write-Ahead Log (WAL) on the disk, obtaining the mapped WAL. By storing the data segments to be processed in the form of data blocks, it is not necessary to read all the data in the WAL during data retrieval, thereby reducing operating system resource consumption. Furthermore, by calling the memory mapping function to map the data blocks, the overhead of switching from user mode to kernel mode can be reduced, thus reducing system resource consumption.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a WAL processing method, apparatus, electronic device and storage medium. Background Technology

[0002] Currently, with the development of computer technology, in order to ensure the recoverability of operational data, operational data can be stored in a write-ahead log (WAL).

[0003] For example, during the operation of the operating system, operational data is stored in the WAL (Write-Ahead Log), and after a system crash, the corresponding operational data is read from the WAL to achieve data recovery.

[0004] See related technologies. Figure 1 The diagram illustrates the WAL processing method in related technologies. Since the operating system only has disk access permissions when running in kernel mode, when storing operational data in the WAL, firstly, a WAL file is created. Once the WAL file creation is complete, the operational data is copied from the user-mode storage space of the operating system to the kernel-mode storage space. Then, the operational data is stored from the kernel-mode storage space to the WAL file on the disk. Furthermore, to enable fast reading of operational data, a corresponding log snapshot can be generated and stored when the WAL file size exceeds a content capacity threshold. When reading operational data, if a log snapshot exists, it is read, along with the WAL file without a snapshot, to obtain the operational data. If no log snapshot exists, the WAL file without a snapshot is read directly to obtain the operational data.

[0005] However, since the operation data needs to be copied from the user-mode storage space to the kernel-mode storage space, a lot of system resources are consumed during the data copying process, thereby reducing the system performance when storing operation data.

[0006] Furthermore, since all operation data is bound together as a whole after the log snapshot is generated, if you want to read a segment of operation data, you need to read the entire log snapshot and then extract the required operation data from the log snapshot. This will increase the resource consumption of the operating system. Summary of the Invention

[0007] This application provides a WAL processing method, apparatus, electronic device, and storage medium to improve the performance of storing operational data and reduce resource consumption.

[0008] The specific technical solutions provided in this application are as follows:

[0009] On one hand, embodiments of this application provide a WAL processing method, including:

[0010] Receive the data segment to be processed and add the received data segment to a preset data block, wherein the data block is set in the buffer of the user-mode storage area;

[0011] When the total number of data segments to be processed contained in the data block is determined to meet the preset quantity condition, the data block is transferred from the buffer to the memory mapping area, which is set in the user-mode storage area;

[0012] Call the preset memory mapping function to map the data block from the memory mapping area to the write-ahead log (WAL) on the disk, and obtain the mapped WAL.

[0013] On one hand, embodiments of this application provide a WAL processing apparatus, including:

[0014] An add module is used to receive data segments to be processed and add the received data segments to a preset data block, wherein the data block is set in the buffer of the user-mode storage area;

[0015] The processing module is used to transfer the data block from the buffer to the memory-mapped area when the total number of data segments to be processed contained in the data block meets the preset quantity condition. The memory-mapped area is set in the user-mode storage area.

[0016] The mapping module is used to call a preset memory mapping function to map the data block from the memory mapping area to the write-ahead log (WAL) on the disk, and obtain the mapped WAL.

[0017] Optionally, when adding the received data segment to be processed to the preset data block, the adding module is further configured to:

[0018] Determine the data attribute information corresponding to the received data segment to be processed;

[0019] According to the preset splicing format, the data attribute information and the data segment to be processed are spliced ​​together to obtain the spliced ​​data segment to be processed.

[0020] The spliced ​​data segment to be processed is added to a preset data block.

[0021] Optionally, the processing module is further configured to:

[0022] If it is determined that the storage mode corresponding to the data block is a single data segment, then the data block is transferred from the buffer to the memory-mapped area;

[0023] If the storage mode is determined to be multi-segment, then when the total number of data segments to be processed contained in the data block is greater than a preset threshold, the data block is transferred from the buffer to the memory-mapped area.

[0024] Optionally, when transferring the data block from the buffer to the memory-mapped area, the processing module is further configured to:

[0025] When it is determined that the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, a memory mapping area with an idle working state is randomly read from the preset mapping pool, and the mapping pool contains multiple memory mapping areas;

[0026] The data block is transferred to the read memory-mapped area.

[0027] Optionally, when transferring the data block from the buffer to the memory-mapped area, the processing module is further configured to:

[0028] When it is determined that the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, a new memory mapping area is generated in the user-mode storage area;

[0029] The data block is transferred from the buffer to the new memory-mapped area.

[0030] Optionally, the mapping module is further configured to:

[0031] Based on the mapping relationship between each memory mapping region and each WAL, determine the WAL mapped to the memory mapping region;

[0032] The memory mapping function is called to map the data block from the memory mapping area to the determined WAL, thereby obtaining the mapped WAL.

[0033] Optionally, the device further includes a data reading module, which is used for:

[0034] Receive a data read request, wherein the data read request includes at least a data index identifier;

[0035] Based on the data index identifier, the target data segment is read from the corresponding data block.

[0036] Optionally, when reading the target data segment from the corresponding data block based on the data index identifier, the data reading module is further configured to:

[0037] The memory mapping function is called to map the target data segment determined based on the data index identifier from the WAL to the memory mapping area, thereby obtaining the mapped memory mapping area.

[0038] Optionally, after adding the received data segment to be processed to a preset data block, the device further includes a caching module, the caching module being used for:

[0039] The data blocks are cached in the data block cache area, which is set in the user-mode storage area.

[0040] Optionally, when reading the target data segment from the corresponding data block based on the data index identifier, the data reading module is further configured to:

[0041] When it is determined that a target data block corresponding to the data block identifier exists in the data block cache, the target data segment corresponding to the data segment identifier is read from the target data block in the data block cache.

[0042] Optionally, the device further includes a verification module, the verification module being used for:

[0043] Calculate the target attribute information corresponding to the target data segment, and determine the data attribute information corresponding to the target data segment from the corresponding data block;

[0044] Based on the target attribute information and the data attribute information, the integrity verification result of the target data segment is determined.

[0045] Optionally, when determining the integrity verification result of the target data segment based on the target attribute information and the data attribute information, the verification module is further configured to:

[0046] When it is determined that the target length in the target attribute information is the same as the data length in the data attribute information, and the target information digest value in the target attribute information is the same as the data information digest value in the data attribute information, the integrity verification result of the target data segment is determined to be complete.

[0047] When it is determined that the target length is different from the data length, and the target information digest value is different from the data information digest value, the integrity verification result of the target data segment is determined to be incomplete.

[0048] On one hand, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor performs the steps of any of the above-described WAL processing methods.

[0049] On one hand, embodiments of this application provide a computer storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the steps of any of the above-described WAL processing methods.

[0050] On one hand, an embodiment of this application provides a computer program product, which includes computer instructions stored in a computer-readable storage medium; when a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the electronic device to perform the steps of any of the above-described WAL processing methods.

[0051] Since the embodiments of this application adopt the above-described technical solution, they have at least the following technical effects:

[0052] The system receives data segments to be processed and adds them to a preset data block. When the total number of data segments to be processed contained in the data block meets the preset quantity condition, the data block is transferred from the buffer of the user-mode storage area to the memory-mapped area set in the user-mode storage area. The system then maps the data block from the memory-mapped area to the WAL on the disk by calling a preset memory-mapped function, thereby obtaining the mapped WAL.

[0053] In this way, since the received data to be processed is added to data blocks, and when the total number of data segments to be processed contained in a data block meets a preset condition, the data block is transferred to the memory-mapped area for mapping. During the data reading process, if it is desired to read a segment of data in the WAL, the required data can be read directly from the data block in the WAL, without having to read all the data in the WAL, thereby reducing the operating system's resource consumption. Furthermore, during the storage of data to be processed, by calling the memory mapping function, the data to be processed is directly mapped from the user-mode memory-mapped area to the WAL on the disk. Therefore, by calling the memory mapping function for data mapping, it is not necessary to copy the data to be processed from the user-mode storage space to the kernel-mode storage space, reducing the overhead of switching from user mode to kernel mode, thereby reducing system resource consumption and improving system performance when storing data. Attached Figure Description

[0054] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0055] Figure 1 This is a flowchart illustrating the WAL processing method in related technologies;

[0056] Figure 2A This is a schematic diagram of the application scenario in the embodiments of this application;

[0057] Figure 2B This is a schematic diagram of the blockchain structure in the embodiments of this application;

[0058] Figure 2C This is a verification diagram in an embodiment of this application;

[0059] Figure 3A This is a flowchart illustrating the WAL processing method in an embodiment of this application.

[0060] Figure 3B This is a flowchart illustrating the process of adding a data segment to be processed in an embodiment of this application;

[0061] Figure 3C This is an example diagram of the data segment for assembling reception processing in an embodiment of this application;

[0062] Figure 3D This is a schematic diagram of the data block structure in an embodiment of this application;

[0063] Figure 3E This is a schematic diagram of the first process of transferring data blocks in an embodiment of this application;

[0064] Figure 3F This is a first example diagram of the transferred data block in an embodiment of this application;

[0065] Figure 3G This is a schematic diagram of the second process for transferring data blocks in an embodiment of this application;

[0066] Figure 3H This is a second example diagram of the transferred data block in an embodiment of this application;

[0067] Figure 3I This is a schematic diagram of the third process for transferring data blocks in an embodiment of this application;

[0068] Figure 3J This is a third example diagram of the data block being transferred in the embodiments of this application;

[0069] Figure 3K This is a flowchart illustrating the mapping of data blocks in an embodiment of this application;

[0070] Figure 3L This is an example diagram of the mapped data block in an embodiment of this application;

[0071] Figure 3M This is a schematic diagram of the data segment reading process in an embodiment of this application;

[0072] Figure 3N This is a schematic diagram of the data integrity verification process in an embodiment of this application;

[0073] Figure 3O This is a flowchart illustrating the process of determining the integrity verification result in an embodiment of this application;

[0074] Figure 4This is an example diagram of the WAL processing method in the embodiments of this application;

[0075] Figure 5 This is a schematic diagram of the process for storing the data segment to be processed in an embodiment of this application;

[0076] Figure 6 This is a schematic diagram of the data segment reading process in an embodiment of this application;

[0077] Figure 7 This is a schematic diagram illustrating the process of writing the data segment to be processed to the memory-mapped area in an embodiment of this application;

[0078] Figure 8 This is a schematic diagram of a data writing process in an embodiment of this application;

[0079] Figure 9 This is another schematic diagram of the data writing process in an embodiment of this application;

[0080] Figure 10 This is a schematic diagram of the WAL processing device in the embodiments of this application;

[0081] Figure 11 This is a schematic diagram of the hardware structure of an electronic device that applies an embodiment of this application. Detailed Implementation

[0082] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0083] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0084] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in sequences other than those illustrated or described herein.

[0085] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0086] Data block: Represents a data set containing at least one data segment to be processed.

[0087] It should be noted that a data block may contain one or more data segments to be processed, and this application embodiment does not impose any restrictions on this.

[0088] User-mode storage area: Represents the memory space where the operating system runs in user mode. The user-mode storage area includes at least a buffer and a memory-mapped area.

[0089] Buffer: Represents the memory space set in the user-mode storage area used to store data blocks.

[0090] Memory-mapped area: Represents the memory space in the user-mode storage area used to map data blocks.

[0091] It should be noted that the memory mapping area and WAL in this embodiment of the application have a one-to-one mapping relationship. Through the memory mapping function, the data blocks stored in the memory mapping area can be mapped to the WAL.

[0092] Mapping pool: The representation is set in the user-mode storage area and contains memory space of at least one memory mapping area.

[0093] Memory mapping functions: used to map data blocks from the memory-mapped area to the WAL on disk.

[0094] In this application, the memory mapping function can be, for example, a memory map, and there is no limitation on this.

[0095] Data block cache: Located in the user-space storage area, it is a memory space used to cache data blocks.

[0096] User mode: Represents an execution state with limited processing permissions. When the operating system runs in user mode, it is used to run user programs. It can only access limited memory space and is not allowed to access peripheral devices.

[0097] Kernel mode: Represents the execution state with full processing privileges. When the operating system is running in kernel mode, it can access all data in the memory space and peripheral devices, such as hard drives and network cards. The operating system can also switch from one program to another.

[0098] The design concept of the embodiments of this application is briefly introduced below:

[0099] Currently, with the development of computer technology, the operating system's operational data can be stored in the WAL (Write-Ahead Log). When the operating system crashes, the operational data can be recovered by reading the WAL.

[0100] In related technologies, since the operating system only has disk access permissions when running in kernel mode, the process of storing operational data in the Write-Ahead Log (WAL) involves first calling the operating system to create the WAL, writing the operational data to user-mode storage space according to certain naming rules, copying the operational data from user-mode storage space to the operating system's kernel-mode storage space, and finally storing the operational data from kernel-mode storage space to the WAL on disk. Furthermore, to improve the speed of reading operational data, when the WAL file on disk becomes too large, a corresponding log snapshot is generated and stored, allowing direct reading of the log snapshot when accessing operational data.

[0101] However, this approach in related technologies requires copying operation data from user-mode storage space to kernel-mode storage space. Therefore, the data copying process consumes a lot of system resources, thereby reducing the performance of the operating system when storing operation data.

[0102] Furthermore, using this method in related technologies, when the WAL file becomes too large, a corresponding log snapshot is generated for storage, and all operation data is bound as a whole. However, if the data nodes involved are included in the log snapshot, the entire log snapshot needs to be read first, and then the corresponding operation data needs to be extracted from the log snapshot. Therefore, it will increase the resource consumption of the operating system.

[0103] In view of this, embodiments of this application propose a WAL processing method, apparatus, electronic device, and storage medium. By introducing memory mapping functions provided by the operating system, a memory space in the user-mode storage area is mapped to the disk's WAL. That is, a mapping relationship exists between the memory mapping area and the WAL. When a data segment to be processed is received, the data block containing the data segment is transferred to the created memory mapping area. This allows the data block to be mapped from the memory mapping area to the disk's WAL by calling the memory mapping function, reducing the overhead of switching between user mode and kernel mode in the operating system, thereby improving the system performance for storing and operating data.

[0104] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0105] See Figure 2AThe diagram shown is a schematic representation of an application scenario in this application. This application scenario includes a data sharing system 200.

[0106] In this embodiment, the data sharing system 200 refers to a system for data sharing between nodes. This system may include multiple nodes 201, which can be clients running operating systems. Each node 201, during normal operation, can receive input information and maintain shared data within the system based on this information. To ensure interoperability within the data sharing system, information connections can exist between each node, allowing for information transmission. For example, when any node in the data sharing system receives input information, other nodes obtain this information according to a consensus algorithm and store it as data in the shared data, ensuring consistency across all nodes. Furthermore, when a node 201 runs the operating system, it generates operation data, i.e., data segments to be processed. After generating these data segments, each node 201 maps the data block containing the data segment from the memory-mapped area to the Write-Ahead Log (WAL) on the disk.

[0107] In one possible implementation, the nodes of the blockchain can be directly or indirectly connected via wired or wireless communication, and this application embodiment does not impose any limitations on this.

[0108] It should be noted that each node in the data sharing system has a corresponding node identifier, and each node in the data sharing system can store the node identifiers of other nodes in the data sharing system. This allows the generated block to be broadcast to other nodes in the data sharing system based on their node identifiers. After receiving the data segment to be processed, other nodes can add the received data segment to a preset data block and map the data block containing the data segment to its own WAL (Write-Ahead Allocation) on its disk from the memory mapping area. Furthermore, each node can maintain a node identifier list as shown in the table below, storing the node name and node identifier in this list. The node identifier can be an IP (Internet Protocol) address or any other information that can be used to identify the node. See Table 1, which is the node identifier table in this embodiment of the application. Table 1 only uses IP addresses as an example for illustration.

[0109] Table 1.

[0110] Node Name Node identifier Node 1 117.114.151.174 Node 2 117.116.189.145 … … Node N 117.123.199.201

[0111] Each node in the data-sharing system stores the same blockchain. A blockchain consists of multiple blocks; see [link / reference]. Figure 2B The diagram shown is a schematic of the blockchain structure in this embodiment. The blockchain consists of multiple blocks. The genesis block includes a block header and a block body. The block header stores the input information feature value, version number, timestamp, and difficulty value. The block body stores the input information. The next block of the genesis block takes the genesis block as its parent block. The next block also includes a block header and a block body. The block header stores the input information feature value of the current block, the block header feature value of the parent block, the version number, the timestamp, and the difficulty value, and so on. This ensures that the block data stored in each block of the blockchain is associated with the block data stored in the parent block, guaranteeing the security of the input information in the block.

[0112] When generating the various blocks in the blockchain, refer to Figure 2C As shown, this is a verification diagram in an embodiment of this application. When the node where the blockchain is located receives input information, it verifies the input information. After verification, it stores the input information in the memory pool and updates its hash tree used to record the input information. Then, it updates the timestamp to the time when the input information was received and tries different random numbers to calculate the feature value multiple times, so that the calculated feature value can satisfy the following formula:

[0113] SHA256(SHA256(version+prev_hash+merkle_root+ntime+nbits+x))<TARGET

[0114] Wherein, SHA256 is the feature value algorithm used to calculate the feature value; version (version number) is the version information of the relevant block protocol in the blockchain; prev_hash is the block header feature value of the parent block of the current block; merkle_root is the feature value of the input information; ntime is the update time of the update timestamp; nbits is the current difficulty, which is a fixed value for a period of time and is determined again after exceeding the fixed time period; x is a random number; TARGET is the feature value threshold, which can be determined based on nbits.

[0115] Thus, when a random number satisfying the above formula is calculated, the information can be stored accordingly, generating a block header and a block body to obtain the current block. Subsequently, the node where the blockchain resides sends the newly generated block to other nodes in its data sharing system based on the node identifiers of other nodes in the data sharing system. The other nodes then verify the newly generated block and add it to their stored blockchain after verification.

[0116] The application scenarios in the embodiments of this application are described below.

[0117] Application Scenario 1:

[0118] The method in this embodiment can be applied to the WAL used in blockchain consensus mechanisms. Specifically, to enable a blockchain network to reach consensus, a consensus algorithm can be used to make decisions about events. Therefore, when one node in the blockchain network receives operation data sent by other nodes for a certain event, it stores the received operation data in a data block in a buffer set in the user-mode storage area. When the total number of data segments to be processed contained in the data block meets a preset quantity condition, the data block is transferred from the buffer in the user-mode storage area to the memory-mapped area set in the user-mode storage area. By calling a preset memory-mapped function, the data block is mapped from the memory-mapped area to the WAL on the disk, thereby realizing the storage of operation data. Based on the stored operation data and in conjunction with the consensus algorithm, the decision result of the event can be determined.

[0119] Application Scenario 2:

[0120] During operating system operation, corresponding operational data is generated. Once generated, this data is added to a pre-set data block in a buffer. When the data block is full, it is transferred from the buffer to the memory-mapped area. A pre-defined memory mapping function then maps the data block from the memory-mapped area to the Write-Ahead Log (WAL) on disk. If the operating system crashes, the operation data in the WAL can be read using the memory-mapped function, thus restoring the operating system to its pre-crash state.

[0121] The WAL processing procedure in the embodiments of this application will be described below with reference to the accompanying drawings. Figure 3A The diagram shown is a flowchart of the WAL processing method in an embodiment of this application. The specific WAL processing flow is as follows:

[0122] S30: Receive the data segment to be processed and add the received data segment to the preset data block.

[0123] The data blocks are set in the buffer of the user-mode storage area.

[0124] In this embodiment of the application, when it is determined that a data segment to be processed has been received, a data block set in the buffer is read, and the received data segment to be processed is added to the read data block.

[0125] It should be noted that when a data segment to be processed is received, the data block read may not contain the data segment to be processed. Of course, the data block read may also contain at least one data segment to be processed. That is, the data block may also contain data segments to be processed received at other times. This is not a limitation.

[0126] Optionally, in this embodiment, to facilitate mapping the data set to the memory mapping area, the data segment to be processed can be transferred to the memory mapping area in the form of a data block. During the process of adding the data segment to be processed to the data block, the data segment to be processed and its corresponding data attribute information can be concatenated and added to the data block. The process of adding the data segment to be processed to the data block in this embodiment is described below. (See reference...) Figure 3B The diagram shown illustrates the process of adding a data segment to be processed in an embodiment of this application, specifically including:

[0127] S301: Determine the data attribute information corresponding to the received data segment to be processed.

[0128] In this embodiment of the application, attribute calculations are performed on the received data segment to be processed to obtain the data attribute information corresponding to the data segment to be processed.

[0129] The data attribute information includes at least one of the following: data length and data information summary value.

[0130] It should be noted that the data length is the length of the data segment to be processed, for example, it can be 64kb, and this embodiment of the application does not impose any limitation on it.

[0131] The data digest value is obtained by calculating the digest of the data segment to be processed using a preset digest algorithm. The digest algorithm can be, for example, the Cyclic Redundancy Check (CRC) 32 algorithm or the CRC 16 algorithm. This application does not limit the specific algorithm used in this embodiment.

[0132] S302: According to the preset splicing format, splice the data attribute information and the data segment to be processed to obtain the spliced ​​data segment to be processed.

[0133] In this embodiment of the application, data attribute information and data segment to be processed are spliced ​​together according to a preset splicing format to obtain spliced ​​data segment to be processed. The spliced ​​data segment to be processed includes data segment to be processed and data attribute information.

[0134] When concatenating the data segment to be processed and the data attribute information, the last character of the data attribute information can be connected with the first character of the data segment to be processed. Of course, the last character of the data segment to be processed can also be connected with the first character of the data attribute information. This application embodiment does not impose any restrictions on this.

[0135] It should be noted that if the data attribute information is data length and data information digest value, the data length and data information digest value need to be concatenated first according to the preset concatenation format. Then, the data segment to be processed is concatenated with the concatenated data length and data information digest value again to obtain the concatenated data segment to be processed.

[0136] For example, see Figure 3C The diagram shown is an example of splicing data segments for processing in an embodiment of this application. Assuming that the data attribute information includes data length and CRC32 value, firstly, the length of the data segment to be processed is calculated to obtain the data length of the data segment to be processed. At the same time, the CRC32 algorithm is used to calculate the CRC32 value of the data segment to be processed. The data length is 8 bits, the CRC32 value is 8 bits, and the data segment to be processed is 184 bits. Then, the data segments to be processed are spliced ​​according to the format of "data length + CRC32 value + data segment to be processed" to obtain the spliced ​​data segment to be processed.

[0137] It should be noted that, in the embodiments of this application, since different attribute calculation methods can be used to calculate the data attribute information of different data segments to be processed, the data format of the data attribute information calculated for different data segments to be processed may be different. For example, the format of the data information digest value calculated based on the CRC32 algorithm is different from the format of the data information digest value calculated based on the hash algorithm. In order to facilitate the storage of data attribute information, the format of data attribute information can be standardized so that the format of data attribute information determined based on different calculation algorithms is the same, that is, the number of bytes occupied by each data attribute information is the same.

[0138] Additionally, it should be noted that in the embodiments of this application, the same attribute calculation method can be used to determine the data attribute information of different data segments to be processed. Of course, different attribute calculation methods can be used to determine the data attribute information of different data segments to be processed, and this embodiment of the application does not impose any restrictions on this.

[0139] S303: Add the spliced ​​data segment to be processed into the preset data block.

[0140] In this embodiment, after obtaining the spliced ​​data segment to be processed, the spliced ​​data segment to be processed is added to a preset data block to obtain the added data block. In this way, by splicing the data attribute information corresponding to the data segment to be processed with the data segment to be processed and adding it to the data block, the accurate position of the required data segment can be quickly located when reading the data segment, thereby improving the data reading efficiency.

[0141] For example, in this embodiment, the data block can be divided into two parts: one for storing the data index and the other for storing the data segments to be processed. First, the data segments to be processed are stored, each segment consisting of three parts: data length, CRC32 value, and the data segment to be processed. Each time a serialized data segment to be processed is written, the length is first calculated, then the CRC32 value is calculated, and finally, the data portion of the data block is written in the format of data length + CRC32 value + data segment to be processed. Second, the data index of the data segment to be processed is recorded in the index area.

[0142] It should be noted that recording the data length is to ensure that the data segment is not lost when reading the data segment, and recording the CRC32 value is to verify the integrity of the data segment.

[0143] For example, see Figure 3D The diagram shown is a structural schematic of a data block in an embodiment of this application. The data block includes a data index area and a data area. The data index area stores the start position 0 and end position 199 corresponding to the data segment A to be processed, the start position 200 and end position 399 corresponding to the data segment B to be processed, and the start position 400 and end position 599 corresponding to the data segment C to be processed. The data area stores the data length, CRC32 value, and data segment to be processed for the data segment A, the data length, CRC32 value, and data segment to be processed for the data segment B, and the data length, CRC32 value, and data segment to be processed for the data segment C.

[0144] S31: When the total number of data segments to be processed contained in the data block is determined to meet the preset quantity condition, the data block is transferred from the buffer to the memory-mapped area.

[0145] The memory-mapped area is located in the user-mode storage area.

[0146] In this embodiment of the application, it is determined whether the total number of data segments to be processed contained in the data block meets the preset quantity condition. When it is determined that the total number meets the preset quantity condition, the data block is transferred from the buffer to the memory mapping area.

[0147] It should be noted that in this embodiment, the data blocks are transferred to the memory mapping area according to the index order corresponding to the data blocks, and a data block identifier is set for each data block. In this way, the required data blocks can be accurately found and read based on the data block identifier during the subsequent reading of data segments, thereby improving the data reading efficiency.

[0148] It should be noted that, in this embodiment of the application, the total number of data segments to be processed that can be stored in the data block can be determined based on the storage mode corresponding to the data block. (See reference...) Figure 3EThe diagram shown illustrates the process of transferring data blocks in an embodiment of this application, specifically including:

[0149] S311: If it is determined that the storage mode corresponding to the data block is a single data segment, then the data block is transferred from the buffer to the memory-mapped area.

[0150] In this embodiment of the application, each data block is a data set. The data set may contain one data segment or multiple data segments. Therefore, the storage mode of the data block can be divided into single data segment and multiple data segment based on the number of data segments contained. If the storage mode corresponding to the data block is single data segment, then only one data segment to be processed can be stored in the data block. Therefore, after the data segment to be processed is added to the data block, the data block is transferred from the buffer to the memory mapping area.

[0151] S312: If the storage mode is determined to be multi-segment, then when the total number of data segments to be processed contained in the data block is greater than the preset number threshold, the data block is transferred from the buffer to the memory-mapped area.

[0152] In this embodiment of the application, if it is determined that the storage mode corresponding to the data block is multiple data segments, then the data block can store at least two data segments to be processed. Therefore, after adding the data segments to be processed to the data block, the total number of data segments to be processed contained in the data block is determined. If the total number is determined to be greater than a preset number threshold, it is determined that the data block is full of data segments to be processed and cannot store any other data segments to be processed. The data block is then transferred from the buffer to the memory mapping area. If the total number is determined not to be greater than the number threshold, it is determined that the data block can still store other data segments to be processed. Other data segments to be processed are then received until the total number is greater than the number threshold. Then the data block is transferred from the buffer to the memory mapping area.

[0153] For example, see Figure 3F The diagram shown is an example of a data block being transferred in an embodiment of this application. First, it is determined that the storage mode corresponding to data block 2 is multiple data segments. Then, it is determined that the total number of data segments to be processed contained in data block 2 is 2. At this time, it is determined that the total number 2 is equal to the preset number threshold 2. Then, data block 1 is transferred to the memory mapping area, which contains data block 1 and data block 2.

[0154] One possible implementation in this application is to set the storage mode of the data block to multi-segment by setting the parameter BlockMultiData=true in the business system, so that a data block can support the writing of multiple data segments. At the same time, a quantity threshold is set by the parameter BlockSize in the business system. When the total number of data segments to be processed contained in the data block is greater than the quantity threshold set by the parameter BlockSize, the data block is transferred to the memory mapping area, thereby improving the write performance under high-frequency write conditions.

[0155] The option to enable or disable BlockMultiData can be configured based on the operating system's business needs. Specifically, when the length of the data block mapped to the WAL by the operating system each time exceeds a preset length threshold, BlockMultiData can be disabled, i.e., the storage mode of the data block is set to single data segment. When the length of the data block mapped to the WAL by the operating system each time does not exceed the preset length threshold, BlockMultiData can be enabled, i.e., the storage mode of the data block is set to multiple data segments. The multiple data segment mode is more suitable for when the data length is small and mapping is frequent. Setting the storage mode to multiple data segments can significantly reduce the mapping frequency, thereby improving system performance.

[0156] It should be noted that the length threshold in this embodiment can be set according to the memory capacity of the WAL, or according to actual business needs. This embodiment does not impose any restrictions on this.

[0157] In addition, in this embodiment of the application, the data set size in each data block can be ensured to be relatively uniform by setting the parameter of the data block length, so that it is neither too large nor too small.

[0158] It should be noted that if the storage mode of the data block is multi-segment, the data segments to be processed can be added to the data block according to the order in which they are added and according to the preset splicing format. In this way, the position of the target data segment to be read in the data block can be quickly located during the subsequent reading of data segments.

[0159] Furthermore, in the embodiments of this application, the memory mapping area can be read from the mapping pool when there is insufficient remaining memory capacity, or it can be newly created when there is insufficient remaining content capacity. Therefore, based on the method of establishing the memory mapping area, the process of transferring data blocks can be divided into the following two types:

[0160] The first method: The memory-mapped area is read from the mapping pool; see [link / reference]. Figure 3GThe diagram shown illustrates the first process of transferring data blocks in an embodiment of this application, specifically including:

[0161] A11: When the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, a memory mapping area with an idle working state is randomly read from the preset mapping pool.

[0162] The mapping pool contains multiple memory mapping regions.

[0163] In this embodiment, it is determined whether the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area. When it is determined that the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, the memory mapping area does not have enough storage space to store the data block. Therefore, a memory mapping area is randomly read from each memory mapping area in the mapping pool that is in an idle working state. When it is determined that the length of the data block corresponding to the data block is not greater than the remaining memory capacity of the memory mapping area, it is determined that the memory mapping area has enough storage space to store the data block. Therefore, the data block can be directly transferred to the memory mapping area.

[0164] It should be noted that the remaining memory capacity in this embodiment refers to the remaining content capacity in the memory-mapped area. The calculation method of the remaining memory capacity in this embodiment is explained below. Specifically, firstly, the memory capacity of each data block stored in the memory-mapped area is determined. Then, based on the difference between the total memory capacity of the memory-mapped area and the memory capacity, the remaining memory capacity of the memory-mapped area is obtained.

[0165] A12: Transfer the data block to the read memory-mapped area.

[0166] In this embodiment of the application, after the memory-mapped area is read out, the data block is transferred to the read-out memory-mapped area.

[0167] For example, see Figure 3H As shown, this is a first example diagram of transferring data blocks in an embodiment of this application. First, the data block length of data block M is determined to be 12kb. At the same time, the remaining memory capacity of the memory mapping area is determined to be 11kb. It is determined that memory mapping area 1 does not have enough storage space to store the data block. Therefore, memory mapping area 4 is read from memory mapping area 2, memory mapping area 3, memory mapping area 4 and memory mapping area 4 included in the mapping pool, and data block M is stored in memory mapping area 4.

[0168] It should be noted that since each memory-mapped region has its own mapped WAL, when the remaining memory capacity of the memory-mapped region is insufficient, the remaining content capacity of the WAL on the disk is also insufficient.

[0169] In this way, during operating system initialization, a number of memory mapping regions and their corresponding Write-Ahead Logs (WALs) are pre-mapped. These pre-mapped memory mapping regions are then placed in a mapping pool. The operating system can then retrieve free memory mapping regions from the pool for data block writing. When the remaining memory capacity of a mapped memory mapping region is insufficient, a new memory mapping region and its corresponding WAL can be obtained from the mapping pool. This eliminates the need to wait for the creation of new memory mapping regions during the business process, saving blocking time and thus improving the write performance of the WALs.

[0170] The second method: The memory-mapped area is newly generated; see [link / reference]. Figure 3I The diagram shown illustrates the second process of transferring data blocks in an embodiment of this application, specifically including:

[0171] A21: When the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, a new memory mapping area is generated in the user-mode storage area.

[0172] In this embodiment, the length of the data block corresponding to the data block is calculated. Then, it is determined whether the length of the data block is greater than the remaining memory capacity of the memory mapping area. If it is determined that the length of the data block is not greater than the remaining memory capacity, it is determined that the memory mapping area has enough storage space to store the data block. Therefore, the data block is directly transferred from the buffer to the memory mapping area. If it is determined that the length of the data block is greater than the remaining memory capacity, it is determined that the memory mapping area does not have enough space to store the data block. Therefore, a new memory mapping area is generated, and a mapping relationship between the new memory mapping area and the corresponding WAL is established.

[0173] A22: Transfer the data block from the buffer to the new memory-mapped area.

[0174] In this embodiment of the application, after the memory mapping area is regenerated, the data block is transferred from the buffer to the newly generated memory mapping area.

[0175] For example, see Figure 3J As shown, this is a second example diagram of transferring data blocks in an embodiment of this application. First, the length of the data block corresponding to data block N is calculated to be 12kb. At the same time, it is determined that the remaining memory capacity of memory mapping area 1 is 10kb. It is determined that memory mapping area 1 does not have enough storage space to store the data block. Therefore, a new memory mapping area 2 is generated, and a mapping relationship between the newly generated memory mapping area 2 and the corresponding WAL is established. Finally, the data block is stored in the newly generated memory mapping area 2.

[0176] In this way, a set of memory mapping areas and WALs for storing and reading data segments need to be mapped. When the remaining memory capacity of the mapped memory mapping area and WAL is insufficient, a new mapping is created. That is, the memory mapping area and WAL, as well as their mapping relationship, are recreated. There is no need to consume mapping resources in advance, and resources can be requested and allocated on demand.

[0177] Furthermore, in this embodiment, a primary-backup mapping can also be used. That is, two mappings are created in advance: one primary mapping and one backup mapping. When storing data segments to be processed, the primary mapping is generally used. When the disk corresponding to the primary mapping is full, the backup mapping can be used directly, thus avoiding blocking. Simultaneously, a new backup mapping can be created asynchronously using a coroutine or thread.

[0178] In this embodiment, a primary-multiple-standby approach can also be used, which involves one primary mapping and multiple standby mappings. This avoids creating too many mappings in advance, which would lead to resource waste, and also prevents the creation of mappings during the process from impacting performance.

[0179] S32: Call the preset memory mapping function to map the data block from the memory mapping area to the WAL on the disk and obtain the mapped WAL.

[0180] In this embodiment of the application, a preset memory mapping function is called to map the data block from the memory mapping area to the WAL, thereby obtaining the mapped WAL, so that subsequent processing can be performed based on the mapped WAL.

[0181] It should be noted that, in this embodiment of the application, before mapping the data block from the memory-mapped area to the WAL, the mapping relationship between the memory-mapped area and the WAL on the disk is established by calling the operating system. The mapping relationship can be established after the operating system starts and before mapping the data block, or it can be established in advance during the initialization phase of the operating system, thereby reducing the number of operation steps and reducing the resource consumption of the operating system.

[0182] Optionally, since there may be multiple memory mapping regions and WALs in this embodiment, a mapping relationship is pre-established between each memory mapping region and its corresponding WAL. This allows data blocks to be mapped to the corresponding WALs based on the mapping relationship during subsequent mapping. The process of mapping data blocks in this embodiment is described below; please refer to [link / reference]. Figure 3K The diagram shown is a flowchart illustrating the mapping of data blocks in an embodiment of this application, specifically including:

[0183] S321: Determine the WAL mapped to the memory mapping area based on the mapping relationship between each memory mapping area and each WAL.

[0184] In this embodiment of the application, when establishing the memory mapping area, the corresponding WAL is also established on the disk. Therefore, based on the mapping relationship between each memory mapping area and the corresponding WAL, the WAL mapped to the memory mapping area is determined from each WAL.

[0185] S322: Call the memory mapping function to map the data block from the memory mapping area to the determined WAL, and obtain the mapped WAL.

[0186] In this embodiment of the application, after the WAL is determined, the data block is mapped from the memory mapping area to the determined WAL by calling the memory mapping function, so as to obtain the mapped WAL.

[0187] It should be noted that in the embodiments of this application, if there is only one memory mapping area and WAL, then there is no need to determine the WAL, and the data block can be directly mapped from the memory mapping area to the WAL.

[0188] For example, see Figure 3L The diagram shown is an example of a mapped data block in an embodiment of this application. Each memory mapping region has its mapped WAL. There is a mapping relationship between memory mapping region 1 and WAL1, memory mapping region 2 and WAL2, memory mapping region 3 and WAL3, and memory mapping region 4 and WAL4. Based on the mapping region identifier 1 corresponding to the memory mapping region, the WAL1 that it is mapped to is determined. Then, the data block N is mapped from memory mapping region 1 to WAL1 to obtain the mapped WAL1.

[0189] Furthermore, in this embodiment of the application, after obtaining the mapped WAL, the corresponding data segment can be read from the WAL after the operating system crashes, see [reference]. Figure 3M The diagram shown illustrates the process of reading the corresponding data segment in an embodiment of this application, specifically including:

[0190] M1: Receives data read requests.

[0191] The data read request must include at least a data index identifier.

[0192] In this embodiment of the application, when the target object needs to restore the operating system, a data read request can be triggered in the operating system.

[0193] The data read request includes a data index identifier.

[0194] It should be noted that the data index identifier must include at least one of the following: a data block identifier or a data segment identifier. When reading a data segment within a data block, the data index identifier includes both the data block identifier and the data segment identifier; when reading the entire data block, the data index identifier includes only the data block identifier.

[0195] M2: Reads the target data segment from the corresponding data block based on the data index identifier.

[0196] In this application, when the data index identifier includes a data block identifier and a data segment identifier, the data block corresponding to the data block identifier is determined from each data block based on the data block identifier. Then, the target data segment corresponding to the data segment identifier is read from the determined data block based on the data segment identifier.

[0197] Optionally, in this embodiment of the application, a possible implementation method for reading the target data segment is provided, which is: calling a memory mapping function to map the target data segment determined based on the data index identifier from WAL to the memory mapping area, thereby obtaining the mapped memory mapping area.

[0198] In this embodiment of the application, firstly, the target data segment is determined based on the data index identifier. Then, the memory mapping function is called to map the target data segment from WAL to the memory mapping area. In this way, the data index identifier is used.

[0199] Specifically, the target data block can be determined from each data block based on the data block identifier in the data index identifier, and the target data segment can be determined from the target data block based on the data segment identifier in the data index identifier. Finally, the determined target data segment is mapped from WAL to the memory-mapped area by calling the memory mapping function.

[0200] Optionally, in this embodiment of the application, since a data block cache area is also provided in the user-mode storage area, after the received data segments to be processed are added to the preset data blocks, the data blocks can also be cached in the data block cache area. The data block cache area caches a preset number of data blocks for fast reading of data segments, which can improve data reading performance.

[0201] It should be noted that, in this embodiment of the application, the data block cache contains the latest preset number of data blocks, and the process of caching data blocks to the data block cache can be executed in parallel with the process of transferring data blocks to the memory mapping area, thereby reducing the time spent storing data blocks and improving the system performance when the operating system stores and operates data.

[0202] Additionally, it should be noted that in this embodiment, data blocks are cached according to their generation order. Furthermore, the data block cache area contains a preset number of data blocks with the latest time point. If the data block cache area does not have sufficient space to cache other data blocks, after a new data block is generated, the earliest data block in the data block cache area is deleted, ensuring that the latest data block is always cached in the data block cache area. For example, when a new data block J is generated, assuming the data block cache area can cache four data blocks, and data blocks L, M, N, and O are already cached in the data block cache area, then the earliest data block L is determined from among the data blocks, and data block L is deleted. Data block J is then cached in the data block cache area.

[0203] Therefore, upon receiving a data read request, it can be determined whether a target data block corresponding to the data block identifier exists in the data block cache based on the data block identifier. Specifically, this can be divided into the following two cases:

[0204] The first scenario: The target data block corresponding to the data block identifier exists in the data block cache.

[0205] Specifically, this includes: when it is determined that there is a target data block corresponding to the data block identifier in the data block cache, reading the target data segment corresponding to the data segment identifier from the target data block in the data block cache.

[0206] In this embodiment, since the data segment is stored in the form of a data block and the data segment has a data segment identifier, the data segment can be quickly located in the data block. Therefore, when it is determined that there is a target data block corresponding to the data block identifier in the data block cache, the target data segment corresponding to the data segment identifier is read from the target data block in the data block cache based on the data segment identifier. In this way, the corresponding target data segment can be directly read from the data block cache, which can greatly improve the data reading performance.

[0207] The second scenario: The target data block corresponding to the data block identifier does not exist in the data block cache.

[0208] In this embodiment of the application, when there is no target data block corresponding to the data block identifier in the data block cache, the corresponding target data segment is read from the WAL.

[0209] Furthermore, in this embodiment of the application, in order to improve the integrity of the determined target data segment, the target data segment can be subjected to integrity verification, see [reference]. Figure 3N The diagram shown is a flowchart illustrating the data integrity verification process in an embodiment of this application, specifically including:

[0210] N1: Calculate the target attribute information corresponding to the target data segment, and determine the data attribute information corresponding to the target data segment from the corresponding data block.

[0211] In this embodiment of the application, attribute calculation is performed on the target data segment to obtain the target attribute information corresponding to the target data segment. Also, since the target data block contains not only the target data segment but also the data attribute information corresponding to the target data segment, the data attribute information corresponding to the target data segment is determined from the target data block.

[0212] Therefore, the target attribute information in this embodiment is determined by attribute calculation based on the read target data segment.

[0213] N2: Determine the integrity verification result of the target data segment based on the target attribute information and data attribute information.

[0214] In this embodiment, the integrity verification result of the target data segment is determined based on the target attribute information and the data attribute information. Specifically, if the target attribute information and the data attribute information are the same, the integrity verification result of the target data segment is determined to be passed; if the target attribute information and the data attribute information are different, the integrity verification result of the target data segment is determined to be failed. In this way, the integrity verification result of the target data segment can be determined by the target attribute information corresponding to the target data segment and the read data attribute information, thereby ensuring the integrity of the target data segment.

[0215] Optionally, in this application embodiment, a possible implementation method is provided for determining the integrity verification result of the target data segment, see reference. Figure 3O The diagram shown illustrates the process for determining the integrity verification result in an embodiment of this application, specifically including:

[0216] N21: When the target length in the target attribute information is the same as the data length in the data attribute information, and the target information digest value in the target attribute information is the same as the data information digest value in the data attribute information, the integrity check result of the target data segment is determined to be complete.

[0217] In this embodiment of the application, determining whether the target length in the target attribute information is the same as the data length in the data attribute information, and determining whether the target information digest value in the target attribute information is the same as the data information digest value in the data attribute information, can be specifically divided into the following four cases:

[0218] The first scenario is that the target length is the same as the data length, and the target message digest value is the same as the data message digest value.

[0219] In this embodiment of the application, if it is determined that the target length is the same as the data length and the target information digest value is the same as the data information digest value, then the integrity verification result of the target data segment is determined to be complete.

[0220] The second scenario: The target length is different from the data length, but the target message digest value is the same as the data message digest value.

[0221] In this embodiment of the application, if it is determined that the target length is different from the data length, but the target information digest value is the same as the data information digest value, then the integrity verification result of the target data segment is determined to be incomplete.

[0222] For example, the target length of the target data segment is calculated to be 184, and the target message digest value of the target data segment is calculated to be 'a'. The data length of the target data segment read from the data block is 182, and the data message digest value of the target data segment read from the data block is 'a'. Therefore, it is determined that the target length of the target data segment is different from the data length, but the target message digest value is the same as the data message digest value. Thus, the integrity check result of the target data segment is determined to be incomplete.

[0223] The third scenario: The target length is the same as the data length, but the target message digest value is different from the data message digest value.

[0224] In this embodiment of the application, if it is determined that the target length is the same as the data length, but the target information digest value is different from the data information digest value, then the integrity verification result of the target data segment is determined to be incomplete.

[0225] The fourth scenario: The target length is different from the data length, and the target message digest value is different from the data message digest value.

[0226] In this embodiment of the application, if it is determined that the target length is different from the data length and the target information digest value is different from the data information digest value, then the integrity verification result of the target data segment is determined to be incomplete.

[0227] N22: When the target length is different from the data length and the target message digest value is different from the data message digest value, the integrity check result of the target data segment is determined to be incomplete.

[0228] In this embodiment of the application, based on the fourth case, it is known that when the target length is different from the data length and the target information digest value is different from the data information digest value, the integrity verification result of the target data segment is determined to be incomplete.

[0229] In this embodiment, multiple data segments are stored in a data block. The integrity of the data segments is ensured by calculating the data length and data digest value. When reading data, the consistency of the data segments can be ensured by verifying the data length and data digest value.

[0230] It should be noted that the CRC used in the embodiments of this application is used for data segment integrity verification and is not limited to CRC32. Other methods can also be used for integrity verification, such as CRC12, CRC16, CRC8, etc.

[0231] In this embodiment, by calling the memory mapping function, the data block is mapped from the memory mapping area to the WAL on the disk, which can solve the performance problem caused by multiple writes and reads, thereby making more effective use of the operating system's features and reducing the performance consumption caused by data storage; at the same time, the data segments to be processed are cached sequentially, which greatly improves the read performance.

[0232] Based on the above embodiments, the WAL processing method in this application embodiment is illustrated below with a specific example. (See attached document.) Figure 4 The diagram shown is an example of the WAL processing method in an embodiment of this application, specifically including:

[0233] First, determine the data length and CRC32 value of the received data segment M to be processed, and then concatenate the data length, CRC32 value and data segment M to be processed according to the preset concatenation format to obtain the concatenated data segment M to be processed. Finally, add the concatenated data segment M to data block P.

[0234] Secondly, since data block P contains data segments M and N to be processed, the total number of data segments to be processed in data block P is determined to be 2. If the total number of data blocks P is not less than the preset number threshold of 2, then data block P is cached from the buffer to the memory mapping area A.

[0235] Then, WAL1, which has a mapping relationship with memory mapping area A, is identified, and the data block P is mapped from memory mapping area A to WAL1 by calling the memory mapping function, thereby obtaining the mapped WAL1.

[0236] Based on the above embodiments, see Figure 5 The diagram shown is a schematic representation of the process for storing WAL in an embodiment of this application, specifically including:

[0237] S500: Enter WAL.

[0238] S501: Receive the data segment to be processed.

[0239] S502: Add the received data segment to be processed to the data block.

[0240] S503: Determine if the data block is full. If yes, execute S504; otherwise, re-execute S501.

[0241] In this embodiment of the application, when determining whether a data block is full, it can be achieved by determining the total number of data segments to be processed contained in the data block. If it is determined that the total number of data segments to be processed contained in the data block is greater than a preset number threshold, then it is determined that the data block is full and S504 is executed. If it is determined that the total number is not greater than the preset number threshold, then it is determined that the data block is not full and S501 is executed again to continue receiving data segments to be processed.

[0242] S504: Cache the data block in the data block cache area.

[0243] In this embodiment, the data block cache area is located in the user-mode storage area.

[0244] S505: Determine whether the total number of data blocks in the data block buffer exceeds the data block quantity threshold. If yes, execute S506; otherwise, execute S507.

[0245] S506: Delete the earliest data block.

[0246] S507: Cache expired.

[0247] In this embodiment of the application, after the caching ends, data blocks are stored in the data block cache area.

[0248] S508: Transfer the data block to the memory-mapped area.

[0249] S509: Call the memory mapping function to map the data block from the memory mapping area to the WAL.

[0250] S510: WAL ends.

[0251] Based on the above embodiments, see Figure 6 The diagram shown is a flowchart illustrating the process of reading a data segment in an embodiment of this application, specifically including:

[0252] S600: Enter WAL.

[0253] S601: Determine whether to read all WALs. If yes, execute S602; otherwise, execute S603.

[0254] In this embodiment of the application, it is determined whether it is necessary to read all data segments in WAL. If so, the entire data segment is mapped from WAL to memory mapping area by calling the memory mapping function. If not, it is determined to read the target data segment.

[0255] S602: By calling the memory mapping function, all data segments are mapped from WAL to the memory-mapped area, and all data segments are read from the memory-mapped area.

[0256] S603: Determine if the data block cache is hit. If yes, execute S604; otherwise, execute S605.

[0257] In this embodiment, based on the data block identifier, it is determined whether there is a target data block corresponding to the data block identifier in the data block cache. If it is determined that there is a target data block corresponding to the data block identifier in the data block cache, then the data block cache is hit, and the corresponding target data segment is read from the data block cache. If it is determined that there is no target data block corresponding to the data block identifier in the data block cache, then the data block cache is not hit, and the corresponding target data segment is read from the memory-mapped area.

[0258] S604: Based on the data segment identifier, read the corresponding target data segment from the data block in the data block cache.

[0259] S605: By calling the memory mapping function, the target data segment corresponding to the data segment identifier is mapped from WAL to the memory mapping area, and the mapped target data segment is read from the memory mapping area.

[0260] S606: WAL ends.

[0261] Based on the above embodiments, see Figure 7 The diagram shown illustrates the process of writing the data segment to be processed to the memory-mapped area in an embodiment of this application, specifically including:

[0262] S700: Enter WAL.

[0263] S701: Waiting for the data segment to be processed to be added to the data block.

[0264] S702: Add to the current data block.

[0265] In this embodiment of the application, the data segment to be processed is added to the current data block.

[0266] S703: Determine whether the storage mode of the data block is multi-segment. If yes, execute S704; otherwise, execute S705.

[0267] S704: Determine if the data block is full. If yes, execute S705; otherwise, re-execute S701.

[0268] In this embodiment of the application, the data block is determined to be full by determining whether the total number of data segments to be processed contained in the data block is greater than a preset number threshold.

[0269] S705: Transfers the data block to the memory-mapped area.

[0270] S706: Execute other processes.

[0271] In this embodiment of the application, other processes include mapping data blocks in the memory-mapped area to the WAL.

[0272] Based on the above embodiments, see Figure 8 The diagram shown illustrates a data writing process according to an embodiment of this application, specifically including:

[0273] S800: Initialize the memory-mapped region.

[0274] S801: Transfer the data block containing the data segment to be processed to the memory-mapped area.

[0275] S802: Determine if the memory-mapped area is full. If yes, execute S803; otherwise, execute S804.

[0276] S803: Generate a new memory-mapped region.

[0277] S804: Execute other processes.

[0278] In this embodiment of the application, other processes include transferring the newly received data block to a newly generated memory-mapped area.

[0279] Based on the above embodiments, see Figure 9 The diagram shown illustrates another data writing process in an embodiment of this application, specifically including:

[0280] S900: Initialize the memory-mapped region.

[0281] S901: Transfer the data block containing the data segment to be processed to the memory-mapped area.

[0282] S902: Determine if the memory mapping area is full. If so, write a new mapping. If not, execute S903.

[0283] S903: Execute other processes.

[0284] In this embodiment of the application, other processes include transferring the newly received data block to the newly acquired memory-mapped area.

[0285] Based on the same inventive concept, embodiments of this application also provide a WAL processing apparatus. See also... Figure 10 The diagram shown is a structural schematic of the WAL processing device in an embodiment of this application, which may include:

[0286] Add module 1000, which is used to receive data segments to be processed and add the received data segments to a preset data block. The data block is set in the buffer of the user space storage area.

[0287] Processing module 1010 is used to transfer the data block from the buffer to the memory-mapped area when the total number of data segments to be processed contained in the data block meets the preset quantity condition. The memory-mapped area is set in the user-mode storage area.

[0288] The mapping module 1020 is used to call a preset memory mapping function to map data blocks from the memory mapping area to the write-ahead log (WAL) on the disk, and obtain the mapped WAL.

[0289] Optionally, when adding the received data segment to be processed to the preset data block, the adding module 1000 is also used for:

[0290] Determine the data attribute information corresponding to the received data segment to be processed;

[0291] According to the preset splicing format, the data attribute information and the data segment to be processed are spliced ​​together to obtain the spliced ​​data segment to be processed.

[0292] Add the spliced ​​data segments to be processed into the preset data blocks.

[0293] Optionally, the processing module 1010 is also used for:

[0294] If it is determined that the storage mode corresponding to the data block is a single data segment, then the data block is transferred from the buffer to the memory-mapped area;

[0295] If the storage mode is determined to be multi-segment, then when the total number of data segments to be processed contained in the data block is greater than the preset number threshold, the data block will be transferred from the buffer to the memory-mapped area.

[0296] Optionally, when transferring data blocks from the buffer to the memory-mapped area, the processing module 1010 is also used to:

[0297] When the length of the data block corresponding to the data block is determined to be greater than the remaining memory capacity of the memory mapping area, a memory mapping area with an idle working state is randomly read from the preset mapping pool. The mapping pool contains multiple memory mapping areas.

[0298] The data block is transferred to the read memory-mapped area.

[0299] Optionally, when transferring data blocks from the buffer to the memory-mapped area, the processing module 1010 is also used to:

[0300] When the length of the data block corresponding to the data block is determined to be greater than the remaining memory capacity of the memory mapping area, a new memory mapping area is generated in the user space storage area.

[0301] The data block is transferred from the buffer to the new memory-mapped area.

[0302] Optionally, the mapping module 1020 is also used for:

[0303] Based on the mapping relationship between each memory mapping region and each WAL, determine the WAL that is mapped to the memory mapping region;

[0304] Call the memory mapping function to map the data block from the memory mapping area to the determined WAL, and obtain the mapped WAL.

[0305] Optionally, the device also includes a data reading module 1030, which is used for:

[0306] Receive a data read request, which must include at least a data index identifier;

[0307] Based on the data index identifier, the target data segment is read from the corresponding data block.

[0308] Optionally, when reading the target data segment from the corresponding data block based on the data index identifier, the data reading module 1030 is also used for:

[0309] Call the memory mapping function to map the target data segment determined based on the data index identifier from WAL to the memory mapping area, and obtain the mapped memory mapping area.

[0310] Optionally, after adding the received data segment to be processed to a preset data block, the device further includes a cache module 1040, which is used for:

[0311] Data blocks are cached in the data block cache area, which is located in the user-space storage area.

[0312] Optionally, when reading the target data segment from the corresponding data block based on the data index identifier, the data reading module 1030 is also used for:

[0313] When it is determined that a target data block corresponding to a data block identifier exists in the data block cache, the target data segment corresponding to the data segment identifier is read from the target data block in the data block cache.

[0314] Optionally, the device also includes a verification module 1050, which is used for:

[0315] Calculate the target attribute information corresponding to the target data segment, and determine the data attribute information corresponding to the target data segment from the corresponding data block;

[0316] Based on the target attribute information and data attribute information, determine the integrity verification result of the target data segment.

[0317] Optionally, when determining the integrity verification result of the target data segment based on the target attribute information and data attribute information, the verification module 1050 is also used for:

[0318] When the target length in the target attribute information is the same as the data length in the data attribute information, and the target information digest value in the target attribute information is the same as the data information digest value in the data attribute information, the integrity verification result of the target data segment is determined to be complete.

[0319] When the target length is different from the data length and the target message digest value is different from the data message digest value, the integrity check result of the target data segment is determined to be incomplete.

[0320] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0321] In some possible implementations, the WAL processing apparatus according to this application may include at least a processor and a memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps of the WAL processing methods according to various exemplary embodiments of this application described in this specification. For example, the processor may perform actions such as... Figure 3A The steps are shown in the figure.

[0322] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device. In one embodiment, the electronic device may be, for example... Figure 2A As shown at node 201, in this embodiment, the structure of the electronic device can be as follows: Figure 11 As shown, it includes a memory 1101, a communication module 1103, and one or more processors 1102.

[0323] The memory 1101 is used to store computer programs executed by the processor 1002. The memory 1001 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0324] Memory 1101 may be volatile memory, such as random-access memory (RAM); memory 1101 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1101 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1101 may be a combination of the above-described memories.

[0325] Processor 1102 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 1102 is used to implement the above-described WAL processing method when calling computer programs stored in memory 1101.

[0326] The communication module 1103 is used to communicate with terminal devices and other servers.

[0327] This application embodiment does not limit the specific connection medium between the memory 1101, communication module 1103, and processor 1102. This application embodiment... Figure 11 The memory 1101 and the processor 1102 are connected via a bus 1104, and the bus 1104 is in Figure 11 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. Bus 1104 can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 11 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0328] The memory 1101 stores a computer storage medium, which stores computer-executable instructions for implementing the WAL processing method of this application embodiment. The processor 1102 is used to execute the above-described WAL processing method, such as... Figure 3A As shown.

[0329] In some possible implementations, various aspects of the WAL processing method provided in this application can also be implemented as a program product comprising program code. When the program product is run on a computer device, the program code causes the computer device to perform the steps of the WAL processing method according to the various exemplary embodiments of this application described above. For example, the computer device can perform actions such as... Figure 3A The steps are shown in the figure.

[0330] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0331] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a computing device. However, the program product of this application is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0332] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0333] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0334] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0335] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0336] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0337] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0338] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0339] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A WAL processing method, characterized in that, include: Receive the data segment to be processed and add the received data segment to a preset data block, wherein the data block is set in the buffer of the user-mode storage area; When the total number of data segments to be processed contained in the data block is determined to meet the preset quantity condition, the data block is transferred from the buffer to the memory mapping area, which is set in the user-mode storage area; Call the preset memory mapping function to map the data block from the memory mapping area to the write-ahead log (WAL) on the disk, and obtain the mapped WAL.

2. The method as described in claim 1, characterized in that, The step of adding the received data segment to be processed to a preset data block includes: Determine the data attribute information corresponding to the received data segment to be processed; According to the preset splicing format, the data attribute information and the data segment to be processed are spliced ​​together to obtain the spliced ​​data segment to be processed. The spliced ​​data segment to be processed is added to a preset data block.

3. The method as described in claim 1, characterized in that, When the total number of data segments to be processed contained in the data block is determined to meet a preset quantity condition, the data block is transferred from the buffer to the memory-mapped area, including: If it is determined that the storage mode corresponding to the data block is a single data segment, then the data block is transferred from the buffer to the memory-mapped area; If the storage mode is determined to be multi-segment, then when the total number of data segments to be processed contained in the data block is greater than a preset threshold, the data block is transferred from the buffer to the memory-mapped area.

4. The method according to any one of claims 1-3, characterized in that, The step of transferring the data block from the buffer to the memory-mapped area includes: When it is determined that the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, a memory mapping area with an idle working state is randomly read from the preset mapping pool, and the mapping pool contains multiple memory mapping areas; The data block is transferred to the read memory-mapped area.

5. The method according to any one of claims 1-3, characterized in that, The step of transferring the data block from the buffer to the memory-mapped area includes: When it is determined that the length of the data block corresponding to the data block is greater than the remaining memory capacity of the memory mapping area, a new memory mapping area is generated in the user-mode storage area; The data block is transferred from the buffer to the new memory-mapped area.

6. The method as described in claim 5, characterized in that, The step of calling a preset memory mapping function to map the data block from the memory mapping area to the WAL on the disk, and obtaining the mapped WAL, includes: Based on the mapping relationship between each memory mapping region and each WAL, determine the WAL mapped to the memory mapping region; The memory mapping function is called to map the data block from the memory mapping area to the determined WAL, thereby obtaining the mapped WAL.

7. The method according to any one of claims 1-3, characterized in that, The method further includes: Receive a data read request, wherein the data read request includes at least a data index identifier; Based on the data index identifier, the target data segment is read from the corresponding data block.

8. The method as described in claim 7, characterized in that, The step of reading the target data segment from the corresponding data block based on the data index identifier includes: The memory mapping function is called to map the target data segment determined based on the data index identifier from the WAL to the memory mapping area, thereby obtaining the mapped memory mapping area.

9. The method as described in claim 7, characterized in that, After adding the received data segment to be processed to the preset data block, the method further includes: The data blocks are cached in the data block cache area, which is set in the user-mode storage area.

10. The method as described in claim 9, characterized in that, The data index identifier includes a data block identifier and a data segment identifier; The step of reading the target data segment from the corresponding data block based on the data index identifier includes: When it is determined that a target data block corresponding to the data block identifier exists in the data block cache, the target data segment corresponding to the data segment identifier is read from the target data block in the data block cache.

11. The method as described in claim 7, characterized in that, The method further includes: Calculate the target attribute information corresponding to the target data segment, and determine the data attribute information corresponding to the target data segment from the corresponding data block; Based on the target attribute information and the data attribute information, the integrity verification result of the target data segment is determined.

12. The method as described in claim 11, characterized in that, The step of determining the integrity verification result of the target data segment based on the target attribute information and the data attribute information includes: When it is determined that the target length in the target attribute information is the same as the data length in the data attribute information, and the target information digest value in the target attribute information is the same as the data information digest value in the data attribute information, the integrity verification result of the target data segment is determined to be complete. When it is determined that the target length is different from the data length, and the target information digest value is different from the data information digest value, the integrity verification result of the target data segment is determined to be incomplete.

13. A WAL processing apparatus, characterized in that, include: An add module is used to receive data segments to be processed and add the received data segments to a preset data block, wherein the data block is set in the buffer of the user-mode storage area; The processing module is used to transfer the data block from the buffer to the memory-mapped area when the total number of data segments to be processed contained in the data block meets the preset quantity condition. The memory-mapped area is set in the user-mode storage area. The mapping module is used to call a preset memory mapping function to map the data block from the memory mapping area to the write-ahead log (WAL) on the disk, and obtain the mapped WAL.

14. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores program code that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1-12.

15. A computer-readable storage medium, characterized in that, It includes program code that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1-12.

16. A computer program product, characterized in that, It includes computer instructions stored in a computer-readable storage medium; when a processor of an electronic device reads the computer instructions from the computer-readable storage medium, the processor executes the computer instructions, causing the electronic device to perform the steps of any of the methods described in claims 1-12.

Images