Computing system, data processing method and related equipment

Abstract

The invention provides a computing system, a data processing method and related equipment, and relates to the technical field of databases. Wherein the computing system comprises: a processor; the main memory is used for storing data; the extended memory comprises N levels of first file layers; the hard disk comprises M levels of second file layers; wherein the first-level second file layer in the M-level second file layers is used for storing data transferred from the Nth-level first file layer. According to the scheme, data processing time consumption can be reduced, and data access response speed is increased.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a computing system, a data processing method and related equipment. Background Art

[0002] In the data storage structure, the database usually uses the main memory to store temporary and latest data, and divides multiple storage levels on the hard disk to store persistent data. However, due to the limited memory space available in the main memory, a large amount of data is stored on the hard disk.

[0003] When accessing stored data through a database, the data access response is slow and the data reading performance is poor due to the large data storage capacity in the hard disk and the low hard disk I / O (Input / Output) performance. Summary of the invention

[0004] The embodiments of the present application provide a computing system, a data processing method and related devices to solve the problem of slow data access response.

[0005] In a first aspect, an embodiment of the present application provides a computing system, including:

[0006] processor;

[0007] Main memory, used to store data;

[0008] The extended memory includes N levels of first file layers, where N≥1; wherein the first level first file layer in the N levels of first file layers is used to store data transferred from the main memory; when N>1, the i-th level first file layer in the N levels of first file layers is used to store data transferred from the i-1th level first file layer, 1 <i≤N;

[0009] A hard disk, wherein the hard disk comprises M-level second file layers, wherein M>1; wherein the first-level second file layer in the M-level second file layers is used to store data transferred from the N-level first file layer; the j-th-level second file layer in the M-level second file layers is used to store data transferred from the j-1-th-level second file layer, wherein 1 <j≤M。

[0010] The extended memory supports the creation of N-level first file layers, the data is transferred between the first-level first file layer in the N-level first file layers and the main memory, and the data is transferred between the N-level first file layer and the hard disk.

[0011] N can be set to 1, in which case the extended memory contains only the first file layer of level 1. The first file layer of level 1 is the first file layer of level N.

[0012] N can be set to be greater than 1. In this case, the extended memory includes multiple levels of first file layers. The first level first file layer and the Nth level first file layer are different file layers.

[0013] When N is greater than 1, the first-level file layer in the N-level first file layer is used to store data transferred from the main memory, the i-th-level first file layer is used to store data transferred from the i-1-th-level first file layer, and the N-th-level first file layer is used to transfer and write the stored data to the hard disk.

[0014] The hard disk supports the creation of an M-level second file layer, and data is transferred between the first-level second file layer in the M-level second file layer and the extended memory.

[0015] The first-level second file layer in the M-level second file layers is used to store data transferred from the N-level first file layer, and the j-th-level second file layer is used to store data transferred from the j-1-th-level second file layer.

[0016] In this embodiment, data can be transferred between the main memory and the extended memory, and can be transferred between the extended memory and the hard disk. Due to the addition of the extended memory, the process of transferring data from the main memory to the hard disk changes. The data is no longer written directly to the hard disk, but the data needs to be transferred downward to the extended memory first, and then the extended memory further transfers the data stored therein downward to the hard disk. The extended memory and the hard disk are different storage media. File layers are divided in the extended memory. The main memory is expanded with the help of the extended memory with a file hierarchy, and the hard disk layering is adapted to realize the data storage connection of the hard disk with multiple file layers.

[0017] The data in the extended memory can be transferred between the main memory and the hard disk based on the divided file layers, realizing the transitional storage of data between the main memory and the hard disk. The added extended memory serves as the transitional storage space between the main memory and the hard disk to realize the caching of data, so that these data can be read and written by memory instead of hard disk reading and writing, reducing the access operations of data operations to the hard disk, reducing the interference of low hard disk I / O performance on data access efficiency, reducing data processing time, improving data access response speed, and improving the read and write performance of the overall process.

[0018] In some optional implementations, when N is greater than 1, the Nth-level first file layer in the N-level first file layers in the extended memory is used to store data transferred from the hard disk.

[0019] Optionally, the i-th level first file layer is used to store data transferred in the (i+1)-th level first file layer, and the first-level first file layer is used to transfer data to the main memory.

[0020] Or, optionally, the j-1th level second file layer is used to store data transferred from the jth level second file layer, and the data in the first level second file layer is transferred to the Nth level first file layer.

[0021] This situation corresponds to the implementation process of some data reading. The data that needs to be read in the hard disk can be transferred layer by layer to the extended memory so that the processor can read it, or the data that needs to be read can be transferred layer by layer from the extended memory to the main memory so that the processor can read it, adapting to the data reading needs in different situations.

[0022] In combination with the first aspect, in some implementations of the first aspect, the extended memory includes a first group of the N-level first file layers, and a second group of the N-level first file layers.

[0023] The first-level first file layer in the first group of N-level first file layers is used to store the first target key transferred from the main memory; the first-level first file layer in the second group of N-level first file layers is used to store the first target value corresponding to the first target key transferred from the main memory; and / or, the hard disk includes the first group of M-level second file layers, and the second group of M-level second file layers; the first-level second file layer in the first group of M-level second file layers is used to store the second target key transferred from the N-th level first file layer in the first group of N-level first file layers in the extended memory; the first-level second file layer in the second group of M-level second file layers is used to store the second target value corresponding to the second target key transferred from the N-th level first file layer in the second group of N-level first file layers in the extended memory.

[0024] It realizes the separate storage of keys and values ​​in key-value pairs, reduces the metadata required for storage, improves the matching efficiency when using keys to search for data, and improves the efficiency of data transfer.

[0025] In combination with the first aspect, in certain implementations of the first aspect, the extended memory is a memory that uses the computing high-speed link CXL protocol to achieve efficient and low-latency interconnection with the server, achieve high-performance and low-latency data transmission and memory sharing, and ensure the access efficiency of data in the extended memory.

[0026] In combination with the first aspect, in certain implementations of the first aspect, a variable memory table for storing data is stored in the main memory, wherein the variable memory table is converted into an immutable memory table when the amount of data written reaches a preset condition; and the first-level first file layer is used to store data transferred from the immutable memory table.

[0027] Immutable memory tables provide an intermediate state for data storage. The setting of immutable memory tables allows write operations to continue on the newly created mutable memory tables. The conversion of mutable memory tables into immutable memory tables ensures that the data in them can be safely written to the extended memory by the background thread, reducing the impact on database performance.

[0028] In combination with the first aspect, in some implementations of the first aspect, a write-ahead log is stored in the extended memory; and / or a manifest file for storing metadata is stored in the extended memory.

[0029] Manifest files are usually used to record and manage metadata such as the structure, content, configuration, and permissions of a database. These metadata are used to implement functions such as the correct operation of the database, data management, and user access.

[0030] In the write-ahead log mechanism, all write operations first write data to the write-ahead log, and then perform the write operation, which is used to ensure that in the event of a system crash or other failure, the data can be restored to a consistent state based on the write-ahead log. In this way, even if a failure occurs during the data writing process, the log file can be used to recover the unfinished write operation.

[0031] In combination with the first aspect, the computing system includes a computing device and an extended memory device, the computing device includes the processor and the main memory; the extended memory device includes the extended memory; or the computing system includes a computing device, the computing device includes the processor, the main memory, and the extended memory.

[0032] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a computing system, characterized in that the computing system includes a main memory, an extended memory, and a hard disk; the extended memory includes an N-level first file layer, where N≥1; the hard disk includes an M-level second file layer, where M>1; the data processing method includes:

[0033] When a first preset condition is met, the first target data in the main memory is written into the first-level first file layer of the extended memory, wherein, when N>1, the i-th level first file layer in the N-level first file layers is used to store the data transferred in the i-1-th level first file layer, 1 <i≤N;

[0034] When the second preset condition is met, the second target data stored in the Nth level first file layer of the extended memory is written into the first level second file layer of the hard disk, wherein the jth level second file layer in the M level second file layers is used to store the data transferred in the j-1th level second file layer, 1 <j≤M。

[0035] The first target data may be part of the data or all of the data stored in the main memory.

[0036] The second target data may be part of or all of the data stored in the Nth level first file layer of the extended memory.

[0037] During this process, the data transfer process from the main memory to the hard disk changes. It is no longer written directly to the hard disk, but the data needs to be transferred down to the extended memory first, and then the extended memory further transfers the data stored in it down to the hard disk.

[0038] By dividing the file layers in the extended memory, the main memory is expanded with the help of the extended memory with file layers, and the hard disk layering is adapted to realize the data storage connection of the hard disk with multiple file layers.

[0039] By introducing extended memory between the main memory and the hard disk, data can be transferred between the first-level file layer in the N-level file layers it includes and the main memory, and data can be transferred between the N-th level file layer and the hard disk. With the help of the extended memory of the storage layer with data files, layer-by-layer caching of data can be achieved, reducing the access to the hard disk for data operations, reducing the interference of low hard disk I / O performance on data access efficiency, reducing data processing time, and improving data access response speed in the computing system.

[0040] In combination with the second aspect, in certain implementations of the second aspect, when the remaining storage space of the main memory is less than a first threshold, it is determined that the first preset condition is met; or, when a first specific user operation is detected, it is determined that the first preset condition is met.

[0041] The main memory, extended memory and hard disk each have a set storage space size.

[0042] When storing data in the main memory, extended memory and hard disk, the total storage space, used space and remaining available space of the main memory, extended memory and hard disk can be obtained to determine whether the remaining storage space in the corresponding storage medium is sufficient.

[0043] The timing of whether to transfer data in the main memory to the extended memory is determined by combining the remaining storage space size of the main memory or the user operation situation.

[0044] In combination with the second aspect, in some implementations of the second aspect, when the remaining storage space of the main memory is less than a first threshold, determining that the first preset condition is satisfied includes:

[0045] If the remaining storage space of the variable memory table in the main memory is less than the amount of data to be stored, the variable memory table is converted into an immutable memory table, and when the remaining storage space in the main memory is insufficient to create a new variable memory table, it is determined that the remaining storage space of the main memory is less than the first threshold.

[0046] This process combines the data volume of different memory tables in the main memory to determine when it is necessary to transfer the data in the main memory to the extended memory.

[0047] In conjunction with the second aspect, in some implementations of the second aspect, the data processing method further includes:

[0048] When the third preset condition is met, the third target data stored in the i-1th level first file layer in the N-level first file layers is stored in the i-th level first file layer; 1 <i≤N。

[0049] The third preset condition may be a user clicking a specific command button to trigger an operation. The user may trigger the migration of data in the extended memory from the first file layer of the i-1th level to the first file layer of the ith level through a specific operation.

[0050] Alternatively, the third preset condition may be that the remaining storage space in the first file layer of the i-1th level in the extended memory is less than a threshold.

[0051] It realizes the timing judgment of whether data needs to be transferred between different file layers in the extended memory.

[0052] In combination with the second aspect, in some implementations of the second aspect, when a third preset condition is satisfied, storing the third target data stored in the i-1th level first file layer in the N-level first file layers into the i-th level first file layer includes:

[0053] When the remaining storage space in the first file layer of the i-1th level is less than the second threshold, starting from the first file layer of the i-1th level, the third target data stored in the first file layer of the current level is written into the first file layer of the i-th level;

[0054] Until the remaining storage space in the first file layer of the (i-1)th level is greater than the second threshold.

[0055] The third target data is part of or all of the data stored in the i-1th level first file layer of the extended memory.

[0056] This process ensures that the upper file layer has enough remaining storage space after transferring data to the lower file layer by transferring data between the various file layers in the extended memory, thereby ensuring the effective storage of data in different file layers in the extended memory.

[0057] In conjunction with the second aspect, in some implementations of the second aspect, the data processing method further includes:

[0058] If the remaining storage space of the variable memory table in the main memory is greater than the amount of data to be stored, then when the remaining storage space of the write-ahead log in the extended memory is greater than the amount of backup data of the data to be stored, the data to be stored is written into the variable memory table.

[0059] In this process, even if a failure occurs during data writing, the write-ahead log can be used to recover the unfinished write operation, ensuring the effective writing of data in the database.

[0060] In combination with the second aspect, in certain implementations of the second aspect, when the remaining storage space of the first file layer of the Nth level is less than a fourth threshold, it is determined that the second preset condition is met; or, when a second specific user operation is detected, it is determined that the second preset condition is met.

[0061] In some scenarios, such as when it is necessary to ensure that the extended memory has sufficient remaining storage space, the user can trigger the migration of data in the extended memory from the first file layer of level N to the hard disk through a specific operation.

[0062] The extended memory has a set storage space size. When the remaining storage space of the Nth level first file layer of the extended memory is less than the fixed value, the data in the extended memory is triggered to migrate from the Nth level first file layer to the hard disk.

[0063] It can determine when to transfer data in the extended memory to the hard disk.

[0064] In conjunction with the second aspect, in some implementations of the second aspect, the data processing method further includes:

[0065] When the fourth preset condition is met, the fourth target data stored in the j-1th level second file layer in the M-level second file layers is stored in the jth level second file layer; 1 <j≤M。

[0066] The fourth preset condition may be a user clicking a specific command button to trigger an operation. The user may trigger the migration of data in the hard disk from the j-1th level second file layer to the jth level second file layer through a specific operation. Alternatively, the fourth preset condition may be that the remaining storage space in the j-1th level first file layer in the hard disk is less than a threshold value.

[0067] It can determine when data needs to be transferred between different file layers in the hard disk.

[0068] In combination with the second aspect, in some implementations of the second aspect, when a fourth preset condition is satisfied, storing the fourth target data stored in the j-1th level second file layer in the M-level second file layers into the jth level second file layer includes:

[0069] When the remaining storage space in the j-1th level second file layer is less than the fifth threshold, starting from the j-1th level second file layer, the fourth target data stored in the current level second file layer is written into the j-th level second file layer;

[0070] Until the remaining storage space in the j-1th level second file layer is greater than the fifth threshold.

[0071] The fourth target data is part of or all of the data stored in the j-1th level second file layer of the hard disk.

[0072] This process ensures that the upper file layer has enough remaining storage space after transferring data to the lower file layer by transferring data between the various file layers of the hard disk, thereby ensuring the effective storage of data in different file layers in the hard disk.

[0073] In conjunction with the second aspect, in some implementations of the second aspect, the extended memory includes a first group of the N-level first file layers, and a second group of the N-level first file layers;

[0074] The step of writing the first target data in the main memory to the first-level first file layer of the extended memory when the first preset condition is met includes:

[0075] When the first preset condition is met, the first target key in the main memory is written into the first level first file layer in the first group of the N level first file layers; and / or the first target value corresponding to the first target key in the main memory is written into the first level first file layer in the second group of the N level first file layers;

[0076] The step of writing the second target data stored in the Nth level first file layer of the extended memory to the first level second file layer of the hard disk when the second preset condition is met includes:

[0077] When the second preset condition is met, the second target key stored in the N-th level first file layer in the first group of the N-level first file layers in the extended memory is written to the hard disk, and / or the second target value corresponding to the second target key stored in the N-th level first file layer in the second group of the N-level first file layers in the extended memory is written to the hard disk.

[0078] The above process stores the key and value separately in the key-value pair data, reduces the metadata required to be stored, improves the matching efficiency when using the key to search for data, and improves the efficiency of data transfer.

[0079] In conjunction with the second aspect, in some implementations of the second aspect, before writing the first target data in the main memory into the first-level first file layer of the extended memory when the first preset condition is met, the method further includes:

[0080] Formatting the extended memory using a memory file system;

[0081] Configure initialization parameters, wherein the initialization parameters at least include the file directory of the extended memory, the storage directory of the write-ahead log in the extended memory, the size of the first file layer of level N in the extended memory, and the number of file layers;

[0082] If it is detected that the remaining storage space in the extended memory meets the required capacity of the N-level first file layer, the N-level first file layer is initialized in the extended memory based on the initialization parameter, and the write-ahead log is initialized.

[0083] In this way, effective initialization construction of the N-level first file layer in the newly added extended memory is achieved.

[0084] In conjunction with the second aspect, in some implementations of the second aspect, the data processing method further includes:

[0085] When the fifth preset condition is met, fifth target data matching the data to be read is read from the main memory; or,

[0086] When a sixth preset condition is met, the fifth target data is read from the extended memory.

[0087] In this way, in the improved storage architecture, the processor can read the target data from the main memory in the conventional way, minimizing the modification of the original system under the data reading function after the newly added extended memory, which is easy to implement; or the processor can also directly read the target data from the newly added extended memory, which is convenient for direct and effective acquisition of data in the memory and improves the data reading response speed. Different data reading implementation methods are provided to meet the data reading implementation requirements in different scenarios.

[0088] In combination with the second aspect, in some implementations of the second aspect, when a fifth preset condition is met, reading fifth target data matching the data to be read from the main memory includes:

[0089] In a case where fifth target data matching the data to be read is stored in the main memory, reading the fifth target data from the main memory; or,

[0090] In the case where the fifth target data matching the data to be read is stored in the extended memory, the fifth target data is written from the first level first file layer of the extended memory to the main memory, and the fifth target data is read from the main memory; or

[0091] In the case where the fifth target data matching the data to be read is stored in the hard disk, the fifth target data is written from the hard disk to the Nth-level first file layer of the extended memory, and the fifth target data is written from the first-level first file layer of the extended memory to the main memory, and the fifth target data is read from the main memory.

[0092] In the above implementation process, corresponding data reading processing is performed according to the different storage situations of the target data in the main memory, extended memory and hard disk, and the storage level of the data file set in the extended memory is used to transfer the target data between the hard disk, extended memory and main memory, so as to realize the final reading of the target data from the main memory.

[0093] In combination with the second aspect, in some implementations of the second aspect, writing the fifth target data from the first-level first file layer of the extended memory to the main memory includes:

[0094] Determine the target-level first file layer where the fifth target data is located in the extended memory;

[0095] When N is greater than 1 and the target-level first file layer is not the first-level first file layer, starting from the target-level first file layer of the extended memory, the fifth target data stored in the current-level first file layer is written to the adjacent previous-level first file layer until the fifth target data is written to the first-level first file layer, and the fifth target data is written from the first-level first file layer of the extended memory to the main memory.

[0096] In this way, by utilizing the multiple storage levels of the data files set in the extended memory and transferring the target data between the multiple storage levels in the extended memory, data can be transferred between the hard disk, the extended memory and the main memory, and the target data can be finally read from the main memory.

[0097] In combination with the second aspect, in some implementations of the second aspect, when a sixth preset condition is met, reading the fifth target data from the extended memory includes:

[0098] In the case where the fifth target data is stored in the extended memory, the fifth target data is read from the extended memory; or, in the case where the fifth target data is stored in the hard disk, the fifth target data is transferred from the hard disk to the Nth level first file layer of the extended memory, and the fifth target data is read from the extended memory.

[0099] In this way, the target data can be directly read from the extended memory, improving data reading efficiency and speed.

[0100] In conjunction with the second aspect, in some implementations of the second aspect, the hard disk includes an M-level second file layer, the first-level second file layer is used to transfer stored data to the N-th-level first file layer, and M is greater than 1;

[0101] The step of transferring the fifth target data from the hard disk to the Nth level first file layer of the extended memory comprises:

[0102] Determine the target-level second file layer where the fifth target data is located in the hard disk;

[0103] In the case where the target-level second file layer is the first-level second file layer in the hard disk, writing the fifth target data from the first-level second file layer into the Nth-level first file layer of the extended memory; or

[0104] In the case that the target-level second file layer is not the first-level second file layer, starting from the target-level second file layer of the hard disk, the fifth target data stored in the current-level second file layer is written to the adjacent previous-level second file layer until the fifth target data is written to the first-level second file layer, and the fifth target data is written from the first-level second file layer to the Nth-level first file layer of the extended memory.

[0105] In this process, the j-1th level second file layer in the hard disk is used to store data transferred from the jth level second file layer.

[0106] In this way, by utilizing multiple storage levels of data files set in the hard disk and by transferring target data between various file levels of the hard disk, data can be transferred from the hard disk to the extended memory, ensuring the orderliness of data access.

[0107] A third aspect of an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the second aspect when executing the computer program.

[0108] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method described in the second aspect are implemented.

[0109] The fifth aspect of an embodiment of the present application provides a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying a computer-readable code. When the computer-readable code is executed in an electronic device, the processor in the electronic device executes the steps in the method described in the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Moreover, the same reference numerals are used throughout the drawings to represent the same components. In the drawings:

[0111] Figure 1 A structural diagram of a computing system provided in an embodiment of the present application;

[0112] Figure 2 A data storage architecture structure provided in the embodiment of the present application Figure 1 ;

[0113] Figure 3 A data storage architecture structure provided in the embodiment of the present application Figure 2 ;

[0114] Figure 4 A data processing method provided in the embodiment of the present application Figure 1 ;

[0115] Figure 5 A data processing method provided in the embodiment of the present application Figure 2 ;

[0116] Figure 6 is a structural diagram of a data processing device in some embodiments of the present application;

[0117] Figure 7 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0118] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0119] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0120] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0121] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0122] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0123] Before describing the solutions in the embodiments of the present application, the relevant concepts are first introduced:

[0124] Main memory, also known as RAM, is temporary storage in a server for storing programs and data. In a server, main memory is usually located close to the processor.

[0125] Extended memory is outside the main memory and serves as an extension of the main memory.

[0126] The file layer is a data storage layer divided in the storage space based on the file system. The data storage layer is the structure in which the file system organizes and manages data. This hierarchical structure helps to efficiently access and manage data on storage devices.

[0127] In order to illustrate the technical solution described in the embodiments of the present application, a specific embodiment is provided below for illustration.

[0128] In some embodiments, in combination Figure 1As shown, a computing system is proposed, including:

[0129] A processor;

[0130] A main memory for storing data;

[0131] An extended memory, the extended memory includes N levels of first file layers, where N ≥ 1; among them, the first-level first file layer in the N-level first file layer is used to store the data transferred from the main memory; when N > 1, the i-level first file layer in the N-level first file layer is used to store the data transferred from the (i - 1)-level first file layer, 1 < i ≤ N;

[0132] A hard disk, the hard disk includes M levels of second file layers, where M > 1; among them, the first-level second file layer in the M-level second file layer is used to store the data transferred from the N-level first file layer; the j-level second file layer in the M-level second file layer is used to store the data transferred from the (j - 1)-level second file layer, 1 < j ≤ M. The above computing system corresponds to a specific hardware architecture.

[0133] In some alternative embodiments, the hardware architecture corresponding to the computing system includes a computing device (such as a server device, a terminal device with computing capabilities, etc.), and an independent extended memory device connected to the computing device. Optionally, the extended memory included in the extended memory device is a memory using the CXL (Compute Express Link) protocol to achieve efficient and low-latency interconnection with the server, realize high-performance and low-latency data transmission and memory sharing, and ensure the access efficiency of data in the extended memory. In this architecture, the processor and the main memory are located in the computing device. In this case, the computing system forms a system including a computing device and an extended memory device independently set from the computing device.

[0134] In an alternative embodiment, the extended memory can be formed into a storage pool for data storage using the CXL protocol. Among them, the memory storage device based on the CXL protocol can be called a CXL memory component, which can effectively realize the extension of the storage device and increase the storage capacity of the server.

[0135] In some implementation processes, multiple CXL memory components can be resource-integrated based on pooling technology to obtain a memory device including multiple CXL memory components. Among them, multiple CXL memory components form storage modules in the remote memory device. Different servers can establish an interconnected network based on the cache coherence protocol with the remote memory device. The processors in the servers can directly access these storage modules in the remote memory device, which can maximize the memory capacity and meet the memory requirements of data-intensive applications.

[0136] In some other optional implementations, the hardware architecture corresponding to the computing system includes a computing device. The processor, main memory, and extended memory are all arranged in the computing device. In this case, the computing system is formed as a system including the computing device.

[0137] Optionally, in the above two hardware architectures, the hard disk can be built into the computing device or externally placed in the computing device, and can be configured according to requirements.

[0138] In the above computing system, a data storage architecture is formed between the main memory, the extended memory and the hard disk.

[0139] Combination Figure 2 As shown, in this data storage architecture, the extended memory and the hard disk are different storage media, and file layers are divided in the extended memory. The main memory is expanded with the help of the extended memory with file layers, and the hard disk layering is adapted to realize the data storage connection of the hard disk with multiple file layers.

[0140] Data can be transferred between the main memory and the extended memory, and between the extended memory and the hard disk. The processor can directly read the data in the main memory or the extended memory.

[0141] The data in the extended memory can be transferred to the main memory and the hard disk based on the divided file layers, realizing the transitional storage of data between the main memory and the hard disk, and caching the data in the extended memory, so that the data can be read and written by memory instead of hard disk, to improve the read and write performance of the overall process.

[0142] Extended memory supports the creation of N-level first file tiers.

[0143] The extended memory transfers data between the first-level first file layer in the N-level first file layers and the main memory, and transfers data between the N-level first file layer and the hard disk.

[0144] In some optional embodiments, in combination Figure 2 As shown, a variable memory table for storing data is stored in the main memory, wherein the variable memory table is converted into an immutable memory table when the amount of data written reaches a preset condition. A new variable memory table will be created in the main memory accordingly, and new data will be continuously written into the variable memory table. The converted immutable memory table no longer accepts new write operations, but can still be read.

[0145] Among them, the mutable memory table is a data structure in the memory, which is used to temporarily store the data in the database. Compared with the immutable memory table, the content of the mutable memory table can be modified or updated at runtime.

[0146] Immutable memtable is another data structure in memory, as opposed to mutable memtable. Once data is written to an immutable memtable, it can no longer be modified or deleted (unless the entire table is replaced or deleted).

[0147] Immutable memory tables provide an intermediate state for data storage. The setting of immutable memory tables allows write operations to continue on the newly created mutable memory tables. The conversion of mutable memory tables into immutable memory tables ensures that the data in them can be safely written to the extended memory by the background thread, reducing the impact on database performance.

[0148] In the N-level first file layer in the extended memory, the first-level first file layer is used to store data transferred from the immutable memory table.

[0149] The preset condition is, for example, that the amount of data written to the variable memory table reaches a quantity threshold, or that the remaining storage space in the variable memory table is insufficient to write the data to be stored, etc. In this way, the data in the immutable memory table is transferred to the extended memory, and the remaining storage space in the main memory increases.

[0150] After the mutable memory table is converted to an immutable memory table, the data table is converted from readable and writable to readable and non-writable. In addition, a new mutable memory table can be created if there is sufficient remaining storage space in the main memory. With the conversion of the data table in the main memory, data can be transferred from the main memory to the newly added extended memory in an orderly manner.

[0151] During configuration, the number of layers N of the first file layer of the extended memory can be configured. N is greater than or equal to 1.

[0152] N can be set to 1, in which case the extended memory only contains the first file layer of level 1. The first file layer of level 1 is the first file layer of level N.

[0153] N can be set to be greater than 1. In this case, the extended memory contains multiple levels of first file layers. The first level first file layer and the Nth level first file layer are different file layers. Figure 2 As shown in . Data can be transferred between the multi-level file layers of the extended memory, and the data can be stored hierarchically in the extended memory to adapt to the multi-level storage requirements of different data in the extended memory.

[0154] When N is greater than 1, on the one hand, the first-level file layer in the N-level first file layer is used to store data transferred from the main memory, the i-th-level first file layer is used to store data transferred from the i-1-th-level first file layer, and the N-th-level first file layer is used to transfer and write the stored data to the hard disk.

[0155] On the other hand, the Nth level first file layer in the N level first file layers is used to store data transferred from the hard disk.

[0156] Optionally, the i-th level first file layer is used to store data transferred in the (i+1)-th level first file layer, and the first-level first file layer is used to transfer data to the main memory.

[0157] This situation corresponds to some data reading implementation processes, where the data in the hard disk can be transferred to the extended memory so that the processor can read it, or the data to be read can be transferred layer by layer from the extended memory to the main memory so that the processor can read it.

[0158] In the above different situations, by using different values ​​of N, it is possible to effectively store data in the extended memory with an appropriate number of file levels to adapt to data access requirements in different scenarios.

[0159] The hard disk supports the creation of a second file layer of level M, combined with Figure 2 shown.

[0160] The hard disk transfers data between the extended memory and the first-level second file layer in the M-level second file layer included in the hard disk.

[0161] The first level second file layer in the M level second file layers is used to store data transferred from the N level first file layer, and the j level second file layer is used to store data transferred from the j-1 level second file layer;

[0162] Or, optionally, the j-1th level second file layer is used to store data transferred from the jth level second file layer, and the data in the first level second file layer is transferred to the Nth level first file layer to be read by the processor through the extended memory.

[0163] The M-level file layer in the hard disk is multi-layered, and is used for hierarchical storage of data that needs to be stored for a long time and for transferring relevant target data to the extended memory layer by layer when the data stored in the hard disk needs to be read, so that the target data can be directly read from the extended memory by the processor, or the target data is transferred to the extended memory and then further transferred to the main memory through the extended memory so that it can be directly read from the main memory by the processor.

[0164] The M-level second file layer in the hard disk and the N-level first file layer in the extended memory may be file layers managed separately. In this case, there is not necessarily a certain sequential relationship between the level numbers of the N-level first file layer and the M-level second file layer.

[0165] Alternatively, the two are file layers managed in a unified manner. In this case, the file layer numbers of the second file layer of level M and the first file layer of level N may be in a sequential relationship.

[0166] In an optional implementation, when data is transferred between the main memory, extended memory and hard disk, the data in the main memory can be gradually transferred downward to the various file layers in the extended memory for storage based on the data heat in the database, and the data in the extended memory can be gradually transferred downward to the various file layers in the hard disk for storage, so that hot data and part of the warm data are stored in the main memory (including the main memory and extended memory), and part of the warm data and cold data are stored in the hard disk.

[0167] Optionally, the N-level first file layer and the M-level second file layer may include multiple SST files for storing data.

[0168] Each file layer contains a series of SST (Sorted String Table) files, and the redundant data between layers will be dynamically compressed and merged.

[0169] During data storage, the data to be transferred in the current file layer can be merged and compressed and transferred to the SST file of the next file layer. Conversely, during data reading, the target data to be read in the SST file of the current file layer can be decompressed and restored and stored in the previous file layer.

[0170] In some optional implementation processes, when data is transferred between multiple levels of first file layers in an extended memory, or when data is transferred between multiple levels of second file layers in a hard disk, or when data is transferred between the Nth level first file layer in an extended memory and the first level second file layer in a hard disk, the data to be transferred in the file layers of one level can be merged and compressed and transferred to the file layers of another level.

[0171] Hot data is usually in the lower layers (file layers with small number values, such as the first-level file layer), and cold data is gradually transferred to the higher layers (file layers with large number values, such as the Nth-level file layer). Colder data is gradually transferred downward from the lower layers to the higher layers. When writing data, the data is first written to the memory, then to the extended memory, and finally to the hard disk, accompanied by the layer-by-layer merging, compression and transfer of the data; when reading data, it is necessary to start from the file layer where the target data is located, decompress the target data layer by layer and transfer it to the memory for reading.

[0172] Data merging can be to filter the data content in multiple files stored in the same level file layer, filter out the data content with the same popularity and merge them into one file, or to de-duplicate the same or similar data stored repeatedly in the file layer (for example, the same Key & Value data may exist between different SST files in the same level file layer. During the merging, compression or decompression process, the newer version of the data is selected to remove the old version of the data to achieve data merging), or include both processing methods, or other data merging processing methods. The merging method is not specifically limited here.

[0173] Data transfer specifically requires that the original data be stored in a new storage area while being deleted from the original storage area.

[0174] Optionally, data transfer between memories can be operated using DSA (Data Streaming Accelerator). DSA is a high-performance data copy and conversion accelerator, mainly used in scenarios that require high-performance data copy and conversion, supports high-performance data movement functions, can improve the efficiency of data movement and conversion operations, and release CPU cycle resources for higher-level functions, improving overall system performance.

[0175] The process can include the transfer of data within the main memory, the transfer of data between the main memory and the extended memory, and the transfer of data between various storage levels in the extended memory. The performance of data transfer using DSA is 3 to 5 times higher than that using CPU or hard disk controller transfer.

[0176] Optionally, combined Figure 2 As shown, the data transfer between the M second file layers in the hard disk uses a hard disk controller, that is, the controller shown in the figure.

[0177] This can improve data transfer performance, offload CPU computing power, further improve data reading and writing performance, and improve data processing performance.

[0178] In some optional implementations, the data type stored in the database is key-value pair data. Optionally, in general, the number of bytes occupied by the key value in the key-value pair data is 128 to 1024 bytes, and the number of bytes occupied by the value value is greater than 2048 bytes.

[0179] To achieve the separation of key and value storage in key-value pairs, combined with Figure 3 As shown, the extended memory includes a first group of N-level first file layers and a second group of N-level first file layers.

[0180] The first-level first file layer in the first group of N-level first file layers is used to store the first target key transferred from the main memory; the first-level first file layer in the second group of N-level first file layers is used to store the first target value corresponding to the first target key transferred from the main memory.

[0181] Optionally, the variable memory table in the main memory includes a first variable memory table for storing target keys and a second variable memory table for storing target values ​​corresponding to the target keys; the first group of N-level first file layers in the extended memory corresponds to the first variable memory table, and the second group of N-level first file layers corresponds to the second variable memory table.

[0182] Optionally, the hard disk may include a first group of M second file layers corresponding to the first group of N-level first file layers, and a second group of M second file layers corresponding to the second group of N-level first file layers.

[0183] By means of such a setting, two groups of N first file layers are divided in the extended memory, and two groups of M second file layers are divided in the hard disk. Different variable memory tables can also be set in the main memory.

[0184] Among them, the first-level first file layer in the first group of N-level first file layers is used to store the first target key transferred from the main memory; the first-level first file layer in the second group of N-level first file layers is used to store the first target value corresponding to the first target key transferred from the main memory.

[0185] The first-level second file layer in the first group of M-level second file layers is used to store the second target key transferred from the N-th-level first file layer in the first group of N-level first file layers in the extended memory; the first-level second file layer in the second group of M-level second file layers is used to store the second target value corresponding to the second target key transferred from the N-th-level first file layer in the second group of N-level first file layers in the extended memory.

[0186] It realizes the separate storage of keys and values ​​in key-value pairs, reduces the metadata required for storage, improves the matching efficiency when using key values ​​for data search, and improves the efficiency of data transfer.

[0187] In some optional embodiments, in combination Figure 2 , Figure 3 As shown, the extended memory also supports the creation of a write-ahead log (WAL Log), a manifest file (Manifest table) for storing metadata, etc.

[0188] Manifest files are usually used to record and manage metadata such as the structure, content, configuration, and permissions of a database, including database version, column family information, SST file list, option configuration, etc. These metadata are used to implement functions such as the correct operation of the database, data management, and user access.

[0189] In the write-ahead log mechanism, all write operations first write data to the write-ahead log, and then perform the write operation, which is used to ensure that in the event of a system crash or other failure, the data can be restored to a consistent state based on the write-ahead log. In this way, even if a failure occurs during the data writing process, the log file can be used to recover the unfinished write operation.

[0190] In the above data storage architecture, combined with Figure 2 As shown, the optional data writing process includes:

[0191] The client writes data to the main memory by calling the database API, and writes the data to the write-ahead log in the hard disk. You can apply for a memory segment in the main memory space to create a new mutable memory table to store the newly written data. When the memory segment is full, it will be converted to an immutable memory table. When the total number of immutable memory tables in the main memory exceeds the set threshold, the data in the immutable memory table will be written to the first-level first file layer of the extended memory, and an SST file will be created in the first-level first file layer of the extended memory to store the newly written data in the immutable memory table.

[0192] The data transferred from the main memory to the extended content can be stored in layers according to different file layers in the extended memory, and the data transferred from the extended memory to the hard disk can be stored in layers according to different file layers in the hard disk.

[0193] A file layer can contain multiple SST files. Since the data in the SST files may come from different immutable memory tables, there may be redundant data between multiple SST files in a file layer.

[0194] For redundant data existing between multiple SST files in the N-level first file layer in the extended memory or the M-level second file layer in the hard disk, you can choose to merge and compress the data, and create a new SST file in the next-level file layer to store the merged and compressed data set. For redundant data between multi-level file layers, data merger and compression are performed layer by layer, and a new SST file is created in the next-level file layer accordingly to store the merged and compressed data set, thereby realizing orderly storage of data in the main memory, extended memory, and hard disk.

[0195] In this process, the CPU executes the processing logic of data merging, compression, and writing data into the new SST table, and updates the corresponding metadata in the inventory file.

[0196] Whenever the database storage structure or organization changes, such as creating a new SST file or deleting an old SST file, the list file will be updated. The list file can be stored in the extended memory.

[0197] In the above data storage architecture, combined with Figure 2 As shown, the optional data reading process includes:

[0198] When the client calls the DB API to read data, it first searches for the requested key from the mutable memory table in the main memory. If the key is not found in the mutable memory table, it searches for the requested key from the immutable memory table. If the key is not found in either table, that is, the key is not found in the main memory, it searches for the index downward according to the hierarchical structure of the file layer in the extended memory.

[0199] First, starting from the first level first file layer, use the index to determine whether the multiple SST files in each layer contain the requested key. If not, continue to search in the next level first file layer until the key is found in the SST file or until the last level first file layer, that is, the Nth level first file layer, is found. If the requested key is not found in any level of the first file layer in the extended memory, the index will continue to search downward according to the hierarchical structure of the file layer in the hard disk.

[0200] First, start from the first-level second file layer, and use the index to determine whether the multiple SST files in each layer contain the requested key. If not, continue to search in the next level second file layer until the key is found in the SST file or until the last level second file layer, that is, the Mth level second file layer, is found.

[0201] The above process of searching for the requested key can use Bloom filters to quickly exclude SST files that do not contain the requested key. For SST files that contain the requested key, the index of the SST file is used to quickly locate the storage location where the requested key may exist.

[0202] When the requested key is found in the main memory, the value corresponding to the requested key is directly read from the main memory and fed back to the user end. When the requested key is found in the extended memory, the value corresponding to the requested key is directly read from the extended memory and fed back to the user end; or the value corresponding to the requested key is transferred layer by layer to the variable memory table of the main memory for reading and fed back to the user end. When the requested key is found in the hard disk, the value corresponding to the requested key is transferred layer by layer to the extended memory for reading and fed back to the user end; or the value corresponding to the requested key is transferred layer by layer to the extended memory and then transferred layer by layer to the variable memory table of the main memory for reading and fed back to the user end.

[0203] In the above computing system, due to the addition of extended memory, the process of transferring data from the main memory to the hard disk changes. The data is no longer written directly to the hard disk, but needs to be transferred down to the extended memory first, and then the extended memory further transfers the data stored therein down to the hard disk.

[0204] In an embodiment of the present application, an extended memory is added between the main memory and the hard disk as a transitional storage space between the main memory and the hard disk, thereby reducing data operations on the hard disk, reducing the interference of low hard disk I / O performance on data access efficiency, reducing data processing time, and improving data access response speed.

[0205] In some implementations, the present application provides a data processing method, which is applied to a computing system, wherein the computing system includes a main memory, an extended memory, and a hard disk; the extended memory includes an N-level first file layer, where N≥1; and the hard disk includes an M-level second file layer, where M>1. When executed, the method can be implemented by a processor in the computing system.

[0206] In one implementation process, corresponding to the data storage process, the data processing method includes:

[0207] When a first preset condition is met, the first target data in the main memory is written into the first-level first file layer of the extended memory.

[0208] Wherein, in the case where N>1, the i-th level first file layer in the N-level first file layers is used to store the data transferred in the i-1-th level first file layer, 1 <i≤N。

[0209] When the second preset condition is met, the second target data stored in the Nth level first file layer of the extended memory is written into the first level second file layer in the hard disk.

[0210] The j-th second file layer in the M-th second file layer is used to store the data transferred in the j-1-th second file layer. <j≤M。

[0211] The first target data may be part of the data or all of the data stored in the main memory. Optionally, the first target data may be part of the data or all of the data stored in the variable memory table and the immutable memory table in the main memory. Alternatively, it may be part of the data or all of the data stored in the immutable memory table, where the part of the data may be, for example, data merged from data with the same heat and relatively low heat in multiple immutable memory tables, or data determined in other ways.

[0212] The second target data may be part of or all of the data stored in the Nth level first file layer of the extended memory. The part of the data may be, for example, data formed by merging data with the same or relatively low popularity in multiple SST files in the Nth level first file layer, or data determined in other ways.

[0213] In this process, extended memory is introduced between the main memory and the hard disk, and data can be transferred between the first-level file layer in the N-level file layers and the main memory, and between the N-th-level file layer and the hard disk. With the help of the extended memory of the storage layer with data files, data caching is achieved, reducing the access to the hard disk for data operations, reducing the interference of low hard disk I / O performance on data access efficiency, reducing data processing time, and improving data access response speed in the computing system.

[0214] In some implementations, when the remaining storage space of the main memory is less than a first threshold, it is determined that the first preset condition is satisfied. Alternatively, when a first specific user operation is detected, it is determined that the first preset condition is satisfied.

[0215] The main memory, extended memory and hard disk each have a set storage space size.

[0216] When storing data in the main memory, extended memory and hard disk, the total storage space, used space and remaining available space of the main memory, extended memory and hard disk can be obtained to determine whether the remaining storage space in the corresponding storage medium is sufficient.

[0217] The first threshold may be a fixed value for determining whether the remaining storage space is sufficient. When the remaining storage space of the main memory is less than the fixed value, it is determined that the first preset condition is satisfied.

[0218] Alternatively, the first threshold may be a dynamic value for determining whether the remaining storage space is sufficient based on the current amount of data to be stored.

[0219] The timing of whether to transfer data in the main memory to the extended memory is determined by combining the remaining storage space of the main memory.

[0220] In one embodiment, if the remaining storage space of the variable memory table in the main memory is less than the amount of data to be stored, the variable memory table is converted into an immutable memory table, and when the remaining storage space in the main memory is insufficient to create a new variable memory table, it is determined that the remaining storage space of the main memory is less than the first threshold, and the first preset condition is determined to be satisfied. The timing of whether to transfer the data in the main memory to the extended memory is determined in combination with the amount of data in different memory tables in the main memory.

[0221] The above-mentioned first specific user operation is, for example, a user clicking a specific command button to trigger the operation. In some scenarios, such as when it is necessary to ensure that the main memory has sufficient remaining storage space, the user can trigger the migration of data in the main memory to the extended memory through a specific operation, and write the first target data in the main memory to the first-level first file layer of the extended memory.

[0222] Combined with user operation conditions, it is possible to determine when to transfer data in the main memory to the extended memory.

[0223] In some implementations, when the remaining storage space of the Nth level first file layer of the extended memory is less than a fourth threshold, it is determined that the second preset condition is met; or, when a second specific user operation is detected, it is determined that the second preset condition is met.

[0224] The extended memory has a set storage space size. When initializing the N-level first file layers in the extended memory, the storage space size of each level of the first file layer can be set. When storing data in a certain level of the first file layer of the main memory, the storage space size, used amount and remaining available amount of the first level of the first file layer can be obtained to determine whether the remaining storage space of the first level of the first file layer is sufficient.

[0225] The fourth threshold may be a fixed value for determining whether the remaining storage space of the Nth level first file layer of the extended memory is sufficient. If the remaining storage space of the Nth level first file layer of the extended memory is less than the fixed value, it is determined that the second preset condition is met.

[0226] Alternatively, the fourth threshold may be a dynamic value for determining whether the remaining storage space is sufficient based on the amount of data in the Nth level first file layer to be transferred to the extended memory. In one embodiment, if the remaining storage space of the Nth level first file layer of the extended memory is less than the amount of data to be transferred, it is determined that the remaining storage space of the Nth level first file layer of the extended memory is less than the fourth threshold, and it can be determined that the second preset condition is met.

[0227] The second specific user operation is, for example, a user clicking a specific command button to trigger the operation. In some scenarios, such as when it is necessary to ensure that the extended memory has enough remaining storage space, the user can trigger the migration of data in the extended memory from the first file layer of the Nth level to the hard disk through a specific operation.

[0228] In some embodiments, when N is greater than 1, the data processing method includes:

[0229] When the third preset condition is met, the third target data stored in the i-1th level first file layer in the Nth level first file layer of the extended memory is stored in the i-th level first file layer; 1 <i≤N。

[0230] The third preset condition may be that the user clicks on a specific command button to trigger the operation. In some scenarios, for example, when it is necessary to ensure that there is enough remaining storage space in the first file layer of the lower level of the extended memory, the user can trigger the migration of data in the extended memory from the first file layer of the i-1th level to the first file layer of the i-th level through a specific operation. Alternatively, the third preset condition may be that the remaining storage space in the first file layer of the i-1th level in the extended memory is less than a threshold value. This allows for the timing judgment of whether data transfer between different file layers in the extended memory is required.

[0231] Correspondingly, when the third preset condition is met, storing the third target data stored in the i-1th level first file layer in the N-level first file layer of the extended memory into the i-th level first file layer includes:

[0232] When the remaining storage space in the first file layer of the i-1th level in the extended memory is less than the second threshold, starting from the first file layer of the i-1th level in the extended memory, the third target data stored in the first file layer of the current level is written into the first file layer of the i-1th level; until the remaining storage space in the first file layer of the i-1th level in the extended memory is greater than the second threshold.

[0233] The second threshold may be a fixed value for determining whether the remaining storage space of the first file layer of the i-1th level of the extended memory is sufficient. If the remaining storage space of the first file layer of the i-1th level of the extended memory is less than the fixed value, it is determined that the third preset condition is met.

[0234] Alternatively, the second threshold value may be a dynamic value for determining whether the remaining storage space is sufficient based on the amount of data in the first file layer of the i-1th level to be transferred to the extended memory. In one embodiment, if the remaining storage space of the first file layer of the i-1th level of the extended memory is less than the amount of data to be transferred, it is determined that the remaining storage space of the first file layer of the i-1th level of the extended memory is less than the second threshold value, and it can be determined that the third preset condition is satisfied.

[0235] The third target data is part of or all of the data stored in the i-1th level first file layer of the extended memory.

[0236] This process ensures that the upper file layer has enough remaining storage space after transferring data to the lower file layer by transferring data between the various file layers, thereby ensuring effective storage of data in different file layers in the extended memory.

[0237] In an optional implementation, the hard disk includes M-level second file layers, the first-level second file layer in the M-level second file layers is used to store data transferred from the N-th-level first file layer, and M>1; the data processing method further includes:

[0238] When the fourth preset condition is met, the fourth target data stored in the j-1th level second file layer in the M-level second file layer of the hard disk is stored in the jth level second file layer; 1 <j≤M。

[0239] The fourth preset condition may be a user's click on a specific command button to trigger an operation. In some scenarios, for example, when it is necessary to ensure that there is enough remaining storage space in the second file layer of the lower level of the hard disk, the user may trigger the migration of data in the hard disk from the second file layer of the j-1th level to the second file layer of the jth level through a specific operation. Alternatively, the fourth preset condition may be that the remaining storage space in the first file layer of the j-1th level in the hard disk is less than a threshold value. This enables the timing judgment of whether data transfer between different file layers in the hard disk is required.

[0240] Correspondingly, when the fourth preset condition is met, storing the fourth target data stored in the j-1th level second file layer in the M-level second file layer of the hard disk into the jth level second file layer includes:

[0241] When the remaining storage space in the j-1th level second file layer in the hard disk is less than the fifth threshold value, starting from the j-1th level second file layer of the hard disk, the fourth target data stored in the current level second file layer is written into the j-th level second file layer; until the remaining storage space in the j-1th level second file layer in the hard disk is greater than the fifth threshold value.

[0242] The fifth threshold may be a fixed value for determining whether the remaining storage space of the j-1th level second file layer in the hard disk is sufficient. If the remaining storage space of the j-1th level second file layer in the hard disk is less than the fixed value, it is determined that the fourth preset condition is met.

[0243] Alternatively, the fifth threshold value may be a dynamic value for determining whether the remaining storage space is sufficient based on the amount of data in the j-1th level second file layer to be transferred to the hard disk. In one embodiment, if the remaining storage space of the j-1th level second file layer of the hard disk is less than the amount of data to be transferred, then it is determined that the remaining storage space of the j-1th level second file layer of the hard disk is less than the fifth threshold value, and it can be determined that the fourth preset condition is met.

[0244] The fourth target data is part of or all of the data stored in the j-1th level second file layer of the hard disk.

[0245] This process ensures that the upper file layer has enough remaining storage space after transferring data to the lower file layer by transferring data between the various file layers of the hard disk, thereby ensuring the effective storage of data in different file layers in the hard disk.

[0246] In an optional implementation, to ensure that the data in the database can be restored to a consistent state in the event of a system crash or other failure, a process for determining the remaining storage space of the write-ahead log is introduced during the data storage process.

[0247] Correspondingly, if the remaining storage space of the variable memory table in the main memory is greater than the amount of data to be stored, the data to be stored will be written to the variable memory table if the remaining storage space of the write-ahead log in the extended memory is greater than the amount of backup data of the data to be stored.

[0248] Optionally, when the remaining storage space of the write-ahead log in the extended memory is insufficient to write the backup data of the data to be stored, a new write-ahead log file with sufficient remaining storage space can be created in the extended memory, the backup data of the data to be stored can be written to the new write-ahead log, and the old write-ahead log file can be deleted.

[0249] In the above processing, even if a failure occurs during the data writing process, the unfinished write operation can be restored by using the pre-write log to ensure the effective writing of data in the database.

[0250] In some optional implementation processes, the data to be stored is key-value pair data, and the extended memory includes a first group of the N-level first file layers and a second group of the N-level first file layers.

[0251] On this basis, when the first preset condition is met, writing the first target data in the main memory into the first-level first file layer of the extended memory specifically includes:

[0252] When the first preset condition is met, the first target key in the main memory is written to the first-level first file layer in the first group of N-level first file layers; and / or, the first target value corresponding to the first target key in the main memory is written to the first-level first file layer in the second group of N-level first file layers.

[0253] Accordingly, when the second preset condition is met, writing the second target data stored in the Nth level first file layer of the extended memory into the first level second file layer of the hard disk specifically includes:

[0254] When the second preset condition is met, the second target key stored in the N-th level first file layer in the first group of the N-level first file layers in the extended memory is written to the hard disk, and / or the second target value corresponding to the second target key stored in the N-th level first file layer in the second group of the N-level first file layers in the extended memory is written to the hard disk.

[0255] In this case, the key and value in the key-value pair data are stored separately in the extended memory and the hard disk.

[0256] For keys, there is a separate set of N-level first file layers in the extended memory to transfer keys, so as to realize the storage and reading of key data in the extended memory, hard disk and main memory. Optionally, there may be a separate set of M-level second file layers in the hard disk to transfer keys. Optionally, there may be a separate variable memory table and a corresponding immutable memory table in the main memory to realize the storage and reading of keys in the main memory. On this basis, the specific implementation process of data transfer, data storage and reading of keys in the extended memory, main memory and hard disk can refer to the description content of other parts of the embodiments of the present application, which will not be repeated here.

[0257] Similarly, for values, there is another separate set of N-level first file layers in the extended memory to transfer keys, so as to realize the storage and reading of value data in the extended memory, hard disk and main memory. Optionally, there may be a separate set of M-level second file layers in the hard disk to transfer values. Optionally, there may be a separate variable memory table and a corresponding immutable memory table in the main memory to realize the storage and reading of values ​​in the main memory. On this basis, the specific implementation process of data transfer, data storage and reading of values ​​in the extended memory, main memory and hard disk can refer to the description content of other parts of the embodiments of the present application, which will not be repeated here.

[0258] The above process stores keys and values ​​separately, reduces the metadata required for storage, improves the matching efficiency when using keys to search for data, and improves the efficiency of data transfer.

[0259] For the above data storage process, an exemplary implementation process is described for the case where the extended memory contains multiple first file layers and the hard disk contains multiple second file layers. Figure 4 As shown, the specific steps include:

[0260] Step 401, based on the data to be stored, determine whether the remaining storage space in the main memory is sufficient.

[0261] Whether the remaining storage space in the main memory is sufficient can be determined by determining whether the remaining storage space of the variable memory table in the main memory is sufficient to write the data to be stored. If the remaining storage space of the variable memory table is greater than the amount of data to be stored, it is considered sufficient to write the data to be stored; or it can be determined whether the remaining storage space in the main memory is sufficient to create a new variable memory table to write the data to be stored in the new variable memory table.

[0262] In some optional implementations, the process of determining whether the remaining storage space in the main memory is sufficient based on the data to be stored includes:

[0263] Determine whether the remaining storage space of the variable memory table in the main memory is sufficient to write the data to be stored;

[0264] If the remaining storage space of the variable memory table is sufficient to write the data to be stored, it is determined that the remaining storage space in the main memory is sufficient;

[0265] If the remaining storage space of the mutable memory table is insufficient to write the data to be stored, the mutable memory table is converted into an immutable memory table, and when the remaining storage space in the main memory is insufficient to create a new mutable memory table, it is determined that the remaining storage space in the main memory is insufficient;

[0266] If the remaining storage space of the variable memory table is insufficient to write the data to be stored, but the remaining storage space in the main memory is sufficient to create a new variable memory table, it is determined that the remaining storage space in the main memory is sufficient.

[0267] Here, the newly created variable memory table must be able to meet the condition that the storage space is sufficient to write the data to be stored.

[0268] The above judgment process, by judging whether the remaining storage space of the variable memory table in the main memory is sufficient to write the data to be stored, and further combining it with judging whether the remaining storage space in the main memory is sufficient to create a new variable memory table, implements effective judgment on different situations of whether the remaining storage space in the main memory is sufficient, thereby ensuring effective storage judgment of data in the database.

[0269] Further, when the remaining storage space in the main memory is insufficient, step 402 is executed. When the remaining storage space in the main memory is sufficient, step 405 is executed.

[0270] Step 402, determining whether the remaining storage space in the N-level first file layer in the extended memory is sufficient.

[0271] The judgment on whether the remaining storage space in the N-level first file layers in the extended memory is sufficient can be to judge whether the remaining storage space of the first-level first file layer in the N-level first file layers is sufficient to transfer the storage data stored in the immutable memory table in the main memory, or to judge in sequence among the N-level first file layers whether the remaining storage space of the i-th level first file layer in the N-level first file layers is sufficient to transfer the storage data stored in the i-1-th level first file layer.

[0272] In some optional implementations, the process of determining whether the remaining storage space in the N-level first file layer in the extended memory is sufficient includes:

[0273] Determine whether the remaining storage space of the first level first file layer in the N level first file layer is sufficient to transfer the storage data stored in the immutable memory table in the main memory;

[0274] If it is judged to be sufficient, then the remaining storage space in the first file layer of level N in the extended memory is sufficient;

[0275] If it is determined to be insufficient, then determining in sequence whether the remaining storage space of the i-th level first file layer in the N-level first file layers is sufficient to transfer the storage data stored in the i-1-th level first file layer;

[0276] If it is judged to be sufficient, then the remaining storage space in the first file layer of level N in the extended memory is sufficient;

[0277] If it is judged to be insufficient, then the remaining storage space in the first file layer of level N in the extended memory is insufficient.

[0278] Among them, in the process of successively judging whether the remaining storage space of the i-th level first file layer in the N-level first file layers is sufficient to transfer the storage data stored in the i-1-th level first file layer, if it is detected in the sequential judgment that the remaining storage space of any level first file layer is sufficient to transfer the storage data stored in the previous level first file layer, then it can be considered that the remaining storage space in the N-level first file layer in the extended memory is sufficient, and there is no need to continue to judge the remaining storage space of the subsequent storage layers.

[0279] While in the process of successively determining whether the remaining storage space of the i-th first file layer in the N-level first file layers is sufficient to transfer the storage data stored in the i-1-th first file layer, if it is determined that the remaining storage space from the second-level first file layer to the N-level first file layer is insufficient to transfer the storage data stored in the previous-level first file layer, then it can be considered that the remaining storage space in the N-th first file layer in the extended memory is insufficient.

[0280] This process, by judging whether the remaining storage space of the first-level first file layer in the N-level first file layers in the extended memory is sufficient to transfer the storage data in the immutable memory table in the main memory, and further combining it with judging whether the remaining storage space of the i-th level first file layer in the N-level first file layers is sufficient to transfer the storage data in the i-1-th level first file layer, implements effective judgment on whether the remaining storage space of the extended memory is sufficient by judging different situations on whether the remaining storage space of the N-level first file layers is sufficient, thereby ensuring effective storage judgment of data in the database.

[0281] Further, if the remaining storage space in the N-level first file layer in the extended memory is sufficient, step 403 is executed. If the remaining storage space in the N-level first file layer in the extended memory is insufficient, step 404 is executed.

[0282] Step 403, transfer the target data stored in the main memory to the first file layer of level N.

[0283] In some optional implementations, the process of transferring the target data stored in the main memory to the first file layer of level N includes:

[0284] If the remaining storage space of the first-level first file layer in the N-level first file layer is sufficient to transfer the data stored in the immutable memory table in the main memory, the data stored in the immutable memory table in the main memory is transferred to the first-level first file layer in the N-level first file layer.

[0285] In an optional implementation process, when the remaining storage space of the first-level first file layer is sufficient, the background thread can be triggered to start and the immutable memory table in the main memory can be flushed to the first-level first file layer in the extended memory through DSA.

[0286] This processing process transfers the data stored in the immutable memory table in the main memory to the first-level first file layer in the N-level first file layer, freeing up available data storage space in the main memory so that the remaining storage space in the main memory can meet the data writing requirements of the data to be stored, ensuring the effective storage of data in the database.

[0287] Alternatively, in some optional implementations, the process of transferring the target data stored in the main memory to the first file layer of level N includes:

[0288] If the remaining storage space of the first-level first file layer in the N-level first file layers is insufficient to transfer the storage data in the immutable memory table in the main memory, and there is a target-level first file layer after the first-level first file layer in the N-level first file layers with remaining storage space sufficient to transfer the storage data in the previous-level first file layer, then the storage data in the previous-level first file layer is stored in the target-level first file layer.

[0289] Until the remaining storage space of the first-level first file layer in the N-level first file layer is sufficient to transfer the stored data in the immutable memory table in the main memory, the stored data in the immutable memory table in the main memory is transferred to the first-level first file layer in the N-level first file layer.

[0290] In an optional implementation process, when the remaining storage space of the first-level first file layer is insufficient to transfer the storage data stored in the immutable memory table in the main memory, it is detected whether the remaining storage space of the second-level first file layer in the extended memory is sufficient to transfer the storage data stored in the first-level first file layer. If sufficient, a background thread is started, a new SST file is created in the second-level first file layer, and the storage data in the first-level first file layer is flushed to the second-level first file layer through DSA and stored in the newly created SST file therein, and it is determined whether the remaining storage space of the file layer of the previous level of the current first file layer is sufficient, specifically whether it is sufficient to transfer the storage data stored in the immutable memory table in the main memory, and the above-mentioned judgment processing process is repeated based on the judgment result.

[0291] If it is determined that the remaining storage space of the second-level first file layer is insufficient to transfer the storage data stored in the first-level first file layer, then it is determined whether the remaining storage space of the third-level first file layer is sufficient to transfer the storage data stored in the second-level first file layer. If sufficient, a background thread is started to create a new SST file in the third-level first file layer, and the storage data in the second-level first file layer is flushed to the third-level first file layer through DSA and stored in the newly created SST file. It is also determined whether the remaining storage space of the upper-level file layer (i.e., the second-level first file layer) of the current first file layer (i.e., the third-level first file layer) is sufficient, specifically whether it is sufficient to store the storage data in the upper-level file layer. The above-mentioned judgment process is repeated based on the judgment result.

[0292] If it is determined that the remaining storage space of the third-level first file layer is insufficient to transfer the storage data stored in the second-level first file layer, then it is determined whether the remaining storage space of the fourth-level first file layer is sufficient to transfer the storage data stored in the third-level first file layer. If sufficient, a background thread is started to create a new SST file in the fourth-level first file layer, and the storage data in the third-level first file layer is flushed to the fourth-level first file layer through DSA and stored in the newly created SST file. It is also determined whether the remaining storage space of the upper-level file layer (i.e., the third-level first file layer) of the current first file layer (i.e., the fourth-level first file layer) is sufficient, specifically whether it is sufficient to store the storage data in the upper-level file layer. The above-mentioned judgment process is repeated based on the judgment result.

[0293] According to the sequential relationship between the N first file layers, and so on, when it is determined that the remaining storage space of the N-1th first file layer is insufficient to transfer the compressed data stored in the N-2th first file layer, then it is determined whether the remaining storage space of the Nth first file layer is sufficient to transfer the stored data in the N-1th first file layer. If sufficient, a background thread is started to create a new SST file in the Nth first file layer, and the stored data in the N-1th first file layer is flushed to the Nth first file layer through DSA and stored in the newly created SST file therein, and it is determined whether the remaining storage space of the upper file layer (i.e., the N-1th first file layer) of the current first file layer (i.e., the Nth first file layer) is sufficient, specifically whether it is sufficient to store the stored data in the upper file layer, and the above-mentioned judgment process is repeated based on the judgment result.

[0294] Until it is determined that the remaining storage space of the first-level first file layer is sufficient to transfer the stored data in the immutable memory table in the main memory, the stored data in the immutable memory table in the main memory is transferred to the first-level first file layer through DSA.

[0295] The above processing process utilizes the data storage method of sinking and compressing the data between N-level first file layers layer by layer to free up available data storage space in the first-level file layer in the N-level first file layers, ensuring that the first-level file layer can transfer the storage data stored in the immutable memory table in the main memory, freeing up available data storage space in the main memory, so that the remaining storage space in the main memory can meet the data writing requirements of the data to be stored, and ensuring the effective storage of data in the database.

[0296] Furthermore, after step 403, the process returns to step 401 to determine whether the remaining storage space in the main memory is sufficient.

[0297] Step 404: If the remaining storage space of the second file layer of level M in the hard disk is sufficient, the target data stored in the extended memory is transferred and stored in the second file layer of level M.

[0298] In some optional implementations, the process of transferring the target data stored in the extended memory to the second file layer of the M level includes:

[0299] If the remaining storage space of the first-level second file layer in the M-level second file layer is sufficient to transfer the compressed data stored in the N-th-level first file layer in the N-level first file layer, then the stored data in the N-level first file layer will be transferred to the first-level second file layer in the M-level second file layer.

[0300] This processing process transfers the storage data in the first file layer of the Nth level in the extended memory to the first second file layer in the hard disk, freeing up available data storage space for the extended memory as much as possible, so that the main memory can be stored in the extended memory, and the remaining storage space of the main memory can meet the data writing requirements of the data to be stored, ensuring the effective storage of data in the database.

[0301] Alternatively, in some optional implementations, the process of transferring the target data stored in the extended memory to the second file layer of the M level includes:

[0302] If the remaining storage space of the first-level second file layer in the M-level second file layers is not sufficient to transfer the storage data in the N-level first file layer in the extended memory, and there is a target-level second file layer after the first-level second file layer in the M-level second file layers to which the remaining storage space is sufficient to transfer the storage data in the previous-level second file layer, then the storage data in the previous-level second file layer is stored in the target-level second file layer.

[0303] Until the remaining storage space of the first-level second file layer in the M-level second file layer is sufficient to transfer the storage data stored in the N-level first file layer, the storage data in the N-level first file layer is transferred to the first-level second file layer in the M-level second file layer.

[0304] In an optional implementation process, when the remaining storage space of the first-level second file layer in the hard disk is insufficient to transfer the storage data in the N-level first file layer in the extended memory, it is detected whether the remaining storage space of the second-level second file layer in the hard disk is sufficient to transfer the storage data in the first-level second file layer. If sufficient, a background thread is started to create a new SST file in the second-level second file layer, and the storage data in the first-level second file layer is flushed to the second-level second file layer through the controller and stored in the newly created SST file, and it is determined whether the remaining storage space of the upper-level file layer (i.e., the first-level second file layer) of the current second file layer (i.e., the second-level second file layer) is sufficient. Specifically, it is determined whether the first-level second file layer is sufficient to transfer the storage data in its upper-level file layer (i.e., the N-level first file layer), and the above-mentioned judgment process is repeated based on the judgment result.

[0305] If it is determined that the remaining storage space of the second-level second file layer is insufficient to transfer the storage data stored in the first-level second file layer, then it is determined whether the remaining storage space of the third-level second file layer is sufficient to transfer the storage data stored in the second-level second file layer. If sufficient, a background thread is started to create a new SST file in the third-level second file layer. The storage data in the second-level second file layer is flushed to the third-level second file layer through the controller and stored in the newly created SST file. It is also determined whether the remaining storage space of the upper-level file layer (i.e., the second-level second file layer) of the current second file layer (i.e., the third-level second file layer) is sufficient, specifically whether it is sufficient to store the storage data in the upper-level file layer. The above-mentioned judgment process is repeated based on the judgment result.

[0306] If it is determined that the remaining storage space of the third-level second file layer is insufficient to transfer the storage data stored in the second-level second file layer, then it is determined whether the remaining storage space of the fourth-level second file layer is sufficient to transfer the storage data stored in the third-level second file layer. If sufficient, a background thread is started to create a new SST file in the fourth-level second file layer, and the storage data in the third-level second file layer is flushed to the fourth-level second file layer through the controller and stored in the newly created SST file, and then it is determined whether the remaining storage space of the upper-level file layer (i.e., the third-level second file layer) of the current second file layer (i.e., the fourth-level second file layer) is sufficient, specifically whether it is sufficient to store the storage data in the upper-level file layer, and the above-mentioned judgment process is repeated based on the judgment result.

[0307] According to the sequential relationship between the M-level second file layers, and so on, when it is determined that the remaining storage space of the M-1-th level second file layer is insufficient to transfer the storage data stored in the M-2-th level second file layer, it is determined whether the remaining storage space of the last storage layer in the M-level second file layer, i.e., the M-th level second file layer, is sufficient to transfer the storage data stored in the M-1-th level second file layer. If sufficient, a background thread is started to create a new SST file in the M-th level second file layer. The storage data in the M-1-th level second file layer is flushed to the M-th level second file layer through the controller and stored in the newly created SST file. It is also determined whether the remaining storage space of the upper file layer (i.e., the M-1-th level second file layer) of the current second file layer (i.e., the M-th level second file layer) is sufficient, specifically, whether the upper file layer is sufficient to store the storage data in its upper file layer. The above-mentioned determination process is repeated based on the determination result.

[0308] Until it is determined that the remaining storage space of the first-level second file layer is sufficient to transfer the storage data stored in the N-level first file layer, the storage data in the N-level first file layer is transferred to the first-level second file layer.

[0309] The above processing process utilizes the data storage method of sinking and compressing the data between the M-level second file layers layer by layer to free up available data storage space for the first-level second file layer in the M-level second file layer, ensuring that the first-level second file layer can transfer the storage data in the last-level second file layer in the extended memory, so that the extended memory can free up available data storage space, and further ensure that the storage data in the extended memory can be transferred to the main memory, freeing up available data storage space in the main memory, so that the remaining storage space in the main memory can meet the data writing requirements of the data to be stored, thereby ensuring the effective storage of data in the database.

[0310] Furthermore, after step 404, the process returns to step 402 to determine whether the remaining storage space in the N-level first file layer in the extended memory is sufficient.

[0311] The above processing is repeated until it is determined that the remaining storage space in the main memory is sufficient, and step 405 is executed. Alternatively, until it is determined that the remaining storage space in the second file layer of level M is insufficient, step 406 is executed.

[0312] Step 405, storing the data to be stored in the main memory.

[0313] When there is sufficient remaining storage space in the main memory, the data to be stored can be stored in the main memory.

[0314] Step 406: If the remaining storage space of the second file layer of level M is insufficient, a response message is returned.

[0315] The response message may be an error message, or a prompt message indicating that storage space in the database is insufficient.

[0316] The above processing process utilizes the data storage method of sinking and compressing data between the main memory, extended memory and hard disk. By judging whether the remaining storage space in different storage spaces is sufficient, combined with the sinking and compressing transfer of data in the main memory to the extended memory, and combined with the sinking and compressing transfer of data in the extended memory to the hard disk, it ensures that available data storage space is freed up in the main memory, so that the remaining storage space in the main memory can meet the data writing requirements of the data to be stored, thereby ensuring the effective storage of data in the database.

[0317] The embodiment of the present application also provides an initialization processing process for the N-level first file layer in the extended memory.

[0318] In some optional implementations, the data processing method further includes:

[0319] Formatting the extended memory using a memory file system;

[0320] Configure initialization parameters, which at least include the file directory of the extended memory, the storage directory of the write-ahead log in the extended memory, the size of the first file layer of level N in the extended memory, and the number of file layers.

[0321] Among them, using the memory file system to format the extended memory can enable the data access program to access the data stored in the extended memory in the form of files, improve data access performance, and reduce the modification and adaptation of the database system caused by adding extended memory.

[0322] Optionally, the memory file system is, for example, FAMFS (Fabric-Attached Memory File System).

[0323] FAMFS is a file system designed for structured memory. This memory file system is a DAX (Direct Access) file system that allows users to access the memory of DAX devices in the form of files without page cache overhead.

[0324] The memory file system can be installed in storage media such as hard disk and extended memory to realize hierarchical initialization and management of storage space in extended memory.

[0325] Optionally, when configuring initialization parameters, the parameter contents include but are not limited to:

[0326] The file directory of the extended memory, the file directory of the hard disk, the storage directory of the write-ahead log in the extended memory, the size of the write cache, the file size of the mutable memory table, the total number of files in the mutable memory table and the immutable memory table, the file flushing threshold of the immutable memory table, the capacity of the first-level first file layer in the extended memory, the value of the number of layers N, the capacity magnification factor of each layer between the N-level first file layers, and the number of background threads.

[0327] If it is detected that the remaining storage space in the extended memory meets the required capacity of the N-level first file layer, then based on the initialization parameters, the N-level first file layer is initialized in the extended memory, that is, Level 0 to LevelN-1 layers are initialized, and the pre-write log is initialized for backing up the data to be stored, creating management files, etc.

[0328] The initialization of the second file layer of the M level of the hard disk needs to be implemented by using other file systems, such as NTFS (New Technology File System).

[0329] The other file system can be installed in a storage medium such as a hard disk and an extended memory to implement hierarchical initialization and management of the storage space in the hard disk.

[0330] The configured initialization parameters at least include the file directory of the hard disk, the size of the second file layer of level M in the hard disk, and the number of file layers.

[0331] In an optional implementation, when configuring initialization parameters, the parameter contents include but are not limited to:

[0332] The capacity of the first-level second file layer in the hard disk, the value of the number of layers M, the capacity magnification factor of each layer between the M-level second file layers, etc.

[0333] Based on the initialization parameters, M-level second file layers, namely, the first-level second file layer to the M-th-level second file layer, are initialized in the hard disk, and management files and the like are created.

[0334] In this way, the data access program can access the data stored in the extended memory in the form of files, thereby improving data access performance.

[0335] Optionally, initialization of the extended memory and initialization of the hard disk may be implemented in the same initialization process, or initialization of the two may be implemented separately in different initialization processes to meet initialization requirements in different situations.

[0336] Optionally, the initialization process also includes initializing the read and write cache areas of the database, the file flushing strategy of the immutable memory table in the main memory, the processing strategy of the SST files in each file layer (such as merging and compression), etc.

[0337] The above process realizes the initialization processing of the database system based on the main memory, the extended memory and the hard disk storage device, and realizes the effective initialization construction of the N-level first file layer in the newly added extended memory.

[0338] A data processing method proposed in an embodiment of the present application also involves a data reading process, which specifically includes:

[0339] When a fifth preset condition is met, fifth target data matching the data to be read is read from the main memory; or, when a sixth preset condition is met, the fifth target data is read from the extended memory.

[0340] In this way, in the improved storage architecture, the processor can read the target data from the main memory in the conventional way, minimizing the modification of the original system under the data reading function after the newly added extended memory, which is easy to implement; or the processor can also directly read the target data from the newly added extended memory, which is convenient for direct and effective acquisition of data in the memory and improves the data reading response speed. Different data reading implementation methods are provided to meet the data reading implementation requirements in different scenarios.

[0341] Optionally, the fifth preset condition may be that target data matching the data to be read is stored in the main memory. Alternatively, target data matching the data to be read is stored in the extended memory, and the target data is written from the first level first file layer of the extended memory to the main memory. Alternatively, target data matching the data to be read is stored in the hard disk, and the target data is written from the hard disk to the Nth level first file layer of the extended memory, and further the target data is written to the main memory via the first level first file layer of the extended memory.

[0342] That is, when the fifth preset condition is met, fifth target data matching the data to be read is read from the main memory, specifically including:

[0343] In a case where the fifth target data matching the data to be read is stored in the main memory, the fifth target data is read from the main memory; or, in a case where the fifth target data matching the data to be read is stored in the extended memory, the fifth target data is written from the first-level first file layer of the extended memory to the main memory, and the fifth target data is read from the main memory; or, in a case where the fifth target data matching the data to be read is stored in the hard disk, the fifth target data is written from the hard disk to the Nth-level first file layer of the extended memory, and the fifth target data is written from the first-level first file layer of the extended memory to the main memory, and the fifth target data is read from the main memory.

[0344] Wherein, when N is 1, the Nth level first file layer of the extended memory is the same level as the first level first file layer. When N is greater than 1, the Nth level first file layer of the extended memory is not the same level as the first level first file layer, and the i-1th level first file layer of the extended memory is used to store the data transferred in the ith level first file layer.

[0345] In the above implementation process, corresponding data reading processing is performed according to the different storage situations of the target data in the main memory, extended memory and hard disk, and the storage level of the data file set in the extended memory is used to transfer the target data between the hard disk, extended memory and main memory, so as to realize the final reading of the target data from the main memory.

[0346] Optionally, in the above processing, writing the fifth target data from the first-level first file layer of the extended memory into the main memory includes:

[0347] determining a first file layer of a target level where the fifth target data is located in the extended memory;

[0348] When N is greater than 1 and the target-level first file layer is not the first-level first file layer, starting from the target-level first file layer of the extended memory, the fifth target data stored in the current-level first file layer is written to the adjacent previous-level first file layer until the fifth target data is written to the first-level first file layer, and the fifth target data is written from the first-level first file layer of the extended memory to the main memory.

[0349] The target first file layer here may be the Nth first file layer, or any first file layer from the second first file layer to the N-1th first file layer.

[0350] If the fifth target data is stored in the extended memory, and the target-level first file layer where the fifth target data is located in the extended memory is the N-th-level first file layer, then when N is 1, the N-th-level first file layer is also the first-level first file layer, and the fifth target data can be directly written into the main memory through this file layer.

[0351] If the fifth target data is stored in the extended memory, then when N is greater than 1 and the target-level first file layer is any first file layer from the Nth-level first file layer to the second-level first file layer (that is, not the first-level first file layer), starting from the target-level first file layer, the fifth target data is transferred to the adjacent previous-level first file layer in sequence until it is transferred to the first-level first file layer, and finally the fifth target data is written from the first-level first file layer to the main memory.

[0352] If the fifth target data is stored in the hard disk, then when N is 1, after the fifth target data is written from the hard disk to the extended memory, the target-level first file layer where the fifth target data is located in the extended memory is the first file layer of the Nth level. At this time, the first file layer of the Nth level is also the first file layer of the first level, and the fifth target data can be directly written into the main memory through this file layer.

[0353] If the fifth target data is stored in the hard disk, then when N is greater than 1, after the fifth target data is written from the hard disk to the Nth-level first file layer of the extended memory, starting from the Nth-level first file layer, the fifth target data is transferred to the adjacent previous-level first file layer in sequence until it is transferred to the first-level first file layer, and finally the fifth target data is written from the first-level first file layer to the main memory.

[0354] In this way, by utilizing the multiple storage levels of the data files set in the extended memory and transferring the target data between the multiple storage levels in the extended memory, data can be transferred between the hard disk, the extended memory and the main memory, and the target data can be finally read from the main memory.

[0355] Optionally, the sixth preset condition may be that the extended memory stores target data matching the data to be read, or the hard disk stores target data matching the data to be read, and the target data is transferred from the hard disk to the Nth level first file layer of the extended memory.

[0356] That is, when the sixth preset condition is met, reading the fifth target data from the extended memory includes:

[0357] In the case where the fifth target data is stored in the extended memory, the fifth target data is read from the extended memory; or, in the case where the fifth target data is stored in the hard disk, the fifth target data is transferred from the hard disk to the Nth level first file layer of the extended memory, and the fifth target data is read from the extended memory.

[0358] In this way, the target data can be directly read from the extended memory, improving data reading efficiency and speed.

[0359] In an optional implementation, the hard disk includes M-level second file layers, the first-level second file layer is used to transfer stored data to the N-th-level first file layer, and M is greater than 1.

[0360] Correspondingly, transferring the fifth target data from the hard disk to the Nth level first file layer of the extended memory includes:

[0361] Determine a target-level second file layer where the fifth target data is located in the hard disk;

[0362] In the case where the target-level second file layer is the first-level second file layer in the hard disk, writing the fifth target data from the first-level second file layer into the Nth-level first file layer of the extended memory; or

[0363] In the case that the target-level second file layer is not the first-level second file layer, starting from the target-level second file layer of the hard disk, the fifth target data stored in the current-level second file layer is written to the adjacent previous-level second file layer until the fifth target data is written to the first-level second file layer, and the fifth target data is written from the first-level second file layer to the Nth-level first file layer of the extended memory.

[0364] In this process, the j-1th level second file layer in the hard disk is used to store data transferred from the jth level second file layer.

[0365] In this way, by utilizing multiple storage levels of data files set in the hard disk and by transferring target data between various file levels of the hard disk, data can be transferred from the hard disk to the extended memory, ensuring the orderliness of data access.

[0366] For the above data reading process, an exemplary implementation process is described for the case where the extended memory contains multiple first file layers and the hard disk contains multiple second file layers. Figure 5 As shown, the specific steps include:

[0367] Step 501, based on the data to be read, determine whether the main memory, the extended memory and the hard disk contain target storage data that matches the data to be read.

[0368] In an optional implementation process, the main memory contains files of variable memory tables and immutable memory tables, the extended memory contains multiple SST files stored in each level of the N-level first file layer, and the hard disk contains multiple SST files stored in each level of the M-level second file layer.

[0369] In this way, corresponding file indexes can be constructed in the main memory, extended memory and hard disk respectively, and the data to be read can be matched with the file index to determine where the target storage data matching the data to be read is stored.

[0370] Alternatively, in an optional implementation process, a corresponding data index can be directly constructed based on the data stored in the main memory, extended memory and hard disk, and the data to be read can be matched with the data index to determine where the target storage data matching the data to be read is stored.

[0371] The target storage data may be storage data having the same content as the data to be read, or storage data including the data to be read, or other set matching relationships.

[0372] When performing index matching, a Bloom filter can be used to detect which files or data items the data to be read may exist in. If the data to be read is a key value input by the user, the key value to be read is detected in which files or data items. These files or data items form the target storage data that matches the data to be read, and then determine whether the data to be read may be stored in the main memory, extended memory or hard disk, thereby improving query efficiency.

[0373] Step 502: If the extended memory contains the target storage data, determine the first file layer of the target level where the target storage data is located.

[0374] When it is determined that the target storage data matching the data to be read is located in the extended memory, first determine in which file layer of the N-level first file layer the target storage data is located, that is, determine the target level first file layer where the target storage data is located, recorded as the X-th level first file layer.

[0375] Subsequently, after step 502, there may be two ways of reading data.

[0376] One way is to perform a major transformation on the original key-value database for the newly added extended memory, and then execute step 503. Alternatively, another way is to perform a minor transformation on the original key-value database for the newly added extended memory, and then execute steps 504 to 506.

[0377] Step 503: Read the target storage data from the first file layer of the target level.

[0378] In this step, after determining at which file level in the first file layer of level N the target storage data is stored, since the file level is a storage area in the memory space, the CPU can choose to directly read the data to be read from the memory address of the target storage data, thereby improving data reading efficiency and ensuring data reading response speed.

[0379] Step 504 , starting from the first file layer of the target level, the target storage data is decompressed layer by layer and then transferred to the first file layer of the previous level until the target storage data is transferred to the first file layer of the N-level first file layer.

[0380] Step 505, decompress the target storage data in the first file layer of the N-level first file layer and transfer it to the main memory.

[0381] Step 506: Read the target storage data from the main memory.

[0382] During step 504 to step 506, the data access rules between the first file layers of the Nth level under the file hierarchy architecture adopted by the extended memory are used to decompress the data of the first file layer of the Xth level layer layer by layer and transfer them to the first file layer of the first level. The data of the first file layer of the first level is decompressed and transferred to the main memory, specifically, it can be decompressed and transferred to the variable memory table in the main memory, and the CPU reads the data to be read from the variable memory table in the main memory. Under the premise of less modification of the original key-value database, the memory space is used to realize data reading, improve the data reading efficiency, and ensure the data reading response speed.

[0383] Among them, the data transfer in this process can be achieved by using DSA, which can further improve the data reading and writing performance in the database and improve the overall data processing performance of the database.

[0384] In some optional implementations, after determining in step 501 whether the main memory, the extended memory, and the hard disk contain target storage data matching the data to be read based on the data to be read, the method further includes:

[0385] If the main memory contains the target storage data, the target storage data is directly read from the main memory.

[0386] When it is determined that the target storage data matching the data to be read is located in the main memory, the CPU directly reads the data to be read from the main memory, realizes data reading in the memory space, improves data reading efficiency, and ensures data reading response speed.

[0387] In some optional implementations, after determining in step 501 whether the main memory, the extended memory, and the hard disk contain target storage data matching the data to be read based on the data to be read, the method further includes:

[0388] If the hard disk contains the target storage data, determining the target level second file layer where the target storage data is located;

[0389] Starting from the second file layer of the target level, the target storage data is decompressed layer by layer and then transferred to the second file layer of the previous level until the target storage data is transferred to the first level second file layer of the M level second file layer in the hard disk;

[0390] Decompressing the target storage data in the first-level second file layer in the M-level second file layer and transferring and storing the data in the N-level first file layer in the extended memory;

[0391] Read the target storage data from the first file layer of level N in the extended memory; or,

[0392] Starting from the first file layer of the Nth level in the extended memory, the target storage data is decompressed layer by layer and then transferred and stored in the first file layer of the previous level until the target storage data is transferred and stored in the first file layer of the first level in the extended memory;

[0393] Decompressing the target storage data in the first file layer of the first level and transferring them to the main memory;

[0394] Read the target storage data from the main memory.

[0395] When it is determined that the target storage data matching the to-be-read data is located in the hard disk, the target-level second file layer where the target storage data is located is recorded as the Xth-level second file layer.

[0396] The data of the second file layer of the Xth level is decompressed layer by layer and transferred to the second file layer of the first level, and the data of the second file layer of the first level is decompressed and transferred to the first file layer of the Nth level in the extended memory. The CPU reads the data to be read from the first file layer of the Nth level based on the memory address of the target storage data, ensuring the effective reading of the data to be read in the hard disk.

[0397] Alternatively, after the data of the second file layer of the Xth level is decompressed and transferred to the second file layer of the first level, and the data of the second file layer of the first level is decompressed and transferred to the first file layer of the Nth level in the extended memory, the data of the first file layer of the Nth level is decompressed and transferred to the first file layer of the first level. The data of the first file layer of the first level is decompressed and transferred to the main memory, specifically, it can be decompressed and transferred to the variable memory table in the main memory, and the CPU reads the data to be read from the variable memory table in the main memory, so as to ensure the effective reading of the data to be read in the hard disk under the premise of less modification of the key-value database.

[0398] In some optional implementations, after determining in step 501 whether the main memory, the extended memory, and the hard disk contain target storage data matching the data to be read based on the data to be read, the method further includes:

[0399] If the main memory, the extended memory and the hard disk do not contain the target storage data that matches the data to be read, a response message is returned.

[0400] The response information may be information for feedback that the query result is 0, or feedback information for a query operation error, etc., so as to provide the user with effective query prompts in the case where no query results are found.

[0401] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0402] Based on the same inventive concept, the embodiment of the present application further provides a data processing device. The data processing device provided in the embodiment of the present application can implement each process of the embodiment of the above-mentioned data processing method, and can achieve the same technical effect, so the specific limitations in one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here to avoid repetition.

[0403] In this embodiment, the computing side can be divided into functional modules according to the above method. For example, each function can be divided into functional modules, or two or more functions can be integrated into one processing module.

[0404] See also Figure 6 , Figure 6 The structure of a data processing device provided in the embodiment of the present application is Figure 1 For ease of explanation, only the parts related to the embodiments of the present application are shown.

[0405] The data processing device 600 includes a main memory, an extended memory and a hard disk; the extended memory includes an N-level first file layer, where N≥1; the hard disk includes an M-level second file layer, where M>1; the data processing device 600 also includes:

[0406] The data storage module 601 is used for:

[0407] When a first preset condition is met, the first target data in the main memory is written into the first-level first file layer of the extended memory, wherein, when N>1, the i-th level first file layer in the N-level first file layers is used to store the data transferred in the i-1-th level first file layer, 1 <i≤N;

[0408] When the second preset condition is met, the second target data stored in the Nth level first file layer of the extended memory is written into the first level second file layer of the hard disk, wherein the jth level second file layer in the M level second file layers is used to store the data transferred in the j-1th level second file layer, 1 <j≤M。

[0409] Optionally, the data processing device 600 further includes:

[0410] The data reading module 602 is used to:

[0411] When the fifth preset condition is met, fifth target data matching the data to be read is read from the main memory; or,

[0412] When a sixth preset condition is met, the fifth target data is read from the extended memory.

[0413] The above integrated modules can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0414] It should be noted that the relevant contents of each step involved in the above method embodiment can all be referred to the functional description of the corresponding functional module, which will not be repeated here.

[0415] In one embodiment, Figure 7 As shown, a computer device is provided. The computer device 7 of this embodiment includes: at least one processor 700 ( Figure 7 Only one is shown in the figure), a memory 701, and a computer program 702 stored in the memory 701 and executable on the at least one processor 700, wherein the processor 700 implements the steps of any of the above-mentioned method embodiments when executing the computer program 702.

[0416] The computer device 7 may be a computing device such as a desktop computer, a notebook, a palm computer, etc. The computer device 7 may include, but is not limited to, a processor 700 and a memory 701. Those skilled in the art will understand that Figure 7 It is only an example of computer device 7 and does not constitute a limitation of computer device 7. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0417] The processor 700 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0418] The memory 701 may be an internal storage unit of the computer device 7, such as a hard disk or memory of the computer device 7. The memory 701 may also be an external storage device of the computer device 7, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device 7. Further, the memory 701 may also include both an internal storage unit and an external storage device of the computer device 7. The memory 701 is used to store the computer program and other programs and data required by the computer device. The memory 701 may also be used to temporarily store data that has been output or is to be output.

[0419] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0420] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0421] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0422] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / computer equipment and methods can be implemented in other ways. For example, the apparatus / computer equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of the apparatus or unit, which can be electrical, mechanical or other forms.

[0423] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0424] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0425] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0426] The embodiments of the present application implement all or part of the processes in the above-mentioned embodiment methods, and may also be implemented through a computer program product. When the computer program product runs on a computer device, the steps in the above-mentioned method embodiments can be implemented when the computer device is executed.

[0427] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A computing system, characterized in that: include: processor; Main memory, used to store data; The extended memory includes N levels of first file layers, where N≥1; wherein the first level first file layer in the N levels of first file layers is used to store data transferred from the main memory; when N>1, the i-th level first file layer in the N levels of first file layers is used to store data transferred from the i-1th level first file layer, 1 <i≤N; A hard disk, wherein the hard disk comprises M-level second file layers, wherein M>1; wherein the first-level second file layer in the M-level second file layers is used to store data transferred from the N-level first file layer; the j-th-level second file layer in the M-level second file layers is used to store data transferred from the j-1-th-level second file layer, wherein 1 <j≤M。 2. The computing system of claim 1, wherein: The extended memory includes a first group of the N-level first file layers and a second group of the N-level first file layers; The first level first file layer in the first group of the N level first file layers is used to store the first target key transferred from the main memory; The first level first file layer in the second group of N level first file layers is used to store the first target value corresponding to the first target key transferred from the main memory; and / or, The hard disk includes a first group of the M-level second file layers and a second group of the M-level second file layers; The first level second file layer in the first group of the M level second file layers is used to store the second target key transferred from the Nth level first file layer in the first group of the N level first file layers in the extended memory; The first level second file layer in the second group of M level second file layers is used to store the second target value corresponding to the second target key transferred from the Nth level first file layer in the second group of N level first file layers in the extended memory.

3. The computing system of claim 1, wherein: The computing system includes a computing device and an extended memory device, wherein the computing device includes the processor and the main memory; the extended memory device includes the extended memory; or The computing system includes a computing device including the processor, the main memory, and the extended memory.

4. A data processing method, applied to a computing system, characterized in that: The computing system includes a main memory, an extended memory and a hard disk; the extended memory includes an N-level first file layer, where N≥1; the hard disk includes an M-level second file layer, where M>1; The data processing method comprises: When a first preset condition is met, the first target data in the main memory is written into the first-level first file layer of the extended memory, wherein, when N>1, the i-th level first file layer in the N-level first file layers is used to store the data transferred in the i-1-th level first file layer, 1 <i≤N; When the second preset condition is met, the second target data stored in the Nth level first file layer of the extended memory is written into the first level second file layer of the hard disk, wherein the jth level second file layer in the M level second file layers is used to store the data transferred in the j-1th level second file layer, 1 <j≤M。 5. The data processing method according to claim 4, characterized in that: When the remaining storage space of the main memory is less than a first threshold, it is determined that the first preset condition is satisfied; or, when a first specific user operation is detected, it is determined that the first preset condition is satisfied.

6. The data processing method according to claim 5, characterized in that: When the remaining storage space of the main memory is less than a first threshold, determining that the first preset condition is satisfied includes: If the remaining storage space of the variable memory table in the main memory is less than the amount of data to be stored, the variable memory table is converted into an immutable memory table, and when the remaining storage space in the main memory is insufficient to create a new variable memory table, it is determined that the remaining storage space of the main memory is less than the first threshold.

7. The data processing method according to claim 4, characterized in that: The data processing method further includes: When the third preset condition is met, the third target data stored in the i-1th level first file layer in the N-level first file layers is stored in the i-th level first file layer; 1 <i≤N。 8. The data processing method according to claim 7, characterized in that: The step of storing the third target data stored in the i-1th level first file layer in the N-level first file layers into the i-th level first file layer when the third preset condition is satisfied includes: When the remaining storage space in the first file layer of the i-1th level is less than the second threshold, starting from the first file layer of the i-1th level, the third target data stored in the first file layer of the current level is written into the first file layer of the i-th level; Until the remaining storage space in the first file layer of the (i-1)th level is greater than the second threshold.

9. The data processing method according to claim 4, characterized in that: The data processing method further includes: If the remaining storage space of the variable memory table in the main memory is greater than the amount of data to be stored, then when the remaining storage space of the write-ahead log in the extended memory is greater than the amount of backup data of the data to be stored, the data to be stored is written into the variable memory table.

10. The data processing method according to claim 4, characterized in that: When the remaining storage space of the first file layer at level N is less than a fourth threshold, determining that the second preset condition is satisfied; or, When a second specific user operation is detected, it is determined that the second preset condition is met.

11. The data processing method according to claim 4, characterized in that: The data processing method further includes: When the fourth preset condition is met, the fourth target data stored in the j-1th level second file layer in the M-level second file layers is stored in the jth level second file layer; 1 <j≤M。 12. The data processing method according to claim 11, characterized in that: The storing of the fourth target data stored in the j-1th level second file layer in the M level second file layers into the jth level second file layer when the fourth preset condition is satisfied includes: When the remaining storage space in the j-1th level second file layer is less than the fifth threshold, starting from the j-1th level second file layer, the fourth target data stored in the current level second file layer is written into the jth level second file layer; Until the remaining storage space in the j-1th level second file layer is greater than the fifth threshold.

13. The data processing method according to claim 4, characterized in that: The extended memory includes a first group of the N-level first file layers, and a second group of the N-level first file layers; The step of writing the first target data in the main memory to the first-level first file layer of the extended memory when the first preset condition is met includes: When the first preset condition is met, the first target key in the main memory is written into the first level first file layer in the first group of the N level first file layers; and / or the first target value corresponding to the first target key in the main memory is written into the first level first file layer in the second group of the N level first file layers; The step of writing the second target data stored in the Nth level first file layer of the extended memory to the first level second file layer of the hard disk when the second preset condition is met comprises: When the second preset condition is met, the second target key stored in the N-th level first file layer in the first group of the N-level first file layers in the extended memory is written to the hard disk, and / or the second target value corresponding to the second target key stored in the N-th level first file layer in the second group of the N-level first file layers in the extended memory is written to the hard disk.

14. The data processing method according to claim 4, characterized in that: Before writing the first target data in the main memory into the first-level first file layer of the extended memory when the first preset condition is met, the method further includes: Formatting the extended memory using a memory file system; Configure initialization parameters, wherein the initialization parameters at least include the file directory of the extended memory, the storage directory of the write-ahead log in the extended memory, the size of the first file layer of level N in the extended memory, and the number of file layers; If it is detected that the remaining storage space in the extended memory meets the required capacity of the N-level first file layer, the N-level first file layer is initialized in the extended memory based on the initialization parameter, and the write-ahead log is initialized.

Images