Data system

Abstract

The specification provides a data system, comprising: a computing layer comprising a plurality of computing nodes, each of which is communicatively connected with a memory layer, the computing layer being configured to write data to the memory layer based on a write request from a client or read data from the memory layer based on a read request from the client; the memory layer comprising a plurality of memory nodes, each of which is communicatively connected with a persistent storage layer, the memory layer being configured to receive data written by the computing layer and dump the data written by the computing layer to the persistent storage layer, and load corresponding data in the persistent storage layer in response to a read request from the computing layer and send the corresponding data to the computing layer; and the persistent storage layer being configured to receive and save data dumped by the memory layer, and send corresponding data to the memory layer in response to a load request from the memory layer.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the technical field of database, and in particular, to a data system. BACKGROUND

[0002] In the field of modern information technology, databases play a crucial role as the core component of data storage and management. With the explosive growth of data volume and the increasing complexity of business requirements, database technology continues to evolve and advance. Shared databases formed by technologies such as object storage serve as an important database architecture, allowing multiple users or applications to access and share the same data resources simultaneously, greatly improving data utilization efficiency and business collaboration capabilities.

[0003] However, the shared database in the related art can only share persistent data, and the memory resources between database instances cannot be shared, resulting in low utilization of memory resources of database instances and waste. SUMMARY

[0004] Therefore, one or more embodiments of the present specification provide a data system.

[0005] To achieve the above-mentioned purpose, one or more embodiments of the present specification provide technical solutions as follows:

[0006] According to a first aspect of one or more embodiments of the present specification, a data system is provided, which comprises:

[0007] a computing layer comprising a plurality of computing nodes, each of which is in communication connection with a memory layer, the computing layer being configured to write data to the memory layer based on a write request of a client or read data from the memory layer based on a read request of the client;

[0008] a memory layer comprising a plurality of memory nodes, each of which is in communication connection with a persistent storage layer, the memory layer being configured to receive data written by the computing layer and dump the data written by the computing layer to the persistent storage layer, and to load corresponding data in the persistent storage layer in response to a read request of the computing layer and send the corresponding data to the computing layer;

[0009] a persistent storage layer configured to receive and save data dumped by the memory layer, and to send corresponding data to the memory layer in response to a load request of the memory layer.

[0010] In one possible embodiment of the present specification, the computing layer is configured to write data indicated by the write request into a memory table of a local memory, and to transfer data in the memory table of the local memory to a memory table of the memory layer.

[0011] In a possible implementation of the present disclosure, the computing layer is further configured to dump the memory table in the local memory to the local disk to form an ordered string table, and transfer the data in the ordered string table in the local disk to the memory table in the memory layer after converting the data in the memory table to the format of the memory table.

[0012] In a possible implementation of the present disclosure, the plurality of computing nodes of the computing layer comprises a master node and at least one follower node.

[0013] The master node is configured to generate a pre-write log based on the write request, synchronize the pre-write log to each follower node and the persistent storage layer, and replay the pre-write log to write data to the memory table in the local memory.

[0014] The follower node is configured to save the pre-write log sent by the master node.

[0015] In a possible implementation of the present disclosure, the memory layer is configured to allocate space for the data transferred by the computing layer when the computing layer transfers data to the memory layer, save the data transferred by the computing layer in the allocated space, and return the address of the allocated space to the computing layer.

[0016] In a possible implementation of the present disclosure, the computing layer is configured to read the data indicated by the read request in the local memory and the local disk, send the data indicated by the read request to the client when the data indicated by the read request is successfully read, forward the read request to the memory layer when the data indicated by the read request is not successfully read, and save the data returned by the memory layer in the local memory.

[0017] The memory layer is configured to read the data indicated by the read request in the memory table and the read cache data in the memory layer, send the data indicated by the read request to the computing layer when the data indicated by the read request is successfully read, send a load request to the persistent storage layer when the data indicated by the read request is not successfully read, save the data returned by the persistent storage layer in the read cache data and send the data to the computing layer, wherein the data indicated by the load request is the same as the data indicated by the read request.

[0018] The persistent storage layer is configured to send the data indicated by the load request to the memory layer.

[0019] In a possible implementation of the present disclosure, the memory layer is configured to allocate space for the data returned by the persistent storage layer when the data returned by the persistent storage layer is saved in the read cache data, save the data returned by the persistent storage layer in the allocated space, and add the allocated space to the read cache data.

[0020] In a possible implementation of the present disclosure, when allocating space, the memory layer allocates a data block as the allocated space and records address information of the data block in a data block address table.

[0021] In a possible implementation of the present disclosure, when allocating space, the memory layer increases the idle space according to at least one of the following if the capacity of the idle space does not meet a preset requirement:

[0022] dumping the memory table to the persistent storage layer;

[0023] deleting part or all of the read cache data.

[0024] In a possible implementation of the present disclosure, the computing layer is configured to load part or all of the read cache data of the memory layer to the added computing node after the computing node is added.

[0025] The computing layer is configured to load part or all of the read cache data of the memory layer to the computing node after the computing node is restarted.

[0026] In a possible implementation of the present disclosure, after any computing node of the computing layer is switched from a follower node to a master node, the computing node is configured to load prewrite logs in the persistent storage layer locally, and replay logs after a write time point in the prewrite logs to write data after the write time point to the memory layer, where the write time point represents an end time point of data that has been written in the memory layer.

[0027] In a possible implementation of the present disclosure, the memory layer is configured to determine a data range of any offline memory node, load prewrite logs in the persistent storage layer to the memory layer, and replay a part of the prewrite logs corresponding to the data range to load data in the data range to other memory nodes.

[0028] In a possible implementation of the present disclosure, each computing node of the computing layer is integrated in the same device as one memory node of the memory layer, or each memory node of the memory layer is a device independent of all computing nodes in the computing layer.

[0029] In a possible implementation of the present disclosure, the persistent storage layer includes a local storage layer and / or an object storage layer.

[0030] The technical solution provided by the embodiments of the present disclosure can have the following beneficial effects:

[0031] The data system provided by the embodiments of the present specification comprises a computing layer, a memory layer and a persistent storage layer. The computing layer is capable of writing data into the memory layer based on a write request of a client, or reading data from the memory layer based on a read request of the client. The memory layer is capable of saving the data written by the computing layer and dumping the data to the persistent storage, and loading corresponding data in the persistent storage layer in response to the read request of the computing layer to send the corresponding data to the computing layer. The persistent storage layer can save the data dumped by the memory layer and send corresponding data to the memory layer in response to the loading request of the memory layer. It can be seen that the computing layer and the memory layer in the system are in the form of clusters, which is beneficial to expansion and resource utilization. Moreover, the memory layer and the computing layer are independent of each other, and the memory layer can be accessed by all computing nodes of the computing layer, thereby realizing the effect of sharing the memory resources in the system among all computing nodes, improving the utilization rate of the memory resources in the system, and further improving the read-write performance of the data system. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 FIG. 1 is a structural schematic diagram of a data system provided by an example embodiment.

[0033] Figure 2 FIG. 2 is a data architecture of a memory layer provided by an example embodiment.

[0034] Figure 3 FIG. 3 is a schematic diagram of writing data by a data system provided by an example embodiment.

[0035] Figure 4 FIG. 4 is a schematic diagram of reading data by a data system provided by an example embodiment.

[0036] Figure 5 FIG. 5 is a structural schematic diagram of an apparatus provided by an example embodiment. DETAILED DESCRIPTION

[0037] The example embodiments will be described in detail herein, with examples shown in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following example embodiments do not represent all implementations consistent with one or more embodiments of the present specification. Instead, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of the present specification, as detailed in the appended claims.

[0038] It should be noted that the steps of the corresponding method are not necessarily performed in the order shown and described in the specification in other embodiments. In some other embodiments, the steps included in the method thereof can be more or less than described in the specification. In addition, a single step described in the specification may, in other embodiments, be broken down into multiple steps for description; and multiple steps described in the specification may, in other embodiments, be combined into a single step for description.

[0039] First, some concepts involved in the specification are explained.

[0040] LSM-Tree: Log-Structured Merge Tree, a high-efficiency storage structure widely used in database systems, aiming to optimize write performance and disk space utilization; it reduces the overhead of random writes by caching write operations in memory and periodically merging data to disk.

[0041] Object storage: Object storage is a data management architecture that efficiently and flexibly stores and retrieves massive amounts of unstructured data by treating data as objects rather than files or blocks. The "everything is an object" concept emphasizes that all data, whether documents, pictures, videos, or logs, can be uniformly processed as objects, helping to improve access efficiency and scalability.

[0042] In the related art, the computing resources and memory resources of each database instance are bound and strongly coupled, so the computing resources and memory resources cannot be independently expanded, and when one of them is expanded, the other needs to be expanded synchronously, and this process wastes resources.

[0043] Based on the above technical problems, at least one embodiment of the specification provides a data system that can be implemented based on an LSM-Tree (Log-Structured Merge Tree) storage engine.

[0044] Please refer to the attached Figure 1The data system comprises a computing layer, a memory layer and a persistent storage layer. The computing layer comprises a plurality of computing nodes, each of which is communicatively connected with the memory layer, and is configured to write data into the memory layer based on a write request from a client or read data from the memory layer based on a read request from the client. The memory layer comprises a plurality of memory nodes, each of which is communicatively connected with the persistent storage layer, and is configured to receive the data written by the computing layer, dump the data written by the computing layer to the persistent storage layer, and load corresponding data in the persistent storage layer in response to the read request from the computing layer and send the corresponding data to the computing layer. The persistent storage layer is configured to receive and save the data dumped by the memory layer, and send corresponding data to the memory layer in response to a load request from the memory layer.

[0045] The computing layer and the memory layer are in the form of a cluster, i.e. a plurality of nodes form a consensus protocol group, such as a Paxos protocol group, and a leader node in the protocol group is configured to interact with the outside and manage the inside. For example, the computing layer has a plurality of data partitions, each of which has a replica on part or all of the computing nodes of the computing layer, and all the replicas of each data partition form a consensus protocol group and elect a leader replica, which is the master node of the data partition and interacts with the outside and manages the inside. In other words, the master node in the computing layer can be the master node of each data partition it contains, and the master nodes of these data partitions coincide or do not coincide, and preferably the master nodes of different data partitions are different computing nodes of the computing layer, so as to share and balance the computing pressure. For another example, a plurality of memory nodes in the memory layer form a consensus protocol group and elect a master node, which can manage all memory resources in the memory layer, such as allocating space and releasing space, and can also interact with the outside, i.e. interact with the computing nodes of the computing layer and interact with the persistent storage layer.

[0046] The computing node has a local cache, which is divided into local memory and local disk. Taking one of the data partitions contained in the computing layer as an example, the plurality of computing nodes of the computing layer include a master node and at least one follower node. Hereinafter, the computing layer is described by taking one of the data partitions it contains as an example, and the content described is applicable to each data partition it contains.

[0047] The memory layer can have a memory table MemTable and a read cache data Micro Cache. The memory table is a write cache data for saving the data written by the computing layer, and the read cache data is used to save the data loaded from the persistent storage device to the memory layer. Please refer to the attached Figure 2The memory resources in the memory layer can be divided into multiple slabs according to specific capacities, and the slabs are distributed in different memory nodes; when the data system allocates memory space for different tenants, the slabs can be used as allocation units. The space in the slab can be divided into multiple chunks, and the chunks can be divided into multiple types according to capacities. When the memory layer allocates memory space for write cache data and read cache data, the chunks can be used as allocation units. For example, the capacity of a slab is 1 GB, and the capacities of chunks are 16 KB, 32 KB, 2 MB, etc.

[0048] The memory layer has a slab address table (Slab Address Table) recording the address of each slab. The address of the slab can include the label (for example, the label) of the memory node where the slab is located, the physical address, the address of the slab in the memory node, the tenant and the data partition to which the slab belongs, etc. The slab address table can have a copy in each memory node of the memory layer, thereby ensuring high availability.

[0049] The memory layer has a chunk address table (Chunk Address Table) recording the address of each chunk. The address of the chunk can include the label (for example, the label) and the physical address of the slab where the chunk is located, and the reference count of the chunk, i.e., the number of times the chunk is referenced. The chunk address table can have a copy in each memory node of the memory layer, thereby ensuring high availability.

[0050] The memory layer and the computing layer can be connected by RDMA (Remote Direct Memory Access) for communication, i.e., each computing node and each memory node are connected by RDMA for communication. The data transferred or the request forwarded by the computing layer to the memory layer is processed by the master node of the memory layer.

[0051] For example, each computing node of the computing layer is integrated with a memory node of the memory layer in the same device. In other words, each database instance in the data system has memory resources and computing resources, and the memory resources are shared among different database instances, i.e., the computing resources of each database instance can access the memory resources on the same device and the memory resources on other database instances; the memory resources of these database instances form a consensus protocol group, and the computing resources of these database instances form a consensus protocol group as the computing layer.

[0052] For another example, each memory node of the memory layer is a device independent of all the computing nodes in the computing layer. In other words, a plurality of computing resource-based devices form a consensus protocol group as the computing layer, and a plurality of memory resource-based devices form a consensus protocol group as the memory layer. In this example, the memory layer is completely independent of the computing layer, is more easily independently expanded, and has a higher utilization rate of memory resources.

[0053] The persistent storage layer can include a local storage layer and / or an object storage layer. The local storage layer can also include a plurality of storage nodes, each of which can be integrated with a memory node in the same device, or integrated with a computing node in the same device, or independent of all the memory nodes and the computing nodes. The object storage layer forms a shared persistent device among different database instances.

[0054] If the persistent storage layer includes a local storage layer and an object storage layer, when the memory layer dumps data, the data can be first dumped to the local storage layer, and then uploaded to the object storage layer directly or after being combined.

[0055] The data system provided by the embodiments of the present disclosure includes a computing layer, a memory layer, and a persistent storage layer. The computing layer can write data into the memory layer based on a write request of a client, or read data from the memory layer based on a read request of the client. The memory layer can save the data written by the computing layer and dump the data to the persistent storage layer, and in response to a read request of the computing layer, load corresponding data in the persistent storage layer to send the corresponding data to the computing layer. The persistent storage layer can save the data dumped by the memory layer, and in response to a load request of the memory layer, send corresponding data to the memory layer. It can be seen that the computing layer and the memory layer in the system are in a cluster form, which is beneficial to expansion and resource utilization. Moreover, the memory layer and the computing layer are independent of each other, and the memory layer can be accessed by all the computing nodes of the computing layer, thereby realizing the effect of sharing memory resources in the system among all the computing nodes, improving the utilization rate of memory resources in the system, and further improving the read-write performance of the data system.

[0056] Please refer to the accompanying drawings Figure 3 which exemplarily shows a process of data writing of the data system. Next, the data writing process will be described in detail.

[0057] First, the operation of the computing layer will be described.

[0058] The computing layer is configured to write data indicated by the write request into a memory table of the local memory, and transfer the data in the memory table of the local memory to a memory table of the memory layer.

[0059] The computing layer writes the data indicated by the write request into a memory table of the local memory in the form of a trans node. The computing layer can transfer the trans node in the memory table to a memory table of the memory layer through an RDMA one-sided operation. It can be understood that when writing the trans node into the memory table of the local memory, the computing layer can transfer the trans node to the memory layer in real time or in batches.

[0060] For example, the master node is configured to generate a pre-write log (WAL) based on the write request, synchronize the pre-write log to each follower node and the persistent storage layer, and replay the pre-write log to write data into a memory table of the local memory; and the follower node is configured to save the pre-write log sent by the master node. In this example, only the master node replays the pre-write log, and after the master node transfers data in the memory table of the local memory to a memory table of the memory layer, each follower node can access the data in the memory table by accessing the shared memory layer.

[0061] If the remaining space of the local memory in the master node is small, for example, less than a certain threshold, due to the fast speed of writing the trans node, the large number of trans nodes to be written, the slow speed of transferring the trans node to the memory layer, or the like, the computing layer (i.e., the master node) is further configured to dump the memory table of the local memory to a local disk to form an ordered string table, and then convert the data in the ordered string table in the local disk to the data format of the memory table and transfer the data to the memory table of the memory layer.

[0062] The ordered string table formed by the master node dumping to the local disk is an ordered string micro table (Micro SSTable). The Micro SSTable is in a sparse format, which is convenient for mutual conversion with the memory table. The data in the data system has a parameter SCN that measures the order of the data. The larger the SCN, the later the data, and the smaller the SCN, the earlier the data. The data range of the Micro SSTable is represented by a start time (Start SCN) and an end time (End SCN). In the process of analyzing the trans node from the Micro SSTable and transferring the trans node to the memory layer, if the maximum SCN of the data in the memory table of the memory layer is greater than the End SCN of the Micro SSTable, the data contained in the Micro SSTable has been completely transferred to the memory layer, and the Micro SSTable can be deleted to recover the storage space occupied by the Micro SSTable in the local disk.

[0063] Next, the operation of the memory layer is introduced.

[0064] The memory layer is configured to allocate space for the data transferred by the computing layer, save the data transferred by the computing layer in the allocated space, and return the address of the allocated space to the computing layer.

[0065] The data transferred by the computing layer to the memory layer can be a trans node transferred from the memory table of the local memory of the computing layer, or a trans node obtained by resolving the Micro SSTable of the local disk of the computing layer, both of which are transferred to the memory table of the memory layer.

[0066] For example, when allocating space, the memory layer takes a data block Chunk as the allocated space, and records the address information of the data block in the data block address table. Since the data block Chunk stores data in the memory table of the memory layer, the data block Chunk is allocated and referenced by the memory table.

[0067] For another example, when allocating space, if the capacity of the idle space does not meet the preset requirement, the idle space is increased according to at least one of the following: dumping the memory table to the persistent storage layer; deleting part or all of the read cache data. If the memory table has reached the dumping condition, the memory table can be dumped to the persistent storage layer to release space; or part or all of the read cache data can be deleted to release space, preferably deleting part of the read cache data with lower heat. Wherein, the capacity of the idle space not meeting the preset requirement can mean that the idle space is less than the preset threshold, or the idle space is less than the space required by the data transferred by the computing layer.

[0068] When the memory table meets the freezing and dumping conditions, the memory layer can freeze the memory table and dump it to the persistent storage layer to form an ordered string table SSTable in the persistent storage layer.

[0069] The ordered string table SSTable is divided into multiple levels, such as low-level Mini SSTable, middle-level Minor SSTable, and high-level Major SSTable, and higher-level SSTable is formed by complication of lower-level SSTable, for example, multiple Mini SSTables can be combined into one Minor SSTable, and multiple Minor SSTables can be combined into one Major SSTable.

[0070] Finally, the operation of the persistent storage layer is introduced.

[0071] If the persistent storage layer includes an object storage layer, the in-memory table is dumped to the object storage layer to form a Mini SSTable, the Mini SSTable is merged level by level upwards to finally form a Major SSTable.

[0072] If the persistent storage layer includes a local storage layer and an object storage layer, the in-memory table is dumped to the local storage layer to form a Mini SSTable, the local storage layer can merge multiple Mini SSTables into a Minor SSTable and upload the Minor SSTable to the object storage layer, and the object storage layer can merge multiple Minor SSTables into a Major SSTable.

[0073] It should be understood that the local storage layer can include multiple storage nodes, and when the in-memory layer dumps the in-memory table to the local storage layer, the local storage layer can form an SSTable of the in-memory table on multiple storage nodes, thereby ensuring high availability of data.

[0074] It should be understood that after the local storage layer merges multiple Mini SSTables into a Minor SSTable and then uploads to the object storage layer, the number of write operations of the object storage layer can be reduced, the data write pressure of the object storage layer can be relieved, and the performance of the object storage layer can be ensured.

[0075] Please refer to the accompanying drawings Figure 4 which exemplarily shows the process of the data system when reading data, and the data reading process will be introduced in detail.

[0076] First, the operation of the computing layer is introduced.

[0077] The computing layer is used to read the data indicated by the read request in the local memory and the local disk, send the data indicated by the read request to the client when the data indicated by the read request is successfully read, forward the read request to the in-memory layer when the data indicated by the read request is not successfully read, and save the data returned by the in-memory layer in the local memory.

[0078] The local memory of each computing node in the computing layer has a memory table, a fusion result buffer FuseRow Cache, a micro block cache Micro Block Cache, etc., and the local disk of each computing node has a MicroSSTable, wherein the Fuse Row Cache is used to store data fused by the computing node in response to a read request, i.e., data returned to the client in response to a read request of the client previously, and the Micro Block Cache is used to store data loaded to the computing node from the read cache data of the memory layer, and the Fuse Row Cache and the Micro Block Cache have limited space and are configured with an eviction mechanism.

[0079] For example, when the computing layer reads data, the master node reads data indicated by a read request in its Fuse Row Cache, and if the data is successfully read, the read data is directly returned to the client; if the data is not successfully read, the data indicated by the read request is read in the memory table, the Micro Block, and the Micro SSTable (if any), and if the data is successfully read, the read data is returned to the client after being fused, and the fusion result is added to the Fuse Row Cache; if the data is not successfully read, it can be determined that the data indicated by the read request does not exist in the computing layer, and therefore the read request is forwarded to the memory layer.

[0080] Next, the operation of the memory layer is introduced.

[0081] The memory layer is used to read data indicated by a read request in the memory table and read cache data in the memory layer, send the data indicated by the read request to the computing layer if the data is successfully read, and send a loading request to the persistent storage layer if the data indicated by the read request is not successfully read, save data returned by the persistent storage layer to the read cache data and send the data to the computing layer, wherein the data indicated by the loading request is the same as the data indicated by the read request.

[0082] The memory table in the memory layer is write cache data, and the read cache data in the memory layer is Micro Cache, which stores micro blocks Micro Block loaded to the memory layer from the persistent storage layer, and the Micro Block is a component of a macro block Macro Block in the persistent storage layer, i.e., each Macro Block contains multiple Micro Blocks. When the memory layer loads data in the persistent storage layer into read cache data, the data is loaded in units of Micro Block to avoid loading too much irrelevant data to the memory layer.

[0083] For example, the memory layer reads the data indicated by the read request from its memory table and Micro Cache, and if the data is successfully read, sends the data (the Micro Block where the data is located) to the master node of the computing layer, so that the master node adds the data to the Micro Block Cache and the Fuse Row Cache, and returns to the client; if the data is not successfully read, it can be determined that the memory layer does not have the data indicated by the read request, so a load request is sent to the persistent storage layer, so that the persistent storage layer returns the data indicated by the read request.

[0084] If the persistent storage layer only includes the object storage layer, the load request is directly sent to the object storage layer; if the persistent storage layer includes the local storage layer and the object storage layer, the load request is sent to the local storage layer.

[0085] When the memory layer saves the data returned by the persistent storage layer to the read cache data, it can allocate space for the data returned by the persistent storage layer, and save the data returned by the persistent storage layer to the allocated space, so that the allocated space joins the read cache data.

[0086] For example, when the memory layer allocates space, a data block Chunk is used as the allocated space, and the address information of the data block is recorded in the data block address table. Since the data block Chunk stores data in the Micro Cache of the memory layer, the data block Chunk is allocated as a Micro Block referenced by the Micro Cache after being allocated.

[0087] For another example, when the memory layer allocates space, if the capacity of the idle space does not meet the preset requirement, at least one of the following is added to the idle space: dumping the memory table to the persistent storage layer; deleting part or all of the data in the read cache data. If the memory table has reached the dumping condition, the memory table can be dumped to the persistent storage layer to release space; part or all of the data in the read cache data can also be deleted to release space, and preferably the part of the data in the read cache data with lower heat is deleted. Wherein, the capacity of the idle space does not meet the preset requirement can mean that the idle space is less than the preset threshold, or the idle space is less than the space required by the data transferred by the computing layer.

[0088] Finally, the operation of the persistent storage layer is introduced.

[0089] The persistent storage layer is used to send the data indicated by the load request to the memory layer.

[0090] For example, the persistent storage layer can send the Micro Block where the data indicated by the loading request is located to the memory layer.

[0091] If the persistent storage layer only includes the object storage layer, the data indicated by the loading request is sent to the memory layer by the object storage layer; if the persistent storage layer includes the local storage layer and the object storage layer, the data indicated by the loading request is read in the local storage layer by the local storage layer, and if the reading is successful, the read data is directly sent to the memory layer; if the reading is not successful, the data indicated by the loading request in the object storage layer can be loaded locally and sent to the memory layer.

[0092] In some embodiments of the present specification, the computing layer is configured to load part or all of the read cache data of the memory layer to the added computing node after the computing node is added; and the computing layer is configured to load part or all of the read cache data of the memory layer to the computing node after the computing node is restarted.

[0093] It can be seen that, due to the decoupling and unbundling of the memory layer and the computing layer in the data system, less data is loaded in the computing node of the computing layer, and only part of the data with high heat in the memory layer needs to be loaded in the newly added computing node or the restarted computing node to preheat when the computing layer is expanded or the computing node is restarted, thereby improving the convenience of expanding the computing layer, reducing the impact of the computing node restart on the data service, and improving the speed of recovering the service after the computing node is restarted.

[0094] In some embodiments of the present specification, after any computing node of the computing layer is switched from a follower node to a master node, the computing node is configured to load the pre-write log in the persistent storage layer locally, and replay the log after the write time point in the pre-write log to write the data after the write time point to the memory layer, wherein the write time point is used to represent the end time point of the data written in the memory layer.

[0095] It can be seen that, since the follower node of the computing layer in the data system only stores the pre-write log and does not replay the pre-write log, and the master node writes the data to the memory layer synchronously or asynchronously after replaying the pre-write log, the follower node only needs to replay part of the pre-write log that has not been replayed before the original master node loses the master node identity after being switched to the master node, and then can run according to the master node. The new master node can determine the start time point (e.g., the start SCN) of replaying the pre-write log by the end time point (e.g., the end SCN) of the data written in the memory layer, i.e., the new master node replays the pre-write log with the end SCN of the data written in the memory layer as the start SCN, so as to ensure the consistency and integrity of the data when the master node is switched.

[0096] In some embodiments of the present disclosure, the memory layer is configured to determine the data range of any offline memory node, load the pre-write log in the persistent storage layer to the memory layer, and replay the part of the pre-write log corresponding to the data range to load the data in the data range to other memory nodes.

[0097] The offline memory node can be offline due to a crash or user operation.

[0098] As the write cache data or read cache data is stored in each memory node of the memory layer of the data system, and the data block address table recording the address of the write cache data and read cache data in the memory layer is stored in each memory node, the data range of the offline memory node, i.e., which write cache data and which read cache data in the memory node, can be determined by the data block address table in other memory nodes, and then the pre-write log in the persistent storage layer (e.g., object storage layer) is loaded to the memory layer and replayed according to the data range of the memory node, so as to load the data in the offline memory node to other memory nodes, thereby recovering the data range of the memory layer before the offline of the memory node.

[0099] Moreover, the memory layer has the ability to parse and replay the pre-write log; the memory layer is distributed, and the replay task can be divided into parallel tasks when replaying the pre-write log, thereby improving the efficiency of replaying the log.

[0100] In addition, when any memory node is deleted from the memory layer, the write cache data and read cache data in the memory node can be transferred to other memory nodes before the memory node is deleted.

[0101] As can be seen from the above embodiments, the computing resources and memory resources of the data system are decoupled and not bound, the computing resources can be independently expanded, reduced, replaced, sold, etc., the memory resources can also be independently expanded, reduced, replaced, sold, etc., and the memory resource failure does not affect the computing resource, and the computing resource failure also does not affect the memory resource. Overall, the memory resource is pooled and shared among different computing nodes, and the utilization rate is high. Therefore, a more flexible computing memory combination mode can be provided, for example, a smaller number of computing resources and a larger memory are combined to meet individual use requirements.

[0102] Figure 5 is a schematic structural diagram of an apparatus provided by an example embodiment. The apparatus can be used to run the functions of a computing node or a memory node in the data system. Please refer to Figure 5At the hardware level, the device includes a processor 502, an internal bus 504, a network interface 506, a memory 508, and a non-volatile memory 510, and can also include other hardware required by tasks. One or more embodiments of the present specification can be implemented in a software manner, such as reading a corresponding computer program from the non-volatile memory 510 by the processor 502 into the memory 508 and then running. Of course, in addition to the software implementation, one or more embodiments of the present specification do not exclude other implementation manners, such as a logic device or a combination of software and hardware, and the like, that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or a logic device.

[0103] The system, apparatus, module or unit illustrated in the above embodiments can be specifically implemented by a computer chip or an entity, or by a product with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0104] In a typical configuration, the computer includes one or more processors (CPU), input / output interfaces, network interfaces, and memories.

[0105] The memory can include a non-persistent memory in a computer readable medium, a random access memory (RAM) and / or a non-volatile memory such as a read-only memory (ROM) or a flash memory (flash RAM). The memory is an example of a computer readable medium.

[0106] Computer-readable media includes permanent and non-permanent, moveable and non- moveable media that can be implemented by any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic disks storage, quantum memory, graphene-based storage media, or other magnetic storage devices, or any other medium which can be used to store information and which can be accessed by a computing device. According to the definition provided herein, computer readable medium does not include transitory media, such as modulated data signals and carrier waves.

[0107] It is also to be noted that the terms "comprising", "including", and any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0108] The above description of certain examples of the disclosure has been presented for the purposes of illustration and description. Other examples are within the scope and range of equivalents of the claims. In some cases, acts or steps can be performed in an order different from that of the examples provided and still obtain desirable results. Additionally, the processes depicted in the accompanying figures can not necessarily require the particular order shown or sequential order to achieve desirable results. In certain implementations, multitasking and parallel processing can be advantageous.

[0109] The terminology used in this description is for the purpose of describing particular embodiments only and is not intended to limit the one or more embodiments of the description. As used in this description and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that the term "and / or," as used herein, refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0110] The user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the specification are information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0111] It should be understood that, although one or more embodiments of the specification can use the terms first, second, third, etc. to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the specification, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "when" or "in response to determining".

[0112] The above only describes the preferred embodiments of one or more embodiments of the specification, and does not limit one or more embodiments of the specification. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the specification shall be included in the protection scope of one or more embodiments of the specification.

Claims

1. A data system comprising: a computing layer comprising a plurality of computing nodes, each of which is communicatively connected with a memory layer, the computing layer being configured to write data to the memory layer based on a write request from a client or read data from the memory layer based on a read request from the client; wherein a plurality of data partitions are provided in the computing layer, each of which comprises a plurality of computing nodes, each of which comprises a local memory and a local disk; the computing layer being configured to read data in a data partition indicated by the read request in the local memory and the local disk, send the data in the data partition indicated by the read request to the client when the data is successfully read, forward the read request to the memory layer when the data in the data partition indicated by the read request is not successfully read, and save data returned by the memory layer in the local memory; the memory layer comprising a plurality of memory nodes, each of which is communicatively connected with a persistent storage layer, the memory layer being configured to receive data written by the computing layer, dump the data written by the computing layer to the persistent storage layer, and load corresponding data in the persistent storage layer in response to a read request from the computing layer and send the corresponding data to the computing layer; the persistent storage layer being configured to receive and save data dumped by the memory layer, and send corresponding data to the memory layer in response to a load request from the memory layer. 2.The data system of claim 1, wherein the computing layer is configured to write data indicated by the write request to a memory table in the local memory, and transfer data in the memory table in the local memory to a memory table in the memory layer. 3.The data system of claim 2, wherein the computing layer is further configured to dump the memory table in the local memory to the local disk to form an ordered string table, and transfer data in the ordered string table in the local disk to the memory table in the memory layer after converting the data to a data format of the memory table. 4.The data system of claim 2, wherein the plurality of computing nodes of the computing layer comprises a master node and at least one follower node; the master node is configured to generate a pre-write log based on the write request, synchronize the pre-write log to each of the follower nodes and the persistent storage layer, and replay the pre-write log to write data to the memory table in the local memory; the follower node is configured to save the pre-write log sent by the master node. 5.The data system of claim 2, wherein the memory layer is configured to allocate space for data transferred by the computing layer when the computing layer transfers data to the memory layer, save the data transferred by the computing layer in the allocated space, and return an address of the allocated space to the computing layer.

6. The data system of claim 1, the memory tier to read data indicated by the read request in memory tables and read cache data within the memory tier, send the data indicated by the read request to the compute tier upon successful read of the data indicated by the read request, send a load request to the persistent storage tier upon unsuccessful read of the data indicated by the read request, save data returned by the persistent storage tier to the read cache data and send to the compute tier, wherein, the data indicated by the load request is the same as the data indicated by the read request; the persistent storage layer is configured to send the data indicated by the load request to the memory layer.

7. The data system of claim 6, wherein the memory layer is configured to allocate space for data returned by the persistent storage layer and save the data returned by the persistent storage layer into the allocated space when saving the data returned by the persistent storage layer into the read cache data, so that the allocated space is added to the read cache data.

8. The data system of claim 5 or 7, wherein the memory layer is configured to allocate a data block as the allocated space and record address information of the data block in the data block address table when allocating the space.

9. The data system of claim 5 or 7, wherein the memory layer is configured to increase the idle space by at least one of the following when the capacity of the idle space does not meet a preset requirement when allocating the space: dumping the memory table to the persistent storage layer; and deleting part or all of the read cache data.

10. The data system of claim 1, wherein the compute layer is configured to load part or all of the read cache data of the memory layer to the added compute node after adding the compute node. The compute layer is configured to load part or all of the read cache data of the memory layer to the compute node after restarting the compute node.

11. The data system of claim 1, any computing node of the computing tier, upon being switched to a primary node by a following node, is configured to load pre-written logs within the persistent storage tier locally and replay the logs within the pre-written logs after a write time to write data after the write time to the in-memory tier, wherein, The write time point is configured to represent an end time point of the data written in the memory layer.

12. The data system of claim 1, wherein the memory layer is configured to determine a data range of the offline memory node after the memory node goes offline, load the prewrite log in the persistent storage layer to the memory layer, and replay a part of the prewrite log corresponding to the data range to load the data in the data range to other memory nodes.

13. The data system of claim 1, wherein each compute node of the compute layer is integrated in the same device as one memory node of the memory layer, or each memory node of the memory layer is a device independent of all compute nodes in the compute layer.

14. The data system of claim 1, wherein the persistent storage layer comprises a local storage layer and / or an object storage layer.

Images