Cloud data warehouse and data processing method
By introducing Fluid layer and persistent volume protocols into cloud data warehouses, the problem of difficult to implement the automatic scaling cache resource management function in the existing technology is solved, and the good elasticity and scalability of cloud data warehouses are achieved, reducing the difficulty of implementation.
Patent Information
- Application Number
- CN202411644447.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-18
- Publication Date
- 2025-03-11
AI Technical Summary
The prior art is difficult to develop automatic scalable cache resource management functions, and requires a lot of manpower and material resources, making it difficult to achieve good elasticity and scalability.
By introducing Fluid layer into the cloud data warehouse, including Dataset management, Runtime management, container storage interface drivers and Volume management, the persistent volume protocol and K8S are used for docking to achieve elasticity and scalability of cloud data warehouses.
It realizes good elasticity and scalability of cloud data warehouses, reduces implementation difficulty, and improves resource management efficiency.
Smart Images
Figure CN119597852A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of cloud data warehouses, and in particular to a cloud data warehouse and a data processing method. Background Art
[0002] Cloud data warehouse is a new generation of data warehouse deployed and operated in cloud environment. Cloud data warehouse can be divided into three-layer architecture: management layer, computing layer and storage layer. The management layer provides tenant access portal, resource management, metadata management, transaction management, parsing and execution scheduling services; the computing layer provides computing resources of different specifications and cache acceleration support; the storage layer provides object storage service and supports the persistence of result set cache.
[0003] In the prior art, when developing or building an automatically scalable cache resource management function, different storage call interfaces are usually developed to support different distributed storage systems, which requires a lot of manpower and material resources and is difficult. Summary of the invention
[0004] In view of this, the present application provides a cloud data warehouse and a data processing method, which can make the cloud data warehouse have good elasticity and scalability and have low implementation difficulty.
[0005] The first aspect of the present application provides a cloud data warehouse, including:
[0006] Compute nodes, underlying storage, and Fluid layer; the Fluid layer includes Dataset management, Runtime management, container storage interface driver, and Volume management;
[0007] The container storage interface driver is used to create the underlying storage of the cloud data warehouse as the underlying storage that complies with the persistent volume protocol;
[0008] Volume management is used to mount persistent volume declaration directories to compute nodes. Compute nodes operate persistent volumes in the same way as they operate local directories.
[0009] When a computing node reads or writes data to a persistent volume declaration directory, Dataset management is used to encapsulate the data into a Dataset and bind the Dataset to the Runtime. The Dataset is used to define the source and destination of the dataset. The Runtime is the cache engine that stores the dataset.
[0010] Runtime management is used to store the Dataset cache in the deployment unit on the same physical node as the compute node.
[0011] Optionally, the Fluid layer uses a persistent volume protocol to connect to K8S.
[0012] Optionally, the container storage interface driver is deployed on each node in the form of a Daemonset.
[0013] Optionally, the Volume management is used to manage the underlying storage of the cloud data warehouse.
[0014] Optionally, the cloud data warehouse further includes:
[0015] If a computing node reads a certain data frequently, Runtime Controller is used to cache the Dataset on the same node or a node close to the physical node where the computing node is located, and create multiple copies according to the frequency of reading by the computing node.
[0016] A second aspect of the present application provides a data processing method, which is applied to a cloud data warehouse, wherein the cloud data warehouse includes computing nodes, underlying storage, and a Fluid layer; wherein the Fluid layer includes Dataset management, Runtime management, a container storage interface driver, and Volume management; the data processing method includes:
[0017] The container storage interface driver creates the underlying storage of the cloud data warehouse as the underlying storage that complies with the persistent volume protocol;
[0018] Volume management mounts the persistent volume declaration directory to the compute node. The compute node operates the persistent volume in the same way as it operates the local directory.
[0019] When a computing node reads or writes data to a persistent volume declaration directory, the Dataset management encapsulates the data into a Dataset and binds the Dataset to the Runtime. The Dataset is used to define the source and destination of the dataset. The Runtime is the cache engine that stores the dataset.
[0020] Runtime management stores the Dataset cache in the deployment unit on the same physical node as the compute node.
[0021] Optionally, if a computing node reads certain data frequently, Runtime Management will cache the Dataset on the same node or a node close to the physical node where the computing node is located, and create multiple copies according to the frequency of reading by the computing node.
[0022] Optionally, the Fluid layer uses a persistent volume protocol to connect to K8S.
[0023] Optionally, the container storage interface driver is deployed on each node in the form of a Daemonset.
[0024] Optionally, the Volume management is used to manage the underlying storage of the cloud data warehouse.
[0025] A third aspect of the present application provides a server, including:
[0026] one or more processors;
[0027] a storage device having one or more programs stored thereon;
[0028] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the second aspects.
[0029] A fourth aspect of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the data processing method as described in any one of the second aspects is implemented.
[0030] As can be seen from the above scheme, the present application provides a cloud data warehouse and data processing method, wherein the cloud data warehouse includes computing nodes, underlying storage and Fluid layer; wherein the Fluid layer includes Dataset management, Runtime management, container storage interface driver and Volume management; the data processing method creates the underlying storage of the cloud data warehouse as underlying storage that complies with the persistent volume protocol through the container storage interface driver; then, Volume management mounts the persistent volume declaration directory to the computing node; wherein the computing node operates the persistent volume in the same way as the local directory; thereafter, when the computing node reads or writes data to the persistent volume declaration directory, Dataset management encapsulates the data into a Dataset and binds the Dataset to the Runtime; wherein Dataset is used to define the source and destination of the data set; Runtime is a cache engine for storing the data set; finally, Runtime management stores the cache of the Dataset in a deployment unit of the same physical node as the computing node. Thus, the cloud data warehouse has good elasticity and scalability, and is less difficult to implement. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0032] Figure 1 A schematic diagram of a cloud data warehouse provided in an embodiment of the present application;
[0033] Figure 2 An architecture diagram of a cloud data warehouse provided in another embodiment of the present application;
[0034] Figure 3 A specific flow chart of a data processing method provided by another embodiment of the present application;
[0035] Figure 4 A schematic diagram of a server for implementing a data processing method provided in another embodiment of the present application. DETAILED DESCRIPTION
[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0037] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0038] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0039] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.
[0040] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".
[0041] The present application embodiment provides a cloud data warehouse, such as Figure 1 As shown, specifically including:
[0042] Computing node 10, underlying storage 20 and Fluid layer 30.
[0043] The Fluid layer 30 includes Dataset management 31 , Runtime management 32 , container storage interface driver 33 and Volume management 34 .
[0044] It should be noted that cloud data warehouse is a new generation of data warehouse deployed and operated based on cloud environment. The storage layer is usually deployed on IaaS layer, and the computing layer runs on K8S, realizing the separation of storage and computing.
[0045] Among them, K8S: full name Kubernetes, is a portable container orchestration management tool created for container services. Containers are instances of image runtimes, which have the characteristics of rapid deployment and delivery, efficient virtualization, easy migration and expansion, and simple management.
[0046] Fluid is an open source Kubernetes-native distributed data set orchestration and acceleration engine that mainly serves data-intensive applications in cloud-native scenarios, such as big data applications, AI applications, etc. Fluid supports cache engines such as Alluxio Runtime, Jindo Runtime, and JuiceFS Runtime.
[0047] Dataset: A dataset is a collection of logically related data. Since different datasets have different characteristics and optimizations, datasets must be managed separately. Files with consistent characteristics will be used by the same computing engine.
[0048] Runtime: The interface of the execution engine that truly implements capabilities such as data set security, version management, and data acceleration. It includes how to create and manage the life cycle, and defines a series of life cycle methods.
[0049] Dataset Controller: Responsible for the lifecycle management of the Dataset, including creation, binding and unbinding with the Runtime, and deletion.
[0050] Runtime Controller: Responsible for the lifecycle management of the Runtime, including creation, scaling, triggering, and deletion of cache preheating and cleanup.
[0051] Volume Controller: Responsible for the creation and deletion of data volumes corresponding to the Dataset.
[0052] In the specific implementation process of the present application, the container storage interface driver 33 is used to create the underlying storage 20 of the cloud data warehouse as an underlying storage that complies with the persistent volume protocol.
[0053] Among them, the container storage interface driver 33 can be but not limited to CSIDriver, which provides a standard CSI interface and communicates with the Kubelet of this node. And cooperates with fuse to realize the remote storage mounting to the directory in the Pod. Pod is the smallest deployable computing unit that can be created and managed in Kubernetes. Containers are all running in Pods, and there can be one or more containers in a Pod.
[0054] In the specific implementation process of this application, the underlying storage 20 may include multiple object storages and multiple HDFS. Object storage is object-based storage, which is a general term used to describe methods for solving and processing discrete units, which are called objects. Object storage products usually provide massive, secure, low-cost, and highly reliable cloud storage services. HDFS is a distributed file system based on Hadoop.
[0055] It should be noted that Persistent Volume (PV) is used to define the configuration information of various storage resources. One PV corresponds to one volume. The definition of a PV includes storage type, storage size, access mode, etc. The actual storage of PV may be in NFS, iSCSI or other cloud storage systems. It shields users from the specific details of implementing storage and abstracts storage resources to the outside world. Persistent Volume Claim (PVC): PVC is used to describe a request for PV. The request information includes storage size, access mode, etc. PVC will only select PVs that meet its requirements for binding, and then specify which PVC to use when defining Pods during the application process. PVC and PV are one-to-one corresponding. Once a PV is bound to a PVC, it cannot be used by other PVCs, making storage management in Kubemetes more flexible and fun. PVC supports multi-tenant environments. Different tenants can use different PVCs to isolate storage resources to ensure data security and isolation. PVC also allows users to dynamically apply for and release storage resources according to demand, improving resource utilization.
[0056] In the specific implementation process of the present application, Volume management 34 is used to mount the persistent volume declaration directory to the computing node 10.
[0057] The computing nodes operate persistent volumes in the same way as they operate local directories.
[0058] In the specific implementation process of the present application, when the computing node 10 reads or writes data to the persistent volume declaration directory, the Dataset management 31 is used to encapsulate the data into a Dataset and bind the Dataset to the Runtime.
[0059] Among them, Dataset is used to define the source and destination of the dataset; Runtime is the cache engine that stores the dataset.
[0060] In the specific implementation process of the present application, Runtime management 32 is used to store the cache of the Dataset in a deployment unit of the same physical node as the computing node 10.
[0061] If the computing node Pod reads the data frequently, then in another embodiment of the present application, RuntimeControlle is used to cache the Dataset on the same node or a node close to the physical node where the computing node is located according to the frequency of reading by the computing node, and create multiple copies.
[0062] Optionally, in another embodiment of the present application, Volume management is also used to manage the underlying storage of the cloud data warehouse.
[0063] Optionally, in another embodiment of the present application, the Fluid layer uses the persistent volume protocol to connect to K8S. K8S can be used to connect to different underlying storages, and the specific connection method can be completed through K8S PVC.
[0064] The specific implementation is through CSIDriver. In another embodiment of the present application, CSIDriver is deployed on each node in the form of Daemonset. It is responsible for interacting with K8S, providing the corresponding CSI interface, creating the underlying storage of the cloud data warehouse as PVC, and mounting the directory to the cloud data warehouse computing node Pod.
[0065] like Figure 2As shown, it is an architecture diagram of a cloud data warehouse (cloud data warehouse) provided in an embodiment of the present application, which adds a Fluid layer and a container storage interface driver (CSIDriver) in the container cloud platform. The Fluid layer uses at least cache engines Alluxio Runtime, Jindo Runtime, JuiceFS Runtime, etc. The controller manager (ControllerManager) includes Dataset Controller (Dataset management), Runtime Controller (Runtime management) and Volume Controller (Volume management). The underlying storage of the cloud data warehouse is created as PVC through Fluid's CSIDriver, and the PVC can be mounted to the computing node Pod using Fluid's Volume Controller. Volume Controller is responsible for the management of the underlying storage of the cloud data warehouse. Fluid uses the PVC (data persistent volume) protocol and K8S for docking. Through K8S, different underlying storage can be docked, and the specific docking method can be completed through K8S's PVC. The specific implementation is through CSIDriver, and CSIDriver is deployed on each node in the form of Daemonset. Responsible for interacting with K8S, providing the corresponding CSI interface, creating the underlying storage of the cloud data warehouse as PVC, and mounting the directory to the cloud data warehouse computing node Pod. When the computing node Pod reads or writes data to the mounted directory, Fluid's DatasetController will encapsulate the data into a Dataset and bind the Dataset to Fluid's Runtime. Dataset is used to define where the dataset comes from and where you want to put it; Runtime is the cache engine that stores these datasets. Currently, Fluid supports Alluxio Runtime, Jindo Runtime, JuiceFS Runtime, etc. If the computing node Pod reads the data frequently, Fluid's Runtime Controller will cache the Dataset on the same node or a node close to the physical node where the computing node Pod is located, and create multiple copies according to the frequency of reading by the computing node Pod.
[0066] As can be seen from the above scheme, the present application provides a cloud data warehouse, which includes a computing node 10, an underlying storage 20 and a Fluid layer 30; wherein the Fluid layer 30 includes Dataset management 31, Runtime management 32, a container storage interface driver 33 and Volume management 34; the underlying storage 20 of the cloud data warehouse is created as an underlying storage that complies with the persistent volume protocol through the container storage interface driver 33; then, the Volume management 34 mounts the persistent volume declaration directory to the computing node 10; wherein the computing node 10 operates the persistent volume in the same way as the local directory; thereafter, when the computing node 10 reads or writes data to the persistent volume declaration directory, the Dataset management 31 encapsulates the data into a Dataset and binds the Dataset to the Runtime; wherein the Dataset is used to define the source and destination of the data set; the Runtime is a cache engine for storing the data set; finally, the Runtime management 32 stores the cache of the Dataset in a deployment unit of the same physical node as the computing node. Thus, the cloud data warehouse has good elasticity and scalability, and is less difficult to implement.
[0067] The embodiment of the present application provides a data processing method, which is applied to a cloud data warehouse, wherein the cloud data warehouse includes computing nodes, underlying storage, and a Fluid layer; wherein the Fluid layer includes Dataset management, Runtime management, a container storage interface driver, and Volume management, and the data processing method, such as Figure 3 As shown, specifically including:
[0068] S301. The container storage interface driver creates the underlying storage of the cloud data warehouse as underlying storage that complies with the persistent volume protocol.
[0069] The container storage interface driver can be, but is not limited to, CSIDriver, which provides a standard CSI interface to communicate with the local node Kubelet. It also works with fuse to mount remote storage to the directory in the Pod. Pod is the smallest deployable computing unit that can be created and managed in Kubernetes. Containers are all running in Pods, and a Pod can have one or more containers.
[0070] In the specific implementation process of this application, the underlying storage may include multiple object storages and multiple HDFS. Object storage is object-based storage, which is a general term used to describe methods for solving and processing discrete units, which are called objects. Object storage products usually provide massive, secure, low-cost, and highly reliable cloud storage services. HDFS is a distributed file system based on Hadoop.
[0071] It should be noted that Persistent Volume (PV) is used to define the configuration information of various storage resources. One PV corresponds to one volume. The definition of a PV includes storage type, storage size, access mode, etc. The actual storage of PV may be in NFS, iSCSI or other cloud storage systems. It shields users from the specific details of implementing storage and abstracts storage resources to the outside world. Persistent Volume Claim (PVC): PVC is used to describe a request for PV. The request information includes storage size, access mode, etc. PVC will only select PVs that meet its requirements for binding, and then specify which PVC to use when defining Pods during the application process. PVC and PV are one-to-one corresponding. Once a PV is bound to a PVC, it cannot be used by other PVCs, making storage management in Kubemetes more flexible and fun. PVC supports multi-tenant environments. Different tenants can use different PVCs to isolate storage resources to ensure data security and isolation. PVC also allows users to dynamically apply for and release storage resources according to demand, improving resource utilization.
[0072] S302, Volume Management mounts the persistent volume declaration directory to the computing node.
[0073] The computing nodes operate persistent volumes in the same way as they operate local directories.
[0074] S303: When a computing node reads or writes data to a persistent volume declaration directory, the Dataset management encapsulates the data into a Dataset and binds the Dataset to the Runtime.
[0075] Among them, Dataset is used to define the source and destination of the dataset; Runtime is the cache engine that stores the dataset.
[0076] S304. Runtime management stores the cache of the Dataset in a deployment unit on the same physical node as the computing node.
[0077] Optionally, in another embodiment of the present application, an implementation of the data processing method further includes:
[0078] If a computing node reads certain data frequently, Runtime Management will cache the Dataset on the same node or a node close to the physical node where the computing node is located, and create multiple copies based on the frequency of the computing node's reading.
[0079] Optionally, in another embodiment of the present application, the Fluid layer uses the persistent volume protocol to connect to K8S. K8S can be used to connect to different underlying storages, and the specific connection method can be completed through K8S PVC.
[0080] The specific implementation is through CSIDriver. In another embodiment of the present application, CSIDriver is deployed on each node in the form of Daemonset. It is responsible for interacting with K8S, providing the corresponding CSI interface, creating the underlying storage of the cloud data warehouse as PVC, and mounting the directory to the cloud data warehouse computing node Pod.
[0081] Optionally, in another embodiment of the present application, the Volume management is used to manage the underlying storage of the cloud data warehouse.
[0082] As can be seen from the above scheme, the present application provides a data processing method, which creates the underlying storage of the cloud data warehouse as the underlying storage that complies with the persistent volume protocol through the container storage interface driver; then, the Volume management mounts the persistent volume declaration directory to the computing node; wherein, the computing node operates the persistent volume in the same way as the local directory; thereafter, when the computing node reads or writes data to the persistent volume declaration directory, the Dataset management encapsulates the data into a Dataset and binds the Dataset to the Runtime; wherein, the Dataset is used to define the source and destination of the data set; the Runtime is a cache engine that stores the data set; finally, the Runtime management stores the cache of the Dataset in the deployment unit of the same physical node as the computing node. This makes the cloud data warehouse have good elasticity and scalability, and is less difficult to implement.
[0083] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0084] Another embodiment of the present application provides a server, such as Figure 4 As shown, including:
[0085] One or more processors 401 .
[0086] The storage device 402 stores one or more programs.
[0087] When the one or more programs are executed by the one or more processors 401 , the one or more processors 401 implement the data processing method as described in any one of the above embodiments.
[0088] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the data processing method as described in any one of the above embodiments is implemented.
[0089] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0090] It should be noted that the computer-readable medium mentioned above in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0091] Another embodiment of the present application provides a computer program product. When the computer program product is executed, it is used to execute the above-mentioned data processing method.
[0092] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device, or installed from a storage device, or installed from a ROM. When the computer program is executed by a processing device, the above-mentioned functions defined in the method of the embodiment of the present application are executed.
[0093] Although the subject matter has been described in language specific to structural features and / or method logic actions, it should be understood that the subject matter defined in this application is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing this application.
[0094] Although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present application. Certain features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination.
[0095] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.
Claims
1. A cloud data warehouse, characterized in that: include: Compute nodes, underlying storage, and Fluid layer; the Fluid layer includes Dataset management, Runtime management, container storage interface driver, and Volume management; The container storage interface driver is used to create the underlying storage of the cloud data warehouse as the underlying storage that complies with the persistent volume protocol; Volume management is used to mount persistent volume declaration directories to compute nodes. Compute nodes operate persistent volumes in the same way as they operate local directories. When a computing node reads or writes data to a persistent volume declaration directory, Dataset management is used to encapsulate the data into a Dataset and bind the Dataset to the Runtime. The Dataset is used to define the source and destination of the dataset. The Runtime is the cache engine that stores the dataset. Runtime management is used to store the Dataset cache in the deployment unit on the same physical node as the compute node.
2. The cloud data warehouse according to claim 1, characterized in that: The Fluid layer uses the persistent volume protocol to connect to K8S.
3. The cloud data warehouse according to claim 1, characterized in that: The container storage interface driver is deployed on each node in the form of a Daemonset.
4. The cloud data warehouse according to claim 1, characterized in that: The Volume management is used to manage the underlying storage of the cloud data warehouse.
5. The cloud data warehouse according to claim 1, characterized in that: Also includes: If a computing node reads a certain data frequently, Runtime Controller is used to cache the Dataset on the same node or a node close to the physical node where the computing node is located, and create multiple copies according to the frequency of reading by the computing node.
6. A data processing method, characterized in that: Applied to a cloud data warehouse, the cloud data warehouse includes computing nodes, underlying storage and a Fluid layer; wherein the Fluid layer includes Dataset management, Runtime management, container storage interface driver and Volume management; the data processing method includes: The container storage interface driver creates the underlying storage of the cloud data warehouse as the underlying storage that complies with the persistent volume protocol; Volume management mounts the persistent volume declaration directory to the compute node. The compute node operates the persistent volume in the same way as it operates the local directory. When a computing node reads or writes data to a persistent volume declaration directory, the Dataset management encapsulates the data into a Dataset and binds the Dataset to the Runtime. The Dataset is used to define the source and destination of the dataset. The Runtime is the cache engine that stores the dataset. Runtime management stores the Dataset cache in the deployment unit on the same physical node as the compute node.
7. The data processing method according to claim 6, characterized in that: Also includes: If a computing node reads certain data frequently, Runtime Management will cache the Dataset on the same node or a node close to the physical node where the computing node is located, and create multiple copies based on the frequency of the computing node's reading.
8. The data processing method according to claim 6, characterized in that: The Fluid layer uses the persistent volume protocol to connect to K8S.
9. The data processing method according to claim 6, characterized in that: The container storage interface driver is deployed on each node in the form of a Daemonset.
10. The data processing method according to claim 6, characterized in that: The Volume management is used to manage the underlying storage of the cloud data warehouse.