Storage and calculation separation method and device based on cloud native construction, and computer equipment

Abstract

The invention belongs to the technical field of data processing, and relates to a storage and calculation separation method and device based on cloud native construction, computer equipment and a storage medium, and the method comprises the steps that a K8s basic environment is constructed based on cloud native, and the K8s basic environment is used for supporting deployment of big data components; based on the K8s basic environment, a data storage layer is deployed, and the data storage layer is used for managing data storage; according to the data storage layer, a data calculation layer is deployed, and the data calculation layer is used for managing data calculation; according to the data calculation layer, a data application layer is deployed, and the data application layer is used for managing data application. The method has the characteristics of dynamic capacity expansion, automatic fault recovery / transfer and the like, operation and maintenance human input is greatly reduced, the computing power of the server and the utilization rate of storage are fully improved, and the effects of cost reduction and efficiency improvement are better achieved; iterative updating of each component version can be managed through K8s, and the version compatibility problem caused by environment difference is reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a storage and computing separation method, device, computer equipment and storage medium based on cloud native construction. Background Art

[0002] With the advent of the Internet of Things, more and more data is generated in various industries. The exponential growth of data scale and computing needs has brought huge challenges to traditional data storage warehouses and data computing frameworks. Traditional technology frameworks can no longer meet the current and future data storage and computing needs.

[0003] Existing big data platforms are mostly based on Hadoop, a traditional integrated storage and computing design. However, this integration forces computing and storage resources to be tightly bound in a certain ratio. Cluster expansion requires an increase in the number of nodes, resulting in wasted memory, CPU, or disk space. Furthermore, triple-copy storage based on the Hadoop ecosystem (100+ nodes, petabyte-level storage) incurs high storage costs. The large number of Hadoop nodes also significantly increases operational costs, requiring significant manpower to manage and operate this massive node population. Vulnerabilities in open source big data components, coupled with the time-consuming compatibility verification and deployment required by traditional software deployment methods due to environmental differences, have increased the human and material resources required for operational maintenance. Summary of the Invention

[0004] The purpose of the embodiments of the present invention is to propose a storage and computing separation method, device, computer equipment and storage medium based on cloud native construction, so as to solve the problem that the existing big data platforms are mostly based on the traditional storage and computing integrated model built on Hadoop, which will lead to waste of memory, CPU or disk.

[0005] To solve the above technical problems, the present invention provides a storage and computing separation method based on cloud native architecture, which adopts the following technical solution and includes the following steps:

[0006] Build a K8s basic environment based on cloud native to support the deployment of big data components;

[0007] Based on the K8s basic environment, we deployed the storage components juiceFS and minio through the kubesphere application store to build a data storage layer, which is used to manage data storage.

[0008] Deploy a data computing layer based on the data storage layer, wherein the data computing layer is used to manage data computing;

[0009] Based on the data computing layer, a data application layer is deployed, and the data application layer is used to manage the application of data.

[0010] Preferably, the steps of building a K8s basic environment based on cloud native and supporting the deployment of big data components in the K8s basic environment specifically include:

[0011] Build user management module, API service module, scheduling module, resource management module, agent module and proxy network module.

[0012] Preferably, the data storage layer is deployed based on the K8s basic environment, and the steps for the data storage layer to manage data storage specifically include:

[0013] Deploy the data storage layer based on the open source JuiceFS cloud-native distributed file system.

[0014] Preferably, the data computing layer is deployed according to the data storage layer, and the steps of managing the data computing by the data computing layer specifically include:

[0015] Build a data computing layer based on Flink and Spark;

[0016] Trino-based multi-data source query and calculation engine meets the business scenarios of data warehouse computing, data analysis, large data aggregation, and multi-data source interactive calculation OLAP type.

[0017] Preferably, the data application layer is deployed according to the data computing layer, and the steps of managing the application of data at the data application layer specifically include:

[0018] Deploy data application module, data development module, data foundation module, and data governance module.

[0019] Preferably, based on the K8s basic environment, the storage components juiceFS and minio are deployed on the kubesphere application store to build a data storage layer. Before the step of managing data storage, the data storage layer further includes:

[0020] Enable the app store function of the K8s cluster;

[0021] Log in to the management node of the K8s cluster and find the Platform Management, Cluster Management, and Custom Resource Definition pages in sequence;

[0022] Modify the openpitrix.store property in the YAML file in ClusterConfiguration, adjust the configuration, download it to the master node of the K8s cluster, and update the cluster configuration by specifying the configuration file through kubectl apply.

[0023] In order to solve the above technical problems, the present invention also provides a storage and computing separation device based on cloud native construction, which adopts the following technical solutions, including:

[0024] The management layer is used to build a cloud-native Kubernetes infrastructure to support the deployment of big data components.

[0025] The storage layer is used to build a data storage layer based on the K8s basic environment and deploy the storage components juiceFS and minio through the kubesphere application store. The data storage layer is used to manage data storage.

[0026] A computing layer is used to deploy a data computing layer based on the data storage layer, and the data computing layer is used to manage the computing of data;

[0027] The application layer is used to deploy a data application layer based on the data computing layer, and the data application layer is used to manage the application of data.

[0028] Preferably, the management layer includes:

[0029] User management module, API service module, scheduling module, resource management module, agent module and proxy network module.

[0030] In order to solve the above technical problems, the present invention also provides a computer device, which adopts the technical solution described below, including a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the above-mentioned storage and computing separation method based on cloud native construction.

[0031] In order to solve the above technical problems, the present invention also provides a computer-readable storage medium, which adopts the technical solution described below. The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the above-mentioned storage and computing separation method based on cloud native construction are implemented.

[0032] Compared with the existing technology, the present invention has the following main beneficial effects: by deploying a big data storage and computing separation environment based on the K8s environment, it has the characteristics of dynamic expansion, automatic fault recovery / transfer, etc., which greatly reduces the manpower investment in operation and maintenance, and also fully improves the utilization rate of server computing power and storage, and better achieves the effect of reducing costs and increasing efficiency; K8s can be used to manage the iterative updates of various component versions, reduce version compatibility issues caused by environmental differences, and can achieve rapid iteration of software versions, shorten the development cycle, and improve development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the solutions in the present invention, a brief introduction is given below to the drawings required for use in describing the embodiments of the present invention. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0034] Figure 1 This is a flowchart of an embodiment of the storage and computing separation method based on cloud native construction of the present invention;

[0035] Figure 2 This is the logical diagram of the offline data warehouse component used in the cloud-native storage and computing separation method of the present invention;

[0036] Figure 3 This is a logical diagram of the real-time data warehouse component used in the cloud-native storage and computing separation method of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of an embodiment of a storage and computing separation device based on cloud native construction of the present invention;

[0038] Figure 5 This is a schematic diagram of the architecture of the cloud-native storage and computing separation device of the present invention;

[0039] Figure 6 It is a structural diagram of an embodiment of a computer device of the present invention. DETAILED DESCRIPTION

[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this invention belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit the present invention. The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. The terms "first" and "second" in the specification and claims of the present invention and the accompanying drawings are used to distinguish different objects, not to describe a specific order.

[0041] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0042] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0043] It should be noted that the storage and computing separation method based on cloud native construction provided in the embodiment of the present invention is generally executed by a server / terminal device. Accordingly, the storage and computing separation device based on cloud native construction is generally set in the server / terminal device.

[0044] It should be understood that the number of terminal devices, networks and servers is merely illustrative and any number of terminal devices, networks and servers may be provided as required.

[0045] Cloud native is an application designed for the "cloud" and therefore relies on the three-tier concepts of traditional cloud computing: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS).

[0046] Example 1

[0047] Please refer to Figure 1 , shows a flow chart of an embodiment of the present invention's storage and computing separation method based on cloud native construction. The storage and computing separation method based on cloud native construction includes the following steps:

[0048] Step S1: Build a K8s basic environment based on cloud native. The K8s basic environment is used to support the deployment of big data components.

[0049] In the server environment, download the KubeKey tool from the kubesphere official website. First, use the downloaded KubeKey to create a Kubernetes template YAML file. Then, modify the configuration based on the actual cluster node requirements and performance, adjust the distribution of component roles and resource usage, and specify the Kubernetes and KubeSphere versions. Then, use the create cluster command to specify the configuration file installation to quickly deploy and start the Kubernetes cluster. During deployment, use the log command to monitor the deployment progress of related modules.

[0050] In this embodiment, the electronic device (e.g., server / terminal device) on which the cloud-native storage and computing separation method is running can receive the cloud-native storage and computing separation request through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAXX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0051] K8s is an open source container orchestrator originally developed by Google for automating the deployment, scaling, and management of containerized applications.

[0052] K8s makes it easy to deploy and manage microservice architecture applications. It does this by forming an abstraction layer above the cluster, allowing development teams to smoothly deploy applications, while K8s primarily handles the following tasks:

[0053] Control and manage application resource usage, automatically load balance requests between multiple instances of an application, monitor resource usage and resource limits, and automatically prevent applications from consuming excessive resources and recover them again. If host resources are exhausted or a host crashes, migrating application instances from one host to another is a viable option. When new hosts join the cluster, the newly added additional resources can be automatically used.

[0054] In this embodiment, step S1 may further include the following steps:

[0055] Build user management module, API service module, scheduling module, resource management module, agent module and proxy network module.

[0056] KubeKey allows for one-click deployment of the user management module, API service module, scheduling module, resource management module, agent module, and proxy network module. These modules are built-in foundational components of a Kubernetes cluster. The API service module provides interfaces for adding, deleting, modifying, and querying various Kubernetes resource objects, as well as HTTP REST interfaces for monitoring and other purposes. These objects include pods, services, and replication controllers.

[0057] The API service module serves REST operations and provides a front end for the shared state of the entire K8s cluster, through which all other components interact.

[0058] The scheduling module is mainly responsible for monitoring the data in the API service module (etcd) and scheduling the service to the appropriate node according to the default or specified scheduling algorithm.

[0059] The resource management module is responsible for monitoring the status of all pods in the cluster and ensuring that all pods available for services are maintained at the specified number and status.

[0060] The agent module is mainly responsible for receiving and executing instructions from the resource management module, such as creating or deleting a specified pod on the local machine and reporting the status of the specified service on the local machine.

[0061] The proxy network proxy module is mainly responsible for managing the network status of all pods, as well as the internal and external communications of the entire cluster service.

[0062] In step S2, based on the K8s basic environment, the storage components juiceFS and minio are deployed through the kubesphere application store to build a data storage layer, which is used to manage data storage.

[0063] In this embodiment, step S2 may further include the following steps:

[0064] Deploy the data storage layer based on the open-source JuiceFS cloud-native distributed file system. JuiceFS doesn't officially have a Docker image, so you'll need to deploy a copy of the JuiceFS component in your local Docker environment and then use the export command to export the Docker image. Place the exported image in a public / private cloud environment that supports online downloads, create a yaml file for the K8s application, and specify the address of the previously exported image file. Once everything is ready, use a tenant account with permission to deploy applications to upload the newly created yaml file to the K8s management interface. Modify the configuration parameters to specify the Minio cluster to use. Once the modifications are complete, deploy with one click. Wait for the application deployment to complete before you can use it out of the box.

[0065] Based on the open source JuiceFS cloud-native distributed file system, the lower layer is connected to the original HDFS cluster, and a new minio cluster is deployed on the cloud native to build a unified entrance and exit, and the lower layer is deployed in a multi-storage cluster mode.

[0066] The upper layer of the data storage layer is hosted and integrated with the JuiceFS system. Leveraging JuiceFS's robust compatibility, it supports integration with HDFS clusters and the S3 access protocol. It can host existing HDFS file systems for archival storage, reducing migration workloads. It also integrates with Minio, which uses the S3 protocol, for hot data storage and computation. Redis is used as JuiceFS's metadata management system, leveraging its multi-data type support, transactional capabilities, and high efficiency to maximize the performance of the JuiceFS system. Furthermore, JuiceFS's local caching mechanism mitigates the performance losses associated with a separate storage and compute architecture. Deployed in a Kubernetes environment, JuiceFS reduces operational costs, facilitates version iterations, and supports dynamic scaling to accommodate rapidly increasing data volumes. The data storage layer is based on the open-source JuiceFS cloud-native distributed file system, compatible with the traditional Hadoop ecosystem and the S3 access protocol. This seamless integration with traditional HDFS file systems reduces data migration and unnecessary resource waste.

[0067] In some optional implementations of this embodiment, the following steps may be included before step S2:

[0068] Enable the app store function of the K8s cluster;

[0069] Log in to the management node of the K8s cluster and find the Platform Management, Cluster Management, and Custom Resource Definition pages in sequence;

[0070] Modify the openpitrix.store property in the YAML file in ClusterConfiguration, adjust the configuration, download it to the master node of the K8s cluster, and update the cluster configuration by specifying the configuration file through kubectl apply.

[0071] Step S3: deploying a data computing layer based on the data storage layer, where the data computing layer is used to manage data computing.

[0072] In this embodiment, step S3 may further include the following steps:

[0073] S31. Build a data computing layer based on Flink and Spark.

[0074] Build a data computing layer based on Flink and Spark to handle stream and batch data processing in different scenarios. Upload the Flink operator and Spark application template to the Kubernetes App Store. Using a tenant account with permission to deploy applications, select the newly uploaded Flink and Spark application templates in the Kubernetes management interface and modify the configuration parameters. Change the Flink memory configuration, the Spark cluster deployment mode, and the node and memory limits. Once the changes are complete, deploy with one click. Wait for the application to deploy and you can use it out of the box.

[0075] S32, a multi-data source query and calculation engine based on trino, meets the business scenarios of data warehouse calculation, data analysis, large data volume aggregation, and multi-data source interactive calculation OLAP type.

[0076] Compared to deploying Flink or Spark, deploying Trino requires a slightly longer step, requiring data source configuration. After deploying the Trino application using the general Trino application template, log in to the Kubernetes management page using the corresponding tenant account, navigate to Project Management -> Configuration Center -> Configuration Dictionary, and modify the catalog to configure Trino's data source connection. Trino supports a variety of data sources, including Minio, Hive, and MySQL. After configuring as needed, restart the Trino instance to use it.

[0077] The data computing layer includes four modules: a data acquisition module based on open source components, a real-time computing module, an offline computing module, and a task scheduling module. Flink, Spark, and query computing are built using Trino. Data warehouse computing (full name: a data warehouse is a subject-oriented, integrated, relatively stable data collection that reflects historical changes and is typically used to support management decisions. Here, the subject refers to the various decision-making data models created for data analysis) is primarily implemented by Trino, and offline scheduling is implemented by Airflow. All of these components (data acquisition module, real-time computing module, and offline computing module) are deployed in a cloud-native manner on a Kubernetes cluster.

[0078] The main data flow pattern in the data computing layer is data ingestion -> real-time / offline computing -> data storage / caching -> external service interfaces. The data ingestion module is implemented based on Pulsar, a cloud-native distributed message streaming platform that integrates messaging, storage, and lightweight functional computing. Its native architecture separates compute and storage, enabling optimal integration with Kubernetes. This design allows for a decoupled data ingestion and caching design, leveraging cloud-native features such as elastic scalability to further leverage the advantages of distributed message caching components. Real-time / offline computing uses Spark native and Flink operators for stream and batch computing. The data warehouse is built using Trino, Hive Metastore, MySQL, and Minio. Trino is used for offline data warehouse scheduling, computation, and interface queries. The Hive Metastore and MySQL architecture manages and stores data warehouse metadata. Minio stores data, and Airflow is used to schedule data warehouse tasks. All of these components are deployed in a Kubernetes environment, enabling unified deployment of component versions, automatic fault recovery, and elastic scalability.

[0079] The data computing layer consists of two main components: big data and databases. The big data component primarily includes Pulsar, a message cache component that natively supports the cloud-native streaming and computing separation architecture; Flink, a stream-batch computing component; Doris, a core component for real-time data warehouses; Spark, an offline batch computing component; Trino, a core query computing engine for offline data warehouses; and Airflow, an offline scheduling component. All components are deployed in a Kubernetes environment and natively support high availability, fault recovery / transfer, and dynamic capacity expansion. The database component includes ElasticSearch, a commonly used log retrieval component; MySQL, a high-performance ClickHouse database; and Redis, a high-performance cache database, to meet the needs of diverse applications and business scenarios.

[0080] Figure 2 This is the logical diagram of the offline data warehouse component used in the cloud native-based storage and computing separation method of the present invention. Figure 2 As shown in the figure, the offline data warehouse architecture includes: an external service API interface, a scheduling and computing module, and a data storage module. The external service API outputs data from the data aggregation result layer to the application area. The scheduling and computing module, the computing core of the entire offline data warehouse architecture, mainly uses Trino for offline SQL calculations, Spark for some batch complex logic calculations, and Airflow to manage the entire offline scheduling. Data storage includes the JuiceFS system that provides unified external storage services. The lower layer connects to Minio and HDFS for real storage, and uses Redis to host JuiceFS metadata. The offline data warehouse scheduling process includes:

[0081] First, the user writes the SQL script for data warehouse scheduling and submits it to the git environment, and then puts the written Spark program on Minio storage.

[0082] Secondly, enter the K8s cluster, configure the pulled git path in the airflow component configuration, and after configuring it to dag, you can log in to airflow to query the scheduled release status.

[0083] Finally, SQL scheduling primarily connects to trino for computation. Trino uses HiveMetaStore to manage metadata for the entire data warehouse table and stores this metadata in MySQL. Once the trino coordinator receives the SQL submitted by Airflow, it begins computation. Trino configures the data storage address to the JuiceFS interface address. JuiceFS stores files in Minio and manages the entire file system's metadata cache in Redis. When trino accesses data, it first establishes a connection through JuiceFS's read / write interface, then accesses Redis to obtain metadata information. After obtaining the data's specific storage address, it accesses Minio or HDFS to read and write the underlying data. When data is frequently accessed, JuiceFS's caching mechanism caches portions of the underlying Minio or HDFS data locally, improving the performance of storage-computing separation and eliminating performance losses caused by bandwidth.

[0084] Figure 3 This is the logical diagram of the real-time data warehouse component used in the cloud-native storage and computing separation method of the present invention. Figure 3 As shown in the figure, the real-time data warehouse architecture mainly includes: Pulsar, a messaging middleware for data access, Flink, a stream-batch computing engine, Doris, a real-time data warehouse, and Clickhouse, an OLAP data storage and external service API. The real-time data warehouse scheduling process includes:

[0085] First, real-time data is sent to the Pulsar cluster through the production client.

[0086] Secondly, after simple processing of pulsar data through flinkSQL, it is distributed to pulsar secondary cache, Doris for further calculation, and clickhouse for external use.

[0087] Again, after the second-layer pulsar obtains the data, the flink program performs secondary complex logical processing and writes it to the JuiceFS distributed storage system.

[0088] Finally, the application layer accesses the summary / result data in JuiceFS or ClickHouse through the data service API.

[0089] In step S4, data applications are packaged into Docker images according to the data computing layer. All applications are abstracted into the data application layer, which is managed by K8s.

[0090] In this embodiment, step S4 may further include the following steps:

[0091] S41. Upload the Airflow component template to the K8s application store, and use a tenant account with permission to deploy applications to deploy the Airflow component on the K8s management page.

[0092] By uploading the airflow component template in the K8s application store, using a tenant account with permission to deploy applications, deploy the airflow component on the K8s management page for offline scheduling of computing tasks.

[0093] S42. Deploy data application module, data development module, data foundation module, and data governance module.

[0094] The deployment process for the data application and data development modules is similar to the deployment of the juiceFS module in step S2. Docker images must be packaged locally and then published to the Kubernetes cluster. Deploying the data infrastructure and data governance modules requires uploading SQL scripts to the juiceFS file system, editing the Airflow scheduling DAG, and pushing it to Airflow through the interface. This triggers Airflow to generate the corresponding offline scheduling tasks to complete the offline scheduling task calculations.

[0095] Traditional big data platform architectures are mostly based on Hadoop, a coupled storage and computing architecture. Separating YARN and HDFS from separate environments results in significant performance losses. Furthermore, HDFS's triple-replication mechanism wastes significant storage resources in cloud environments, as cloud environments already offer multiple replication mechanisms. Adding HDFS's triple-replication mechanism can lead to exponentially higher costs.

[0096] Traditional Hadoop cluster deployment requires independent machine creation, basic environment deployment, and the addition of operation and maintenance monitoring. After deployment based on K8S, all components can be deployed with one click and are ready to use out of the box. It also supports dynamic expansion and elastic scaling. All resources can be used flexibly, which can not only cope with the performance pressure brought by business peaks, but also release additional computing power resources for use by other projects during business troughs.

[0097] The beneficial effects of implementing the present invention are as follows: by deploying a big data storage and computing separation environment based on the K8s environment, it has the characteristics of dynamic expansion, automatic fault recovery / transfer, etc., which greatly reduces the manpower investment in operation and maintenance, and also fully improves the utilization rate of server computing power and storage, and better achieves the effect of reducing costs and increasing efficiency; K8s can be used to manage the iterative updates of various component versions, reduce version compatibility issues caused by environmental differences, and achieve rapid iteration of software versions, shorten the development cycle, and improve development efficiency.

[0098] The present invention can be used in a wide variety of general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0099] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0100] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0101] Example 2

[0102] Further references Figure 4 , as a response to the above Figure 1 The present invention provides an embodiment of a storage and computing separation device based on cloud native construction, which is similar to the embodiment of the present invention. Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0103] like Figure 4 As shown, the cloud-native storage and computing separation device 50 described in this embodiment includes: a management layer 51, a storage layer 52, a computing layer 53, and an application layer 54. Among them:

[0104] The management layer 51 is used to build a Kubernetes basic environment based on cloud native, which is used to support the deployment of big data components.

[0105] The storage layer 52 is used to deploy a data storage layer based on the K8s basic environment. The data storage layer is used to manage data storage.

[0106] The computing layer 53 is used to deploy a data computing layer based on the data storage layer, and the data computing layer is used to manage the computing of data;

[0107] The application layer 54 is used to deploy a data application layer based on the data computing layer, and the data application layer is used to manage data applications.

[0108] Figure 5 This is a schematic diagram of the architecture of the cloud-native storage and computing separation device of the present invention. Figure 5 As shown, in this embodiment, the management layer 51 refers to the entire K8s basic environment, including six parts: user management module, API service module, scheduling module, resource management module, agent module, and proxy network module.

[0109] Storage layer 52 is the core of the storage-computing separation architecture. It's based on the open-source JuiceFS and provides unified access management for multiple storage systems. The underlying layer supports minio, an object storage platform, for data storage. It's also compatible with the original Hadoop system, ensuring seamless integration of legacy environments with the new storage-computing separation architecture. Redis manages JuiceFS system metadata, ensuring the stability of the storage system.

[0110] The data in the computing layer mainly consists of two parts: big data and database. The big data part mainly includes the message cache component Pulsar that natively supports the cloud-native storage and computing separation architecture, the stream-batch computing component Flink, the real-time data warehouse core component Doris, the offline batch computing Spark, the offline data warehouse core query computing engine Trino, and the offline scheduling component Airflow. All components are deployed based on the K8s environment and natively support high availability, fault self-healing / transfer, dynamic expansion and other features. The database part includes the commonly used log retrieval component ElasticSearch, the commonly used MySQL data, the high-performance ClickHouse database, and the high-performance cache database Redis to meet different applications and business scenarios.

[0111] The application layer mainly includes data application module, data development module, data basic module and data governance module.

[0112] The beneficial effects of implementing the present invention are as follows: by deploying a big data storage and computing separation environment based on the K8s environment, it has the characteristics of dynamic expansion, automatic fault recovery / transfer, etc., which greatly reduces the manpower investment in operation and maintenance, and also fully improves the utilization rate of server computing power and storage, and better achieves the effect of reducing costs and increasing efficiency; K8s can be used to manage the iterative updates of various component versions, reduce version compatibility issues caused by environmental differences, and achieve rapid iteration of software versions, shorten the development cycle, and improve development efficiency.

[0113] Example 3

[0114] To solve the above technical problems, the embodiment of the present invention also provides a computer device. Figure 6 , Figure 6 This is a basic structural block diagram of the computer device in this embodiment.

[0115] The computer device 6 includes a memory 61, a processor 62, and a network interface 63 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 6 having components of a memory 61, a processor 62, and a network interface 63, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc.

[0116] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0117] The memory 61 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 61 may be an internal storage unit of the computer device 6, such as the hard disk or memory of the computer device 6. In other embodiments, the memory 61 may also be an external storage device of the computer device 6, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash memory card, etc. equipped on the computer device 6. Of course, the memory 61 may also include both the internal storage unit of the computer device 6 and its external storage device. In this embodiment, the memory 61 is generally used to store the operating system and various application software installed on the computer device 6, such as computer-readable instructions for the storage and computing separation method based on cloud native construction. In addition, the memory 61 can also be used to temporarily store various types of data that have been output or are about to be output.

[0118] In some embodiments, the processor 62 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 62 is generally used to control the overall operation of the computer device 6. In this embodiment, the processor 62 is used to execute computer-readable instructions stored in the memory 61 or process data, such as computer-readable instructions for executing the aforementioned storage and computing separation method based on cloud native construction.

[0119] The network interface 63 may include a wireless network interface or a wired network interface. The network interface 63 is generally used to establish a communication connection between the computer device 6 and other electronic devices.

[0120] The beneficial effects of implementing the present invention are as follows: by deploying a big data storage and computing separation environment based on the K8s environment, it has the characteristics of dynamic expansion, automatic fault recovery / transfer, etc., which greatly reduces the manpower investment in operation and maintenance, and also fully improves the utilization rate of server computing power and storage, and better achieves the effect of reducing costs and increasing efficiency; K8s can be used to manage the iterative updates of various component versions, reduce version compatibility issues caused by environmental differences, and achieve rapid iteration of software versions, shorten the development cycle, and improve development efficiency.

[0121] Example 4

[0122] The present invention also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the storage and computing separation method based on cloud native construction as described above.

[0123] The beneficial effects of implementing the present invention are as follows: by deploying a big data storage and computing separation environment based on the K8s environment, it has the characteristics of dynamic expansion, automatic fault recovery / transfer, etc., which greatly reduces the manpower investment in operation and maintenance, and also fully improves the utilization rate of server computing power and storage, and better achieves the effect of reducing costs and increasing efficiency; K8s can be used to manage the iterative updates of various component versions, reduce version compatibility issues caused by environmental differences, and achieve rapid iteration of software versions, shorten the development cycle, and improve development efficiency.

[0124] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better embodiment. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the various embodiment methods of the present invention.

[0125] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. The preferred embodiments of the present invention are given in the accompanying drawings, but they do not limit the patent scope of the present invention. The present invention can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present invention more thorough and comprehensive. Although the present invention has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present invention specification and drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present invention.

Claims

1. A storage and computing separation method based on cloud native construction, characterized in that: The steps include: Build a K8s basic environment based on cloud native, which is used to support the deployment of big data components; Based on the K8s basic environment, the storage components juiceFS and minio are deployed through the kubesphere application store to build a data storage layer, which is used to manage data storage; Deploy a data computing layer according to the data storage layer, wherein the data computing layer is used to manage the computing of data; According to the data computing layer, a data application layer is deployed, and the data application layer is used to manage the application of data.

2. The storage and computing separation method based on cloud native construction according to claim 1 is characterized in that: The steps of building a K8s basic environment based on cloud native and supporting the deployment of big data components include: Build user management module, API service module, scheduling module, resource management module, agent module and proxy network module.

3. The storage and computing separation method based on cloud native construction according to claim 1 is characterized in that: Based on the K8s basic environment, the storage components juiceFS and minio are deployed on the kubesphere application store to build a data storage layer. The steps of managing data storage in the data storage layer specifically include: Deploy the data storage layer based on the open source JuiceFS cloud-native distributed file system.

4. The storage and computing separation method based on cloud native construction according to claim 3 is characterized in that: The step of deploying a data computing layer according to the data storage layer, wherein the data computing layer is used to manage the data computing specifically includes: Build a data computing layer based on flink and spark; Based on trino's multi-data source and query computing engine, it can meet the business scenarios of data warehouse computing, data analysis, large data volume aggregation, and multi-data source interactive computing OLAP type.

5. The storage and computing separation method based on cloud native construction according to claim 1 is characterized in that: The step of deploying a data application layer according to the data computing layer, wherein the data application layer is used to manage the application of data specifically includes: Deploy data application module, data development module, data foundation module, and data governance module.

6. The storage and computing separation method based on cloud native construction according to any one of claims 1 to 5, characterized in that: Based on the K8s basic environment, the storage components juiceFS and minio are deployed on the kubesphere application store to build a data storage layer. The steps of managing data storage in the data storage layer also include: Enable the application store function of the K8s cluster; Log in to the management node of the K8s cluster and find the platform management, cluster management, and custom resource definition pages in turn; Modify the openpitrix.store property of the yaml in ClusterConfiguration, adjust the configuration, download it to the master node of the K8s cluster, and update the cluster configuration by specifying the configuration file through kubectl apply.

7. A storage and computing separation device based on cloud native construction, characterized in that: include: The management layer is used to build a K8s basic environment based on cloud native. The K8s basic environment is used to support the deployment of big data components. The storage layer is used to build a data storage layer based on the K8s basic environment and deploy storage components juiceFS and minio through the kubesphere application store. The data storage layer is used to manage data storage. A computing layer, which is used to deploy a data computing layer according to the data storage layer, and the data computing layer is used to manage the computing of data; The application layer is used to deploy a data application layer according to the data computing layer, and the data application layer is used to manage the application of data.

8. The storage and computing separation device based on cloud native construction according to claim 7 is characterized in that: The management layer includes: User management module, API service module, scheduling module, resource management module, agent module and proxy network module.

9. A computer device comprising a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the storage and computing separation method based on cloud native construction as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the storage-computation separation method based on cloud-native construction as described in any one of claims 1 to 6.

Images