Big data analysis system and intelligent embodied platform

Abstract

The embodiment of the application provides a big data analysis system, and relates to the technical field of big data. The system comprises: a container meta management platform, which is used for constructing and managing a plurality of container cloud clusters, creating a tenant corresponding to each user to configure each tenant to a required container cloud cluster; a plurality of big data clusters, which are used for calculating submitted corresponding data query tasks; a multi-tenant platform, a service layer of which is used for transmitting a data query task to a computing engine layer in the case that the data query task submitted by a user is acquired; the computing engine layer is used for submitting the data query task to a corresponding big data cluster for calculation; a distributed file system, which is used for storing read-write permissions of each tenant to establish a connection with a corresponding object storage cluster according to read-write permissions of a tenant corresponding to a data query task; and an object storage system, which comprises a plurality of object storage clusters to store read-write data required by a corresponding tenant.

Description

Technical Field

[0001] This application relates to the field of big data technology, specifically to a big data analysis system and an intelligent embodied platform. Background Technology

[0002] With the continuous development and widespread application of data science and big data technologies, big data technology has permeated various industries and sectors to varying degrees. Many enterprises and institutions have built corresponding big data systems through various means to process the massive amounts of data generated in daily operations, management, and production activities, to mine the potential value of the data, and to further promote the development of related businesses. Its in-depth application not only helps enterprise operations but also contributes to the development of the national economy. It is also of great significance in promoting innovation in the information industry, addressing the challenges of big data storage and management, and changing the face of economic and social management. The application of big data technology can not only improve efficiency, reduce costs, optimize management, and enhance innovation, but also improve people's livelihoods, promote social governance, safeguard national security, and drive economic transformation.

[0003] Current big data analytics systems are primarily built using a single-tenant approach, where a unified big data cluster is constructed, storing and processing data from various departments and business units uniformly. The drawback is the inability to guarantee proper data isolation and effectively avoid competition for computing, I / O, and network resources. Alternatively, each department can build its own independent big data cluster for analyzing and processing its own data. This approach suffers from data silos, hindering data fusion and analysis, preventing service sharing, and resulting in redundant system construction and resource waste. Other solutions achieve multi-tenancy through system design and code development, but these still fail to address the end-to-end multi-tenancy issues in the big data analytics system construction process, including computing and storage, and remain flawed. Therefore, there is an urgent need for a big data system that can simultaneously provide services to multiple tenants while ensuring the independence of each tenant through various means (data isolation, computing resource isolation, cluster management, monitoring, and maintenance). Summary of the Invention

[0004] The purpose of this application is to provide a big data analysis system and intelligent embodied platform to solve the technical defects in the prior art that the inability to properly isolate tenant data during big data analysis leads to resource competition among tenants and mutual interference between computing tasks.

[0005] To achieve the above objectives, the first aspect of this application provides a big data analysis system, which includes:

[0006] The container meta management platform is used to build and manage multiple container cloud clusters and to create tenants corresponding to each user to configure each tenant to the required container cloud cluster. Each container cloud cluster in the multiple container cloud clusters is used to provide containers for the big data cluster required by each configured tenant to manage the big data cluster.

[0007] Multiple big data clusters, each of which is used to perform calculations for the corresponding data query tasks submitted by the tenant;

[0008] The multi-tenant platform includes a service layer and a computing engine layer. The service layer provides query interface services to establish a connection with the computing engine layer to transmit the data query tasks to the computing engine layer when the user submits a data query task through the query interface service. The computing engine layer is used to create a data reading interface corresponding to the data query task when it receives the data query task transmitted by the service layer, so as to submit the data query task to the corresponding big data cluster for computing through the data reading interface.

[0009] The distributed file system connects to both the computing engine layer and the object storage system. It stores each tenant's read and write permissions for the object storage system, so that when a data query task is submitted at the computing engine layer, a connection is established with the corresponding object storage cluster based on the read and write permissions of the tenant corresponding to the data query task.

[0010] An object storage system comprises multiple object storage clusters, each of which stores the read and write data required by its corresponding tenant.

[0011] In embodiments of this application, the container meta management platform is further configured to: obtain a first image file of the container meta management platform and a second image file of each of the multiple container cloud clusters; add tags to the first image file and the second image file to push the tagged first image file and the second image file to the image repository, wherein the tags include the address of the image repository and the name of the repository used to store the image file in the image repository; and pull the first image file and the second image file from the image repository in response to a run command to run the container meta management platform according to the first image file and the second image file.

[0012] In the embodiments of this application, for each required container cloud cluster, the container meta management platform is further configured to: obtain container cloud cluster construction instructions to create a container cloud cluster according to the container cloud cluster construction instructions; determine the master node server and child node servers of the container cloud cluster; obtain the configuration information of the container cloud cluster, wherein the configuration information includes a key-value pair database, management nodes and worker nodes; generate privileged mode startup instructions based on the configuration information to push the privileged mode startup instructions to the master node server and child node servers, so that the master node server and child node servers respond to the privileged mode startup instructions to grant permissions to the container cloud cluster.

[0013] In the embodiments of this application, for each required big data cluster, the container meta management platform is further configured to: construct a third image file of the big data cluster, wherein the third image file includes client dependency packages of the distributed file system and configuration information of the distributed file system; and deploy the big data cluster in the corresponding container cloud cluster according to the third image file.

[0014] In the embodiments of this application, the container meta management platform is further configured to: construct a fourth image file of the multi-tenant platform based on the third image file; define a configuration file for the multi-tenant platform, and deploy the configuration file in the container cloud cluster where the big data cluster resides as a configmap resource, wherein the configuration file includes environment variables, service ports, operating modes, the name of the third image file, and resource configuration information of the big data cluster required for the operation of the multi-tenant platform; and deploy the fourth image file in the container cloud cluster where the big data cluster resides as a deployment resource.

[0015] In embodiments of this application, the container meta management platform is further configured to: obtain tenant creation instructions to construct a tenant corresponding to each user according to the tenant creation instructions; create a project role for each user's tenant according to each user's identity information; and determine the container cloud cluster required for each project role in each tenant to add each project role to the required container cloud cluster.

[0016] In the embodiments of this application, for each required container cloud cluster, the container meta management platform is further configured to: create multiple projects in each container cloud cluster; for each of the multiple projects, create an application and allocate resource requirements for each project; and add each user to the required project.

[0017] In the embodiments of this application, each of the multiple big data clusters includes: a driver service, which is used to parse the data query tasks submitted by the engine layer according to the logical execution plan, the physical execution plan and the DAG directed acyclic graph; and an executor service, which is used to perform calculations on the parsed data query tasks.

[0018] In the embodiments of this application, the multi-tenant platform is further configured to: register all engine layer instances of the multi-tenant platform to ZooKeeper, wherein the directory label of each engine layer instance in all engine layer instances includes the label information of the corresponding tenant; when the multi-tenant platform obtains the data query task submitted by the user through the query interface service, it matches the corresponding engine layer instance from ZooKeeper according to the label information of the user's corresponding tenant, so as to submit the data query task to the corresponding big data cluster for calculation according to the corresponding engine layer instance.

[0019] In embodiments of this application, the object storage system further includes: a cluster controller, used to create multiple object storage clusters and manage read and write permissions between tenants and each of the multiple object storage clusters.

[0020] The above technical solution provides a big data analytics system comprising: a container meta-management platform for building and managing multiple container cloud clusters, and for creating tenants corresponding to each user to configure each tenant to the required container cloud cluster; each container cloud cluster providing containers for the big data clusters required by each configured tenant for managing the big data clusters; multiple big data clusters, each big data cluster performing computation on corresponding data query tasks submitted by tenants; and a multi-tenant platform including a service layer and a computing engine layer, wherein the service layer provides query interface services to establish a connection with the computing engine layer when user-submitted data query tasks are obtained through the query interface services. Data query tasks are transmitted to the computing engine layer. Upon receiving a data query task from the service layer, the computing engine layer creates a data read interface corresponding to the task, submitting it to the corresponding big data cluster for computation. A distributed file system, connected to both the computing engine layer and the object storage system, stores each tenant's read / write permissions for the object storage system. When a data query task is submitted from the computing engine layer, a connection is established with the corresponding object storage cluster based on the tenant's read / write permissions. The object storage system comprises multiple object storage clusters, each storing the read / write data required by its respective tenant. This implements a multi-tenant architecture for the big data storage system, achieving isolation of permissions and resources across the entire multi-tenant chain and service sharing at different levels. It effectively reduces the cost of big data analysis and maintenance under multi-tenancy, ensuring the stability and reliability of big data computing tasks while also guaranteeing data security.

[0021] The second aspect of this application provides an intelligent embody platform, which is equipped with the aforementioned big data analysis system.

[0022] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0023] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0024] Figure 1 This schematic diagram illustrates a structural block diagram of a big data analysis system according to an embodiment of this application;

[0025] Figure 2 This illustration schematically shows an architecture diagram illustrating the relationship between big data clusters of different tenants according to an embodiment of this application;

[0026] Figure 3 This illustration schematically shows an architecture diagram of Kyubi, a multi-tenant platform according to an embodiment of this application;

[0027] Figure 4 A flowchart illustrating a multi-tenant implementation of a big data storage system according to an embodiment of this application is shown. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0029] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0030] Figure 1 A schematic diagram illustrating the structure of a big data analysis system according to an embodiment of this application is shown. Figure 1 As shown in the figure, this application provides a big data analysis system, which may include:

[0031] The container meta management platform 110 is used to build and manage multiple container cloud clusters 112, and to create tenants corresponding to each user to configure each tenant to the required container cloud cluster. Each container cloud cluster in the multiple container cloud clusters is used to provide containers for the big data cluster required by each configured tenant for the management of the big data cluster. There can be multiple container cloud clusters 112, such as 2, 3, 4 or more.

[0032] It's important to note that the container meta-management platform can refer to Rancher. Rancher is an open-source, enterprise-grade, full-stack container deployment and management platform primarily used for managing Docker containers and Kubernetes. Rancher makes deploying and managing containers in production environments easier and more efficient by providing an intuitive web interface. It offers multi-tenancy and role-based access control (RBAC), helping enterprises build and manage Kubernetes clusters. Essentially, it's a multi-tenant management platform for Kubernetes clusters, featuring flexible network and security configuration, visualized logging and monitoring, ease of use, and high security and reliability. Container cloud clusters can also refer to Kubernetes clusters. Kubernetes (K8S) is an open-source container orchestration platform used for automating the deployment, scaling, and management of containerized applications. A K8S cluster consists of a group of physical or virtual machines organized into a single pool of compute resources, running the Kubernetes platform. A K8S cluster can include a Master node and Worker nodes. The Master node is responsible for managing the overall cluster status and configuration information, including components such as the API server, controller manager, scheduler, and etcd. The API server provides the Kubernetes API interface, the controller manager monitors the cluster status and performs corresponding operations, the scheduler schedules container applications to available worker nodes, and etcd stores the cluster's status and configuration information. Worker nodes run container applications, including components such as kubelet, kube-proxy, and container runtimes (e.g., Docker). kubelet manages containers on the local node, and kube-proxy acts as a network proxy between nodes, ensuring that container applications can communicate with each other. In this technical solution, cloud-native technologies can be used to deploy big data-related systems in a Kubernetes cluster, indirectly enabling Rancher to manage multi-tenant big data systems. Rancher can easily and quickly build and manage multiple Kubernetes clusters, and indirectly manage multiple big data analytics systems. Rancher can provide unified management, monitoring, operation, and maintenance of multiple big data systems, as well as tenant permission and role management. It can also easily add new clusters and tenants, and flexibly expand existing big data clusters. Specifically, Rancher can create tenants corresponding to each user and configure each tenant to the required container cloud cluster according to user needs. Each of the multiple container cloud clusters provides containers for the big data cluster required by each configured tenant for the management of the big data cluster. For example, in this technical solution, such as Figure 2 The diagram shown illustrates the relationship architecture between big data clusters of different tenants. Figure 2 In this context, Kubernetes (K8S) refers to a container cloud cluster, responsible for resource scheduling of the computing and query engines. Spark Pods are offline computing engine containers, responsible for offline batch data computation. FlinkPods are real-time computing engine containers, responsible for real-time data computation. Dorisdb Pods are query engine containers, responsible for distributed storage and fast querying of analysis results. JuiceFS is a distributed file system, responsible for data encryption, read / write acceleration, data organization, and metadata storage. Minio is a distributed object storage system, responsible for data storage, storing data in the form of objects. Cluster refers to a cluster of data.

[0033] Specifically, Rancher creates two tenants, Cluster1 and Cluster2, and creates the necessary container cloud clusters K8S-1 and K8S-2 for Cluster1 and Cluster2 respectively. Cluster1 and Cluster2 are then configured onto K8S-1 and K8S-2 respectively. Next, the big data clusters (Spark, Flink, and Doris) required by tenants Cluster1 and Cluster2 are containerized and deployed onto container cloud clusters K8S-1 and K8S-2 respectively, enabling management of these big data clusters through K8S-1 and K8S-2. Through Rancher and K8S, multi-tenant management and construction of big data clusters are achieved, enabling the sharing of service systems such as monitoring and operation, identity authentication and authorization, and big data cluster deployment management, while also meeting the personalized needs of tenant permission and role management.

[0034] There are 120 big data clusters, and each big data cluster is used to perform calculations for the corresponding data query tasks submitted by the tenant.

[0035] It should be noted that multiple big data clusters can refer to, but are not limited to, clusters of big data computing components such as Spark, Flink, and Foris, to perform computations on the corresponding data query tasks submitted by tenants. Here, a data query task can refer to an SQL analysis task submitted by a tenant, to... Figure 2 For example, when tenant Cluster1 submits a Spark SQL analysis task, the Spark SQL analysis task is transmitted to the Spark big data cluster in the container cloud cluster K8S-1 for computation.

[0036] The multi-tenant platform 130 includes a service layer 132 and a computing engine layer 134. The service layer provides query interface services to establish a connection with the computing engine layer to transmit the data query tasks to the computing engine layer when the user submits a data query task through the query interface service. The computing engine layer creates a data reading interface corresponding to the data query task when it receives the data query task transmitted by the service layer, so as to submit the data query task to the corresponding big data cluster for computing through the data reading interface.

[0037] It should be noted that the multi-tenant platform can refer to Kyubi. Kyubi is a distributed multi-tenant ThriftJDBC / ODBC server used for large-scale data management, processing, and analysis. Kyubi provides serverless SQL, supports multiple computing engines, and enables simplified and secure access to any computing engine cluster resources through a unified gateway, allowing end users to deploy different workloads. Serverless SQL refers to the ability to run SQL queries without relying on any database server. Therefore, in this technical solution, to better submit SQL to the big data cluster and isolate computing resources, the unified multi-tenant SQL development framework Kyubi can be chosen to integrate with each big data cluster. In this technical solution, Kyubi can include a service layer (Kyuubi serve) and a computing engine layer (Kyuubi engine). The service layer (Kyuubi serve) provides a Hive JDBC query interface service to retrieve user-submitted SQL analysis tasks through Hive JDBC queries and establishes a connection with the computing engine layer (Kyuubi engine) to transfer the SQL analysis tasks to the computing engine layer (Kyuubi engine). After receiving an SQL analysis task from the Kyubi serve layer, the compute engine layer (Kyuubi engine) first creates a data read interface session corresponding to the SQL analysis task. This session is then used to submit the SQL analysis task to the corresponding big data cluster for computation. For example, after the Kyubi serve layer receives a Spark SQL analysis task submitted by tenant Cluster1 via the Hive JDBC query interface, it establishes a connection with the Kyubi engine layer to transmit the Spark SQL analysis task. The Kyubi engine layer then creates a Spark session to further transmit the Spark SQL analysis task to the Spark big data cluster in the Kubernetes container cluster (K8S-1) for computation.

[0038] The distributed file system 140 is connected to both the computing engine layer and the object storage system. It is used to store the read and write permissions of each tenant for the object storage system. When a data query task is submitted at the computing engine layer, a connection is established with the corresponding object storage cluster based on the read and write permissions of the tenant corresponding to the data query task.

[0039] It's important to note that the distributed file system may refer to JuiceFS. JuiceFS is a high-performance, POSIX-compliant, cloud-native distributed file system designed specifically for cloud environments. It uses object storage as its underlying storage and supports various metadata types. Under current technological conditions, because different tenants read, write, and create data / files within the same distributed storage cluster, they can access each other's data files, posing serious data security risks. Therefore, this technical solution selects the JuiceFS distributed file system as an acceleration layer. JuiceFS connects to the upper-layer computing engine layer and the underlying object storage system, storing each tenant's read and write permissions for the object storage system to uniformly provide data access and read / write services to different tenants. Thus, when the computing engine layer submits a data query task, JuiceFS can establish a connection with the corresponding object storage cluster based on the read and write permissions of different tenants. This achieves multi-tenancy of the object storage system, ensuring physical isolation and uniformity of storage resources for different tenants, avoiding data silos, and guaranteeing data security.

[0040] The object storage system 150 includes multiple object storage clusters, each of which stores the read and write data required by the corresponding tenant.

[0041] It's important to note that the object storage system referred to here is Minio. Minio is a high-performance, distributed object storage platform that provides high-performance, S3 protocol-compatible object storage, making it ideal for storing large volumes of unstructured data. As an object storage platform, Minio processes objects at the file level. This means that data storage is achieved simply by uploading various data files (electronic files) containing massive amounts of data to the Minio client as files. In this technical solution, to ensure data isolation between different tenants, Minio can provide multiple object storage clusters for different tenants to read and write data at the underlying storage layer. Simultaneously, to prevent users from directly perceiving that their data is stored in different storage systems and to avoid data silos, this technical solution utilizes the distributed file system JuiceFS to uniformly provide data access and read / write services to different tenants. Different tenants uniformly interface with JuiceFS, and JuiceFS reads and writes data in different Minio clusters according to the different tenants' permissions. This achieves multi-tenancy of the storage system, ensuring physical isolation and uniformity of storage resources for different tenants, avoiding data silos, and ensuring data security.

[0042] The above technical solution provides a big data analytics system comprising: a container meta-management platform for building and managing multiple container cloud clusters, and for creating tenants corresponding to each user to configure each tenant to the required container cloud cluster; each container cloud cluster providing containers for the big data clusters required by each configured tenant for managing the big data clusters; multiple big data clusters, each big data cluster performing computation on corresponding data query tasks submitted by tenants; and a multi-tenant platform including a service layer and a computing engine layer, wherein the service layer provides query interface services to establish a connection with the computing engine layer when user-submitted data query tasks are obtained through the query interface services. Data query tasks are transmitted to the computing engine layer. Upon receiving a data query task from the service layer, the computing engine layer creates a data read interface corresponding to the task, submitting it to the corresponding big data cluster for computation. A distributed file system, connected to both the computing engine layer and the object storage system, stores each tenant's read / write permissions for the object storage system. When a data query task is submitted from the computing engine layer, a connection is established with the corresponding object storage cluster based on the tenant's read / write permissions. The object storage system comprises multiple object storage clusters, each storing the read / write data required by its respective tenant. This implements a multi-tenant architecture for the big data storage system, achieving isolation of permissions and resources across the entire multi-tenant chain and service sharing at different levels. It effectively reduces the cost of big data analysis and maintenance under multi-tenancy, ensuring the stability and reliability of big data computing tasks while also guaranteeing data security.

[0043] In this embodiment, the container meta management platform is further configured to: obtain a first image file of the container meta management platform and a second image file of each of the multiple container cloud clusters; add tags to the first image file and the second image file to push the tagged first image file and the second image file to the image repository, wherein the tags include the address of the image repository and the name of the repository used to store the image file in the image repository; and pull the first image file and the second image file from the image repository in response to the run command to run the container meta management platform according to the first image file and the second image file.

[0044] In this embodiment, it should be noted that before running the container meta-management platform Rancher, Rancher needs to be successfully deployed to the Linux servers where all services reside. Specifically, it is assumed that Docker containers and the private container image repository Harbor are already installed on the Linux servers where all services reside. To successfully start the Rancher service in Docker, firstly, the following command needs to be executed on a Linux server: `docker pull rancher / rancher:stable` to download the first Rancher image file from the network. Then, the following command is executed: `docker tag rancher / rancher:stable ${ip}:${port} / ${repository} / rancher / rancher:v2.5.3` to tag the first image file, where `${ip}:${port}` is the address of the Harbor image repository, and `${repository}` is the name of the repository in the Harbor image repository used to store the image file. After tagging the first image file, you can execute the following command: `docker push ${ip}:${port} / ${repository} / rancher / rancher:v2.5.3` to push the tagged first image to the image repository Harbor, which can then be reused. Similarly, following the same steps, you can tag each container cloud cluster Kubernetes-related second image file (including calico-cni, calico-kube-controllers, calico-node, cluster-proportional-autoscaler, and coredns-coredns, etc.) and push them to the image repository Harbor for subsequent operations.

[0045] Next, you can start the Rancher service in the Docker container using the following command:

[0046] docker run --name rancher -d --privileged --restart=unless-stopped-v / usr / local / rancher: / usr / local / rancher -p 880:80 -p 8443:443 \

[0047] -e CATTLE_SYSTEM_DEFAULT_REGISTRY=1 ${ip}:${port} \

[0048] -e CATTLE_SYSTEM_CATALOG=bundled \

[0049] ${ip}:${port} / ${repository} / rancher / rancher:v2.5.3 \

[0050] / bin / bash

[0051] Where ${ip}:${port} is the access address of the image repository Harbor, and ${ip}:${port} / ${repository} / rancher / rancher:v2.5.3 is the address of the first image file in the image repository Harbor. After the above command executes successfully, you can further view the Rancher startup logs using the following command: `docker logs -f rancher`. If there are no error messages in the logs, it means that the Rancher service has started successfully. Finally, you can access the Rancher web management interface (http: / / ${ip}:8443 / rancher) through a browser, where ${ip} is the IP address of the server where Rancher is located. Enter the username and password to see the specific content of the Rancher management interface.

[0052] In the application embodiments, for each required container cloud cluster, the container meta management platform is further configured to: obtain container cloud cluster build instructions to create a container cloud cluster according to the container cloud cluster build instructions; determine the master node server and child node servers of the container cloud cluster; obtain the configuration information of the container cloud cluster, wherein the configuration information includes a key-value pair database, management nodes and worker nodes; generate privileged mode startup instructions based on the configuration information, and push the privileged mode startup instructions to the master node server and child node servers so that the master node server and child node servers respond to the privileged mode startup instructions to grant permissions to the container cloud cluster.

[0053] In this embodiment, it should be noted that after successfully deploying the container meta-management platform Rancher to the Linux servers where all services reside, a container cloud cluster Kubernetes (K8S) can be further built using Rancher. Specifically, users can access the Rancher service interface and select the "Add Custom K8S Cluster" button to trigger the container cloud cluster build command. Rancher can then create the K8S container cloud cluster based on the triggered command. Following the prompts, a name is chosen for the cluster to be created, such as Cluster1. Since a K8S cluster includes a master node and worker nodes, and is a combination of multiple servers, the service role of the K8S cluster to be deployed needs to be specified for each server. Therefore, the services to be installed on the master node are selected. In the host options, the roles of Etcd (key-value database), control plane (management node), and worker (worker node) can be selected sequentially. All other configurations are left at their default settings to obtain the configuration information for the container cloud K8S cluster. The Docker container can generate the following command based on the configuration information: `sudo docker run -d --privileged --restart=unless-stopped --net=host -v / etc / kubernetes: / etc / kubernetes -v / var / run: / var / runrancher / rancher-agent:v2.5.3 --server https: / / 192.168.241.138 --token 4692pw82n2jhm7qmtzz7dptlpvhwz7ls9f27v2mkv5n9s89mwp6684 --ca-checksum 9cfb504b8d3f05bd9e37477173524ccf1a2b24129ffc941f52dcc2e9c9175310 --node-name K8S-master01 --controlplane` The command `--worker` executes the above command on the K8S cluster master node server. After execution, you can see the K8S cluster with only one master node through the Rancher web interface.Then select the cluster, edit the cluster, select "Add Child Node," and choose a service role for the child node. Here, only the Worker role is selected for the child node. Rancher will then generate the following command: `sudo docker run -d --privileged --restart=unless-stopped --net=host -v / etc / kubernetes: / etc / kubernetes -v / var / run: / var / runrancher / rancher-agent:v2.5.3 --server https: / / 192.168.241.138 --token 4tldjd8szx8ndgk9k88hmscdx46q2k8lf6l9z7kg5mbjgmwpb452nd --ca-checksum 9cfb504b8d3f05bd9e37477173524ccf1a2b24129ffc941f52dcc2e9c9175310 --node-name` After executing the command `K8S-worker-node01 -- worker` on the Linux server where the corresponding child node resides, you will see through the Rancher web interface that the child node has been added to the aforementioned Kubernetes cluster. This process is repeated for each desired cluster size, until all machines are eventually added to the Kubernetes cluster `Cluster1`. Thus, a multi-node Kubernetes cluster `Cluster1` has been created.

[0054] Similarly, if you want to create other clusters such as Cluster2, Cluster3, etc., follow the same steps: select "Custom Cluster" in the Rancher web interface, and then repeat the deployment operations and command execution process for the master and child nodes. This will result in a container cloud Kubernetes cluster named Cluster2 and a container cloud Kubernetes cluster named Cluster3. This process allows you to easily build multiple independent container cloud Kubernetes clusters using Rancher, while simultaneously managing all of these clusters under Rancher's unified control, including unified cluster monitoring, management of roles and permissions for different tenants, and unified operation and maintenance management.

[0055] In this embodiment of the application, for each required big data cluster, the container meta management platform is further configured to: construct a third image file of the big data cluster, wherein the third image file includes client dependency packages of the distributed file system and configuration information of the distributed file system; and deploy the big data cluster in the corresponding container cloud cluster according to the third image file.

[0056] In this embodiment, it should be noted that after building the container cloud cluster Kubernetes (K8S) through the container meta-management platform Rancher, it is further necessary to configure the big data cluster required by each tenant in the K8S container cloud cluster configured for each tenant. Specifically, the big data cluster can include, but is not limited to, Spark, Flink, and Foris. In this technical solution, taking Spark as the big data cluster as an example, firstly, according to the K8S container image file creation rules, a third image file for Spark to run in the container is created. The conf directory of the Spark image file should contain the JuiceFS client dependency package juicefs-hadoop-1.1.0.jar. In this way, when Spark executes SQL, it can find the hive-metastore service based on this configuration information, obtain metadata information, and then find the corresponding data in JuiceFS and perform calculations. After the third image file of Spark is built, the big data cluster Spark can be containerized and deployed in the corresponding K8S container cloud cluster based on the third image file.

[0057] In this embodiment, the container meta management platform is further configured to: construct a fourth image file for the multi-tenant platform based on the third image file; define a configuration file for the multi-tenant platform, and deploy the configuration file as a configmap resource in the container cloud cluster where the big data cluster resides, wherein the configuration file includes environment variables, service ports, operating modes, the name of the third image file, and resource configuration information of the big data cluster required for the operation of the multi-tenant platform; and deploy the fourth image file as a deployment resource in the container cloud cluster where the big data cluster resides.

[0058] In this embodiment, it should be noted that, to better submit SQL analysis data to the big data cluster and isolate computing resources, the unified multi-tenant SQL development framework Kyubi can be integrated with the big data cluster. In this technical solution, taking Spark as an example, the Kyubi image file depends on the Spark image file. Therefore, when building the fourth image file in the Kyubi Kubernetes container, it must be built based on the third Spark image file. After the fourth image file is created, the Kyubi configuration file is modified to specify the environment variables (JAVA_HOME), the Kyubi service port, the Kyubi running mode, the name of the Spark image file, and the Spark resource configuration information required for Kyubi to run. Then, this configuration file is containerized and deployed in the Kubernetes container cloud cluster where Spark resides as a configmap resource. The configmap resource format refers to key-value pairs, mainly used to store non-confidential configuration information. Configmaps can decouple configuration information from the container image, allowing the configuration to be managed and updated independently of the application, thus facilitating configuration updates without rebuilding the image.

[0059] Next, the fourth image file of Kyuubi can be further deployed as a deployment resource in the Kubernetes (K8S) container cloud cluster where Spark resides. The deployment resource form is the controller for the most common stateless applications, supporting operations such as scaling up and down, and rolling updates. Similarly, the deployment steps for big data clusters Flink and Foris are the same.

[0060] In this embodiment of the application, each of the multiple big data clusters includes: a driver service, which is used to parse the data query tasks submitted by the engine layer according to the logical execution plan, the physical execution plan and the DAG directed acyclic graph; and an executor service, which is used to perform calculations on the parsed data query tasks.

[0061] In this embodiment, it should be noted that the Kyubi-server service automatically loads the configuration file in the aforementioned configmap during deployment. After the Kyubi-server container starts, SQL analysis tasks can be submitted through the Hive JDBC port provided by Kyubi-server. At this time, Kyubi-server will automatically start a containerized big data cluster in the Kubernetes cluster according to the Kyubi configuration file. In this technical solution, taking a Spark cluster as an example, the Spark cluster consists of two services: driver and executor. The driver is responsible for parsing the Spark SQL analysis tasks passed from Kyubi-server in the order of logical execution plan, physical execution plan, and DAG (Directed Acyclic Graph), and finally submitting them to the executor service for specific execution and computation. Therefore, in a Spark containerized cluster, there are two types of containers: driver pods and executor pods. To avoid repeatedly starting the Spark cluster, Kyubi's Kyubi-engine service maintains the Spark cluster's session information. When a new SQL analysis task is submitted through Kyubi-server, it first searches for a suitable Spark cluster in the Spark sessions maintained by Kyubi-engine to execute the SQL task. If no suitable cluster is found, Kyubi will then start a new session, i.e., create a new Spark cluster, and then execute the corresponding SQL computation task. Similarly, the operating mode of Flink and Trino clusters is similar to that of Spark, and these steps will not be described again here.

[0062] In this embodiment of the application, the container meta management platform is further configured to: obtain tenant creation instructions to construct a tenant corresponding to each user according to the tenant creation instructions; create a project role for each user's tenant according to each user's identity information; determine the container cloud cluster required for each project role in each tenant, so as to add each project role to the required container cloud cluster.

[0063] In this embodiment, it should be noted that the above steps have basically achieved the installation and deployment of multiple big data clusters. At this point, multiple tenants (such as U1, U2, U3, etc.) can be created through the Rancher service's web interface, along with multiple different roles (such as Create Projects, Manage Cluster Backups, Manage Cluster Catalogs, Manage Cluster Members, Manage Nodes, Manage Storage, View AllProjects, View Cluster Catalogs, View Cluster Members, and View Nodes). Then, for each Kubernetes cluster, the tenants that need to use the cluster are selected, and project roles are created for each user's corresponding tenant based on each user's identity information. Each project role corresponds to different resource and cluster operation permissions. The same tenant can also be added to different Kubernetes clusters, thus achieving coarse-grained resource isolation and sharing (different container cloud clusters) between tenants.

[0064] In this embodiment of the application, for each required container cloud cluster, the container meta management platform is further configured to: create multiple projects in each container cloud cluster; for each project in the multiple projects, create an application and divide the resource requirement scope for each project; and add each user to the required project respectively.

[0065] In this embodiment, it should be noted that this technical solution can further achieve finer-grained resource isolation. Specifically, for each Kubernetes cluster, multiple projects (such as Project1, Project2, Project3, etc.) can be created within that cluster. Within each project, different applications can be created and their specific resource requirements (such as CPU, memory, etc.) can be allocated. Entering a specific project, such as Project1, tenants can be added to that project, and roles with different permissions can be selected for each tenant. This allows each tenant to work independently in one or more projects, thereby achieving project-level resource isolation and sharing between tenants.

[0066] It should be noted that, in addition to project-level resource isolation, this technical solution can further partition and isolate resources. Different namespaces (n1, n2, n3…) can be created under a project, and different resource allocations (such as CPU and memory) can be assigned to different namespaces, enabling finer-grained resource management for different tenants. Through these innovative operations, resource isolation and sharing between different tenants are achieved. Furthermore, since all tenants' Kubernetes clusters are monitored uniformly by Rancher, Rancher can provide unified operation and real-time monitoring of various aspects of different Kubernetes clusters, including operational status, service status, resource usage, network I / O, disk I / O, and security. Therefore, all tenants can share a unified operation and monitoring service, reducing the complexity of redundant deployment of operation and monitoring systems, lowering costs, and maximizing service sharing and reuse.

[0067] In this embodiment, the multi-tenant platform is further configured to: register all engine layer instances of the multi-tenant platform to ZooKeeper, wherein the directory label of each engine layer instance includes the label information of the corresponding tenant; when the multi-tenant platform obtains the data query task submitted by the user through the query interface service, it matches the corresponding engine layer instance from ZooKeeper according to the label information of the user's corresponding tenant, so as to submit the data query task to the corresponding big data cluster for calculation according to the corresponding engine layer instance.

[0068] In this embodiment, it should be noted that while the above achieves resource isolation and service sharing at the cluster, project, and application levels among different tenants, in actual big data development, there are often situations where a large number of big data computing tasks queue up and compete for resources within the same big data cluster. Therefore, this solution can choose the open-source tool Kyubi as a unified gateway for submitting computing tasks. Taking Spark as an example, Kyubi effectively encapsulates the Spark Thrift server (a client for submitting computing tasks to Spark), making it easier for tenants to submit Spark-SQL distributed computing tasks to the Kubernetes container. The architecture diagram of the multi-tenant platform Kyubi is shown below. Figure 3 As shown in the diagram, Kyuubi mainly consists of the Kyuubi server service layer and the Kyuubi engine computing engine layer. Kyuubi can also include a service discovery side layer, which can choose ZooKeeper to act as the service discovery and election layer. Specifically... Figure 3In this context, User Side refers to the user layer, Service Discovery Side refers to the service discovery layer, KyuubiServer Side refers to the unified SQL gateway service layer, Kyuubi Engine Side refers to the unified SQL gateway engine layer, Kyuubi Server refers to the unified SQL gateway server, Kyuubi Server Instance refers to the unified SQL gateway server instance, SparkSQLEngine refers to the offline computing SparkSQL engine, SparkSQLEngine Instance refers to the offline computing SparkSQL engine instance, SparkSession refers to the offline computing Spark session, Zookeeper Namespace refers to the namespace, Kyuubi Server Namespace refers to the unified SQL gateway server namespace, and Register refers to registration.

[0069] When the Kyubi service starts, multiple Kyubi server instances are launched simultaneously. These Kyubi server instances are registered in ZooKeeper. When a tenant submits a Spark SQL task through the Hive-JDBC client provided by the Kyubi server, the Kyubi server automatically starts a Kyubi engine service. The Kyubi engine service automatically creates a Spark session, which is responsible for submitting the Spark SQL task passed from the Kyubi server to the distributed Spark computing cluster for computation. The Kyubi engine instance is also registered in ZooKeeper. Through refactoring of the Kyubi code, tenants can selectively choose specific Kyubi engine instances from ZooKeeper to submit Spark SQL analysis tasks. Specifically, when the Kyuubi engine registers with ZooKeeper, the directory tag of the registered instance includes the tenant's tag information. This allows any Spark SQL task submitted by the same tenant or user to be retrieved from ZooKeeper based on the user's tag information, and then submitted to the corresponding Spark cluster. This means that computational tasks submitted by different tenants can be submitted to different Spark clusters for computation, while tasks submitted by the same tenant at any time will run within the same Spark cluster. This innovative design achieves resource isolation between computational tasks of different tenants, avoiding resource contention, while simultaneously ensuring maximum service sharing among different tenants. For example, all tenants can submit, monitor, and maintain computational tasks through the same SQL gateway, Kyuubi, and can share the same container cloud platform and its associated services, greatly reducing operation and monitoring costs while ensuring service stability and reliability. Figure 3 In this context, "request" means to request, "find" means to search, and "register" means to register.

[0070] In this embodiment of the application, the object storage system further includes: a cluster controller, used to create multiple object storage clusters and manage read and write permissions between tenants and each of the multiple object storage clusters.

[0071] In this embodiment, it should be noted that the cluster controller can refer to the Minio operator. The Minio operator is an automation tool for Kubernetes clusters, specifically designed for deploying, managing, and scaling Minio instances. By extending the Kubernetes API, the Minio operator makes deploying Minio instances on public and private clouds more convenient. In this technical solution, the above steps have achieved resource isolation, permission isolation, service autonomy, and service sharing across different dimensions such as projects, container clusters, applications, and computing tasks in a multi-tenant big data analytics scenario. To achieve multi-tenant resource, permission, and computing isolation across the entire big data system chain, this step utilizes innovative solutions and process design to achieve multi-tenancy in the final critical link – storage. Specifically, the open-source object storage system Minio can be chosen as the underlying storage for the big data system. Minio object storage has advantages such as high throughput, low storage cost, and good system stability, but it suffers from poor data read performance. To address this, the JuiceFS distributed file system can be chosen as an acceleration layer, connecting the upper-layer computing engine layer (Kyuubi engine) and the underlying storage system Minio. Figure 4 The diagram shows a flowchart of a multi-tenant implementation for a big data storage system. Among them, Figure 4 In these contexts, Minio Administrator refers to the administrator of the distributed object storage system. Juicefs refers to the distributed file system. Minio Console refers to the distributed object storage console. Minio Object Storage refers to distributed object storage. Minio Kubernetes Operator refers to the distributed object storage container processor. Kubernetes refers to the container cloud platform. RACK refers to a computer rack.

[0072] To achieve data isolation between different tenants, their data needs to be stored in different object storage clusters. Therefore, deploying Minio on a container cloud platform is a viable option. For rapid deployment of Minio clusters, the Minio operator can be used to control cluster creation and manage cluster users and permissions. Leveraging cloud-native technologies, the Minio operator can easily create different object storage systems for different tenants and manage access permissions for different Minio integrations, achieving data isolation at the underlying storage layer. Furthermore, to prevent users from directly perceiving that their data is stored in different storage systems and to avoid data silos, this solution utilizes the distributed file system JuiceFS to uniformly provide data access and read / write services to different tenants. Different tenants uniformly interface with JuiceFS, which reads and writes data in different Minio clusters based on the tenant's permissions. This achieves multi-tenancy of the storage system, ensuring physical isolation and uniformity of storage resources for different tenants, avoiding data silos, and guaranteeing data security.

[0073] This technical solution, through the aforementioned innovative process design and technological innovations, realizes a multi-tenant architecture across the entire big data processing system, effectively achieving resource isolation at each level of the big data system. By managing multiple Kubernetes container cloud clusters through Rancher, resource isolation and access control are achieved at the cluster, project, and application levels for multiple tenants. Simultaneously, unified management, monitoring, and operation and maintenance across multiple tenants and clusters are achieved, maximizing service sharing at the cluster level. Through the reconstruction and innovative design of the unified SQL gateway Kyuubi, resource isolation and service sharing are realized between different tenants and between multiple computing tasks within the same tenant. Cloud-native technologies enable rapid containerized deployment and tenant access control for the underlying storage system, Minio. By introducing the JuiceFS distributed file system as an acceleration layer between the computing engine layer and the underlying object storage Minio, physical isolation and formal uniformity of data storage are achieved within the multi-tenant architecture, avoiding data silos and ensuring data security among multiple tenants. Through this innovative process design, a multi-tenant architecture for the big data storage system is successfully implemented, achieving isolation of permissions and resources across the entire multi-tenant big data system and service sharing at different levels. It effectively reduces the cost of big data analysis and operation and maintenance in multi-tenant environments, ensures the stability and reliability of big data computing tasks, and also guarantees data security.

[0074] This application provides an intelligent avatar platform, which is equipped with a big data analysis system.

[0075] In this embodiment, it should be noted that embodied intelligence typically refers to intelligent systems that possess physical entities and can interact with their environment through sensors and actuators, such as robots and autonomous vehicles. Embodied intelligence usually requires coordinating the computing resources of multiple devices. Kubernetes (K8S), as a container orchestration and cluster management technology, can provide embodied intelligence systems with efficient resource scheduling, distributed computing support, dynamic scalability, and fault-tolerant management, especially valuable in scenarios involving multi-device collaboration, edge computing, and real-time response. Specifically, embodied intelligence systems (such as robot swarms and autonomous vehicle fleets) typically need to coordinate the computing resources of multiple devices, such as sensor data processing, AI model inference, and path planning. In this case, the K8S scheduler can automatically allocate computationally intensive tasks (such as SLAM and 3D environment modeling) to idle nodes in the cluster (physical robots, edge servers, or the cloud). For example, in a group of collaborative warehouse robots, K8S can dynamically migrate the image recognition task of a certain robot to an edge server for processing based on the load of each robot, avoiding overloading local computing resources. Embodied intelligence systems need to handle dynamic loads (such as sudden task interruptions) and hardware failures (such as robot crashes). In these cases, Kubernetes health checks can automatically restart faulty Pods or migrate tasks to healthy nodes. For example, in a service robot cluster, if a robot goes offline due to hardware failure, Kubernetes can reschedule its tasks to other robots, ensuring the continuity of cleaning or inspection tasks. Embodied intelligence involves the security of sensitive data (such as user interaction records and environmental maps) and physical devices. In this case, restricting inter-Pod communication can prevent unauthorized data access, and secure container technology can be used to isolate critical tasks. Lieu, a patient data processing service for healthcare robots, runs in an independent namespace, allowing only authorized services access.

[0076] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0077] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A big data analytics system, characterized by, The big data analysis system comprises: a container meta management platform, configured to build and manage a plurality of container cloud clusters, and to create a tenant corresponding to each user to configure each tenant to a required container cloud cluster, each container cloud cluster in the plurality of container cloud clusters being configured to provide a container for each configured tenant required big data cluster to manage the big data cluster; a plurality of big data clusters, each big data cluster in the plurality of big data clusters being configured to perform computation for a corresponding data query task submitted by a tenant; a multi-tenant platform, comprising a service layer and a computing engine layer, the service layer being configured to provide a query interface service to establish a connection with the computing engine layer to transmit a data query task submitted by a user to the computing engine layer in a case where the data query task is acquired through the query interface service; and the computing engine layer being configured to create a data reading interface corresponding to the data query task to submit the data query task to a corresponding big data cluster for computation through the data reading interface in a case where the data query task transmitted by the service layer is received; a distributed file system connected with the computing engine layer and an object storage system respectively, configured to store read-write permissions of each tenant for the object storage system, to establish a connection with a corresponding object storage cluster according to read-write permissions of a tenant corresponding to the data query task in a case where the data query task is submitted to the computing engine layer; the object storage system, comprising a plurality of object storage clusters, each object storage cluster in the plurality of object storage clusters being configured to store read-write data required by a corresponding tenant; wherein the multi-tenant platform is further configured to: register all engine layer instances of the multi-tenant platform to a zookeeper, wherein a directory label of each engine layer instance in the all engine layer instances comprises label information of a corresponding tenant; in a case where a data query task submitted by a user is acquired by the multi-tenant platform through the query interface service, match a corresponding engine layer instance from the zookeeper according to label information of a tenant corresponding to the user, and submit the data query task to a corresponding big data cluster for computation according to the corresponding engine layer instance.

2. The big data analytics system of claim 1, wherein, The container meta management platform is further configured to: acquire a first image file of the container meta management platform and a second image file of each container cloud cluster in the plurality of container cloud clusters; add a label to the first image file and the second image file to push the first image file and the second image file after the label is added to an image warehouse, wherein the label comprises an address of the image warehouse and a warehouse name in the image warehouse for storing image files; pull the first image file and the second image file from the image warehouse in response to a running instruction to run the container meta management platform according to the first image file and the second image file.

3. The big data analytics system of claim 1, wherein, For each required container cloud cluster, the container meta management platform is further configured to: acquire a container cloud cluster construction instruction to create a container cloud cluster according to the container cloud cluster construction instruction; determining a master node server and a sub-node server of the container cloud cluster; obtaining configuration information of the container cloud cluster, wherein the configuration information comprises a key-value pair database, a management node, and a worker node; generating a privileged mode starting instruction based on the configuration information, and pushing the privileged mode starting instruction to the master node server and the sub-node server, so that the master node server and the sub-node server respond to the privileged mode starting instruction to authorize the container cloud cluster.

4. The big data analytics system of claim 1, wherein, For each required big data cluster, the container meta management platform is further configured to: construct a third image file of the big data cluster, wherein the third image file comprises a client dependency package of the distributed file system and configuration information of the distributed file system; containerize and deploy the big data cluster in a corresponding container cloud cluster according to the third image file.

5. The big data analytics system of claim 4, wherein, The container meta management platform is further configured to: construct a fourth image file of the multi-tenant platform according to the third image file; define a configuration file of the multi-tenant platform, and containerize and deploy the configuration file in the container cloud cluster where the big data cluster is located in the form of a configmap resource, wherein the configuration file comprises environment variables, service ports, running modes, names of third image files, and resource configuration information of the big data cluster required for running the multi-tenant platform; containerize and deploy the fourth image file in the container cloud cluster where the big data cluster is located in the form of a deployment resource.

6. The big data analytics system of claim 1, wherein, The container meta management platform is further configured to: obtain a tenant creation instruction to construct a tenant corresponding to each user according to the tenant creation instruction; create a project role for the tenant corresponding to each user according to the identity information of each user; determine a required container cloud cluster for each project role in each tenant, and add the each project role to the required container cloud cluster.

7. The big data analytics system of claim 1, wherein, For each required container cloud cluster, the container meta management platform is further configured to: create a plurality of projects in each container cloud cluster; for each project in the plurality of projects, create an application and divide a resource demand range for the each project; add each user to a required project respectively.

8. The big data analytics system of claim 1, wherein, Each big data cluster in the plurality of big data clusters comprises: a driver service configured to parse a data query task submitted by the engine layer according to a logical execution plan, a physical execution plan, and a DAG directed acyclic graph; an executor service configured to perform calculation on the parsed data query task.

9. The big data analytics system of claim 1, wherein, The object storage system further comprises: a cluster controller configured to create the plurality of object storage clusters, and manage read-write permissions between tenants and each object storage cluster in the plurality of object storage clusters.

10. An intelligent embodied platform, characterized by The intelligent embodiment platform is deployed with the big data analysis system of any one of claims 1 to 9.

Images