Container-based IoT data batch processing method, processor, and storage medium

By deploying kyuubi and spark containers and combining them with Kubernetes resource management, the problem of poor resource isolation in distributed computing architecture is solved, data storage and computation are separated, resource utilization and development efficiency are improved, and operation and maintenance costs are reduced.

CN115981789BActive Publication Date: 2026-11-10ZOOMLION HEAVY INDUSTRY SCIENCE AND TECHNOLOGY CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211734910.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-30
Publication Date
2026-11-10
Estimated Expiration
2042-12-30

AI Technical Summary

Technical Problem

In existing distributed computing architectures, poor resource isolation leads to different tasks competing for computing resources, preventing urgent data analysis tasks from being computed first and affecting data analysis efficiency.

Method used

We adopt a container-based batch processing method for IoT data, deploy it using kyuubi and spark containers, manage resources using Kubernetes, separate data storage from computation, and ensure the free scaling of computing resources through Kubernetes' automatic maintenance function.

Benefits of technology

It simplifies the data analysis task submission process, improves development efficiency, increases resource utilization, reduces operation and maintenance costs, and ensures the stability and efficiency of data analysis tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115981789B_ABST
    Figure CN115981789B_ABST
Patent Text Reader

Abstract

This invention provides a container-based IoT data batch processing method, processor, and storage medium, belonging to the field of computer technology. The method involves: pulling a KYUUBI image from a pre-set KYUUBI image server; deploying KYUUBI in Kubernetes using this image; submitting data analysis tasks to the deployed KYUUBI container via a JDBC client; initiating a Spark container build request to the Kubernetes API server via a downloaded Kubernetes client using the KYUUBI container; pulling a Spark image from a pre-set Spark image server based on this request; deploying Spark in Kubernetes using this image; and finally, retrieving stored data from a Hive data warehouse using the deployed Spark container and executing data analysis tasks based on this stored data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a container-based IoT data batch processing method, processor, and storage medium. Background Technology

[0002] In the Industrial Internet of Things (IIoT) field, core value data (equipment, processes, materials) from various industrial equipment is transmitted in real-time to the IoT management platform via 4G / 5G wireless networks through various types of sensors and terminal devices, encrypted using various transmission protocols. This data is characterized by its large volume (PB level), diversity, rapid accumulation, and low value density. Therefore, the IoT management platform needs to perform extensive distributed computing analysis on this data daily to uncover its potential value. In the field of distributed computing, existing solutions typically employ two common big data computing architectures: Spark (a distributed computing engine) + YARN (a computing resource management and scheduling engine) or Spark Standalone (an architecture that independently deploys Spark clusters and relies on Spark itself for business and computing resource management) to perform statistical analysis on massive amounts of IoT data. However, in existing big data computing architectures, different Spark computing tasks share the same computing resources, all performing analysis and computation on the same Spark cluster. This results in poor resource isolation, leading to competition for computing resources (CPU, memory, bandwidth, I / O, etc.) between different tasks, preventing some urgent data analysis tasks from being prioritized. Summary of the Invention

[0003] In view of the above-mentioned shortcomings in the prior art, the purpose of this invention is to provide a container-based IoT data batch processing method, processor and storage medium.

[0004] To achieve the above objectives, a first aspect of the present invention provides a container-based IoT data batch processing method, comprising:

[0005] Pull the kyuubi image from the preset kyuubi image server to perform containerized deployment of kyuubi in Kubernetes based on the kyuubi image;

[0006] Submit data analysis tasks to the deployed kyuubi container via a JDBC client;

[0007] The kyuubi container initiates a Spark container build request to the k8sapiserver based on the downloaded k8s client;

[0008] Spark container build requests pull Spark images from a pre-defined Spark image server to perform containerized Spark deployment in Kubernetes based on the Spark images.

[0009] The deployed Spark container retrieves stored data from the Hive data warehouse to perform data analysis tasks based on the stored data.

[0010] In this embodiment of the invention, the steps for creating a Spark image include:

[0011] Obtain and extract the Spark installation package;

[0012] Obtain external system configuration files, including system configuration files corresponding to Hadoop, Hive, and MinIO clusters;

[0013] Determine the first directory to extract the Spark installation package;

[0014] Copy the external system configuration files as dependency files to the first unzipped directory;

[0015] Add the environment variable containing the location of the external configuration file to the configuration script in the first unzipped directory;

[0016] Obtain the Spark image build command file, which includes the system configuration file and configuration script in the first unzipped directory;

[0017] Execute the build instructions based on the Spark image build instruction file to obtain the Spark image.

[0018] In this embodiment of the invention, before the kyuubi container initiates a Spark container build request to the k8sapi server based on the downloaded k8s client, the following steps are also included:

[0019] Execute the first script, which is used to install the Kubernetes client and add the Kubernetes cluster authentication certificate when the kyuubi container starts.

[0020] In this embodiment of the invention, the steps for creating the kyuubi image include:

[0021] Obtain and extract the kyuubi installation package;

[0022] Build the first script to install the Kubernetes client when the kyuubi container starts and to add certificates for Kubernetes cluster authentication;

[0023] Determine the second extraction directory for the kyuubi installation package;

[0024] Add the location and startup method of the first script to the kyuubi startup script in the second unzipped directory;

[0025] Obtain the kyuubi image build instruction file, which includes the kyuubi startup script;

[0026] The kyuubi image is obtained by executing the build instructions based on the kyuubi image build instruction file.

[0027] In this embodiment of the invention, before pulling the kyuubi image from a preset kyuubi image server and deploying kyuubi containerizedly in Kubernetes based on the kyuubi image, the method further includes:

[0028] Start the corresponding resource files for kyuubi in sequence. These resource files include deployment resource files, resource configuration files, and service resource files.

[0029] In this embodiment of the invention, the step of constructing the deployment resource file includes:

[0030] Determine the namespace of the kyuubi container that needs to be started;

[0031] Determine the name of the kyuubi container and the corresponding configuration parameters for the kyuubi container;

[0032] The deployment resource file is built based on the namespace, name, and configuration parameters corresponding to the kyuubi container.

[0033] In this embodiment of the invention, the step of constructing a resource configuration file includes:

[0034] Retrieve environment variable configuration parameters used to determine kyuubi environment variables, submission configuration parameters used to submit data analysis tasks, and Spark configuration parameters;

[0035] Obtain the shuttle configuration parameters for the simplified information aggregation service and the dynamic resource adjustment parameters for Spark;

[0036] Obtain the connection parameters used to determine the connection relationship between Spark and MinIO;

[0037] A resource configuration file is built based on environment variable configuration parameters, submission configuration parameters, Spark configuration parameters, shuttle configuration parameters, Spark resource dynamic adjustment parameters, and connection parameters.

[0038] In this embodiment of the invention, the step of constructing the service resource file includes:

[0039] Determine the port configuration parameters used to define the source port, destination port, and port mapping format of the kyuubi container;

[0040] A service resource file is constructed based on port configuration parameters. The resource type specified in the service resource file is service, and the namespace is kyuubi.

[0041] A second aspect of the present invention provides a processor, characterized in that it is configured to execute the container-based IoT data batch processing method as described in the above embodiments.

[0042] A third aspect of the present invention provides a machine-readable storage medium storing instructions for causing a machine to perform a container-based IoT data batch processing method as described in the above embodiments.

[0043] The above technical solution involves: pulling a KYUBI image from a pre-set KYUBI image server to deploy KYUBI in Kubernetes using the KYUBI image; submitting data analysis tasks to the deployed KYUBI container via a JDBC client; initiating a Spark container build request to the Kubernetes API server via the KYUBI container using a downloaded Kubernetes client; pulling a Spark image from a pre-set Spark image server based on the Spark container build request to deploy Spark in Kubernetes using the Spark image; and finally, retrieving stored data from the Hive data warehouse through the deployed Spark container to execute data analysis tasks based on the stored data. No special configurations are needed for Spark images related to individual data tasks, maximizing the reusability of Spark images. The integration of Kyuubi and Spark simplifies the data analysis task submission process, improving development efficiency. Based on containers, data storage and computation are separated; the Spark computing engine runs within containers, enabling free scaling of computing resources. Separating data storage from computation allows for flexible scaling of data storage as well. Data is retrieved from the Hive data warehouse through Spark containers, facilitating data exchange between the two environments and significantly improving resource utilization. Furthermore, based on Kubernetes' automatic container maintenance capabilities—automatically starting a new pod when a pod fails—Spark ensures it never crashes while executing data analysis tasks, greatly reducing operational costs.

[0044] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0045] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings:

[0046] Figure 1 This is a flowchart illustrating a container-based IoT data batch processing method according to an embodiment of the present invention.

[0047] Figure 2 This is a schematic diagram of the kyuubi architecture design according to an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of a framework for performing data analysis tasks according to an embodiment of the present invention. Detailed Implementation

[0049] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the scope of the present invention.

[0050] Figure 1 This is a flowchart illustrating a container-based IoT data batch processing method according to an embodiment of the present invention. Figure 1 As shown in the embodiment of the present invention, a container-based IoT data batch processing method is provided. Taking the application of this method to a processor as an example, the method may include the following steps:

[0051] Step S100: Pull the kyuubi image from the preset kyuubi image server to perform kyuubi containerized deployment in k8s based on the kyuubi image;

[0052] It's important to note that Kyuubi is a distributed and multi-tenant gateway that provides SQL query services to data lake query engines (such as Spark, Flink, or Trino). In this embodiment, Kyuubi is integrated with Spark, and both are containerized for deployment. This reduces the difficulty of Spark application development, lowers the barrier to entry for Spark, and reduces operational costs. Kyuubi, as a distributed and multi-tenant gateway, provides SQL access services to Spark, the IoT data analysis and computation engine. This means that simply submitting the SQL code for data analysis tasks to Kyuubi via a JDBC client enables the submission, computation, and analysis of massive IoT data analysis tasks. There's no need to pre-build a Spark cluster, write task submission scripts, or configure computing resource parameters, significantly simplifying the development, deployment, and operation of Spark data analysis tasks.

[0053] refer to Figure 2 The kyuubi architecture is mainly divided into 4 layers:

[0054] The UserSide refers to users (User1, User2) who use Kyubi in different ways, such as through JDBC or Beeline.

[0055] The Service Discovery Side relies on Zookeeper for its implementation and is further divided into service discovery at the KyuubiServer layer and service discovery at the KyuubiEngine layer.

[0056] The KyuubiServer layer consists of multiple different KyuubiServer instances. Each KyuubiServer instance is essentially an RPC server implemented based on Apache Thrift. It receives requests from users but does not actually execute the relevant SQL operations for the request. Instead, it acts as a proxy to forward the request to the SparkSQLEngine instance to which the user in the KyuubiEngine layer belongs.

[0057] The KyuubiEngine layer consists of multiple distinct SparkSQLEngine instances. Each SparkSQLEngine instance is essentially an RPC server implemented using Apache Thrift and holding a SparkSession instance. It receives requests from the KyuubiServer instance and executes them through the SparkSession instance. At the kyuubi USER shared layer, each SparkSQLEngine instance is user-level; different users hold different SparkSQLEngine instances to achieve user-level resource isolation and control. To submit tasks to containerized Spark, kyuubi must also be containerized. In this embodiment, kyuubi is containerized and integrated with containerized Spark.

[0058] In this embodiment, big data-related components are deployed based on Kubernetes (k8s) containerization technology. k8s is a Docker container orchestration and management system. For kyuubi to achieve containerized deployment, a virtual file system is required. This virtual file system contains all kyuubi directories. These directories, along with the host machine's kernel, provide a Linux virtual environment. When this virtual Linux environment is started within the Docker container managed and orchestrated by k8s, the kyuubi cluster can be started. The virtual file system containing all kyuubi directories is the kyuubi image, which can only run within k8s. To simplify the deployment process for different data analysis tasks, the kyuubi image needs to be reused. The kyuubi image can be pre-built and uploaded to a pre-defined kyuubi image server, such as the Harbor image server, so that the kyuubi image can be quickly pulled each time a version is released or updated, avoiding the repeated creation of the kyuubi image.

[0059] Specifically, the kyuubi image is pulled from the preset kyuubi image server to perform containerized deployment of kyuubi in Kubernetes based on the kyuubi image.

[0060] Step S200: Submit a data analysis task to the deployed kyuubi container via a JDBC client;

[0061] In this embodiment, it should be noted that the data analysis task includes Spark SQL analysis code. Using the Hive Beeline client, by entering the JDBC connection address of kyuubi and following the prompts to enter the username and password, a JDBC connection with the kyuubi server can be established. At this point, the pre-prepared data analysis task is submitted through this JDBC client. After receiving the data analysis task request, the kyuubi server sends the request to the kyuubi engine.

[0062] Step S300: The kyuubi container initiates a Spark container build request to the k8sapiserver based on the downloaded k8s client.

[0063] Specifically, before the kyuubi container sends a Spark container build request to the k8sapiserver based on the downloaded k8s client, the following steps are also included:

[0064] Execute the first script, which is used to install the Kubernetes client and add the Kubernetes cluster authentication certificate when the kyuubi container starts.

[0065] In this embodiment, it should be noted that when the processor executes the first script, it will download the Kubernetes client and add the Kubernetes cluster authentication certificate when the kyuubi container starts. The kyuubiengine initiates a Spark container build request to the k8sapiserver service through the Kubernetes client kubectl to request the establishment of a Spark cluster.

[0066] Step S400: The Spark container build request pulls the Spark image from the preset Spark image server to perform Spark containerized deployment in Kubernetes based on the Spark image.

[0067] In this embodiment, Kubernetes (k8s) is the orchestration and management system for Docker containers. For Spark to run within containers, a virtual file system is required. This virtual file system contains all the directories of the Spark distributed computing engine. These directories, along with the host machine's kernel, provide a Linux virtual environment. When this virtual Linux environment is started within the Docker container managed and orchestrated by k8s, the Spark cluster can be launched. The virtual file system containing all Spark directories is the Spark image. This image can only run within k8s. To facilitate the quick and easy setup of the Spark distributed computing environment in k8s when submitting different Spark data analysis tasks, the Spark image needs to be reusable. Therefore, the Spark image needs to be prepared in advance and uploaded to a pre-defined Spark image server so that when submitting a data analysis task, the Spark image can be retrieved simply from that Spark image server.

[0068] Specifically, kyuubiengine sends a request to the k8sapiserver service through the k8s client kubectl to create a sparkdriverpod. The sparkdriverpod requests computing resources from k8s, starts a certain number of sparkexecutorpods, and begins computing for data analysis tasks.

[0069] Step S500: Obtain stored data from the Hive data warehouse through the deployed Spark container, and perform data analysis tasks based on the stored data through the Spark container.

[0070] In this embodiment, massive amounts of IoT data are stored in the Hadoop file system, while only the Spark computing engine runs in a container. This achieves separation of storage and computation, completely decoupling them to enable elastic scaling of Spark compute nodes. It's important to note that the stored data refers to the massive amounts of IoT data. The Hive data warehouse is based on the Hadoop Distributed File System. When executing data analysis tasks in the Spark container, it needs to pull stored data from the remote Hive data warehouse to perform the data analysis tasks.

[0071] In this embodiment, after the data analysis task is completed, different Spark executor pods will mark the pod status as "completed" according to whether the calculation is complete, and then release the resources of that pod for use by other pods to maximize resource utilization. When the status of all Spark executors becomes "completed", it means that the data analysis task has ended, and all resources of the Spark cluster can be released. At the same time, the Spark executor pod will return the calculation and analysis results to the JDBC client, and the entire data analysis task ends.

[0072] The above-described container-based IoT data batch processing method involves: pulling a KYUBI image from a pre-set KYUBI image server for containerized deployment in Kubernetes; submitting data analysis tasks to the deployed KYUBI container via a JDBC client; initiating a Spark container build request to the Kubernetes API server via the downloaded Kubernetes client through the KYUBI container; pulling a Spark image from a pre-set Spark image server based on the Spark container build request for containerized deployment in Kubernetes; and retrieving stored data from the Hive data warehouse through the deployed Spark container to execute data analysis tasks based on the stored data. No special configurations are needed for Spark images related to individual data tasks, maximizing the reusability of Spark images. The integration of Kyuubi and Spark simplifies the data analysis task submission process, improving development efficiency. Based on containers, data storage and computation are separated; the Spark computing engine runs within containers, enabling free scaling of computing resources. Separating data storage from computation allows for flexible scaling of data storage as well. Data is retrieved from the Hive data warehouse through Spark containers, facilitating data exchange between the two environments and significantly improving resource utilization. Furthermore, based on Kubernetes' automatic container maintenance capabilities—automatically starting a new pod when a pod fails—Spark ensures it never crashes while executing data analysis tasks, greatly reducing operational costs.

[0073] refer to Figure 3 The prepared Spark SQL analysis code is submitted via the JDBC client. Upon receiving the data analysis task request, the KyuubiServer sends it to the KyuubiEngine. The KyuubiEngine then uses the Kubernetes client kubectl to request the creation of a Spark DriverPod from the Kubernetes APIServer. The DriverPod requests computing resources from Kubernetes, starts a certain number of Spark ExecutorPods, and begins computation for the Spark SQL analysis task. During the startup process of each service container, the corresponding service pods are automatically started according to the YAML constraints specified in the previous steps, and the computation is completed automatically. In this embodiment, massive IoT data is stored in the Hadoop file system, and only the Spark computing engine runs in a container.

[0074] In one embodiment, the steps for creating a Spark image include:

[0075] Obtain and extract the Spark installation package;

[0076] Obtain external system configuration files, including system configuration files corresponding to Hadoop, Hive, and MinIO clusters;

[0077] Determine the first directory to extract the Spark installation package;

[0078] Copy the external system configuration files as dependency files to the first unzipped directory;

[0079] Add the environment variable containing the location of the external configuration file to the configuration script in the first unzipped directory;

[0080] Obtain the Spark image build command file, which includes the system configuration file and configuration script in the first unzipped directory;

[0081] Execute the build instructions based on the Spark image build instruction file to obtain the Spark image.

[0082] In this embodiment, it should be noted that there are multiple versions of the Spark installation package, each with different functionalities, and they can be adapted to other versions of big data components. Different versions of Spark images can be created based on the different versions of the Spark installation package, distinguished by different Docker image tags. The first decompression directory is the directory where the Spark installation package is decompressed under the Linux system. After decompression, you can enter any subdirectory under the first decompression directory to view, modify files, and configure parameters.

[0083] In this embodiment, Spark is integrated with Hadoop, Hive, and MinIO (a distributed object storage system). Massive amounts of IoT data are stored in the Hadoop file system, with only the Spark computing engine running in a container, achieving separation of storage and computation, thus completely decoupling them. During the Spark image creation process, many data analysis tasks rely on third-party JAR files or configuration files that are task-specific. Therefore, to obtain a reusable Spark image, this embodiment places these task-specific files in a pre-built MinIO system.

[0084] To enable a Spark Cluster cluster started from a Spark image to access IoT data in Hive and Hadoop Distributed File System (HDFS) outside the Kubernetes cluster, as well as related dependency object files in MinIO, it is necessary to copy the external system configuration files related to the existing Hadoop, Hive, and MinIO clusters, along with the dependent Java JAR files, to a specific location in the first unzipped directory. The external system configuration files mainly include hdfs-site.xml, core-site.xml, and hive-site.xml. These should be copied to the conf folder in the first unzipped directory, and the dependent JAR files should also be copied to the jars folder in the first unzipped directory.

[0085] It's important to note that to ensure the Spark container can locate the Hadoop and Hive configurations upon startup, enabling the Spark cluster to connect with Hadoop and Hive and access the massive amounts of IoT data in the Hive data warehouse, the environment variables containing these external configuration files are added to the configuration script in the first unzipped directory. Specifically, the `spark-env.sh` shell script in the `conf` directory of the first unzipped directory is configured to add the environment variables containing the Hadoop and Hive configuration files.

[0086] It's important to note that when building a Spark image using Docker, a Spark image build instruction file, also known as a Dockerfile, must be prepared. This file contains numerous instructions and specifications required for building the Spark image. In this example, the Spark image build instruction file specifies: the JDK environment that Spark depends on, the user account for the Spark cluster, the system tools that need to be installed within the container after the Spark image starts, the creation of some directories within the container, copying some files and directories from the first extracted directory to the container, specifying the environment variables and working directory within the container, granting permissions to some directories within the Spark container, and specifying the shell script `entrypoint.sh` to execute the preprocessing logic at runtime. Executing the build instructions based on this Spark image build instruction file will produce the Spark image. After the Spark image is built, it is tagged and pushed to a pre-defined Spark image server, such as the Harbor image server, so that the Spark image is automatically pulled from the Spark image server each time a data analysis task is published, avoiding repeated Spark image builds.

[0087] In this embodiment, by pre-building a Spark image, the Spark image can be created once and reused repeatedly, simplifying the deployment process for different data analysis tasks.

[0088] In one embodiment, the steps for creating a kyuubi image include:

[0089] Obtain and extract the kyuubi installation package;

[0090] Build the first script to install the Kubernetes client when the kyuubi container starts and to add certificates for Kubernetes cluster authentication;

[0091] Determine the second extraction directory for the kyuubi installation package;

[0092] Add the location and startup method of the first script to the kyuubi startup script in the second unzipped directory;

[0093] Obtain the kyuubi image build instruction file, which includes the kyuubi startup script;

[0094] The kyuubi image is obtained by executing the build instructions based on the kyuubi image build instruction file.

[0095] In this embodiment, it should be noted that there are multiple versions of the kyuubi installation package, each with different functionalities. These versions can be adapted to different versions of big data components. Different versions of the kyuubi installation package are used to create different kyuubi images, each tagged with a different Docker image. The second decompression directory is the directory where the kyuubi installation package is extracted under the Linux system. After decompression, users can navigate to any subdirectory within the second decompression directory to view files, modify files, and configure parameters.

[0096] It's important to note that, in order for the kyuubi container to be able to initiate requests to the k8sapiserver via the Kubernetes client within the container when a user submits a Spark data analysis task through kyuubi—that is, to start a Spark computing cluster and execute the Spark data analysis task request passed by kyuubi—a first script needs to be built to install the Kubernetes client and add the Kubernetes cluster authentication certificate when the kyuubi container starts. Specifically, the first script includes the following instructions: creating a directory and granting read and write permissions, copying the Kubernetes cluster authentication file to the appropriate directory, configuring the Kubernetes authentication address environment variable within the container, and downloading the Kubernetes client kubectl. The second extraction directory for the kyuubi installation package should be determined; in the second extraction directory, the kyuubi startup script should be modified to include the location and startup method of the first script.

[0097] It's important to note that to obtain a working kyuubi image, we need to create a Dockerfile, which contains numerous instructions and explanations required to build the kyuubi image. In this embodiment, the Dockerfile includes: modifying the OpenJDK source, modifying the Debian source, installing related software, copying the first script (used to modify the ` / etc / hosts` file after the image is started), setting the user to 0 (root), copying the second extracted directory to the corresponding directory inside the container, and specifying the instructions for running kyuubi within the container. Executing the build instructions based on this Dockerfile will produce the kyuubi image. After the kyuubi image is built, it is tagged and pushed to a pre-defined kyuubi image server so that it can be quickly pulled for each release or update, avoiding repeated builds. When you need to build a multi-tenant unified submission service platform for kyuubisql, you only need to pull the kyuubi image from the kyuubi image server and deploy it in Kubernetes.

[0098] In this embodiment, the deployment process for different data analysis tasks is further simplified by pre-building the kyuubi image.

[0099] In one embodiment, before pulling the kyuubi image from a preset kyuubi image server to perform containerized deployment of kyuubi in Kubernetes based on the kyuubi image, the method further includes:

[0100] Start the corresponding resource files for kyuubi in sequence. These resource files include deployment resource files, resource configuration files, and service resource files.

[0101] In this embodiment, the corresponding kyuubi resource files are started sequentially on one of the nodes in the Kubernetes cluster. This involves executing the Kubernetes workload creation commands, which sequentially create three resource objects: configmap, service, and deployment. The kyuubi workload is then successfully created, and the kyuubi-deployment will include pods for kyuubi-server and kyuubi-engine, with the specific containers running within each pod. Once the kyuubi-related pods are fully running in the Kubernetes containers, data analysis tasks can be submitted from outside the Kubernetes cluster by connecting to the kyuubi-server using the Beeline client.

[0102] Specifically, the steps for building deployment resource files include:

[0103] Determine the namespace of the kyuubi container that needs to be started;

[0104] Determine the name of the kyuubi container and the corresponding configuration parameters for the kyuubi container;

[0105] The deployment resource file is built based on the namespace, name, and configuration parameters corresponding to the kyuubi container.

[0106] In this embodiment, it should be noted that before submitting the Spark data analysis task to Kubernetes, the kyuubi-related services (kyuubiserver and kyuubiengine) need to be started in the Kubernetes container. Deploying services in Kubernetes simply requires clearly describing the corresponding resource objects using YAML, and finally submitting the YAML file to the Kubernetes cluster to start the corresponding service. The YAML file is a resource file; in this embodiment, the resource file includes the deployment resource file, which is the kyuubi-deployment resource YAML file.

[0107] Kubernetes deployments are the most commonly used workloads and the foundation for infrastructure scaling. The purpose of deployments is to publish, upgrade, roll back, and scale applications. Therefore, we also use deployments to publish Kubernetes applications. The deployment resource file configuration mainly includes: the name and tag of the kyuubi-deployment; the namespace of the Kubernetes container to be started; the mapping relationship between the IP address and hostname to be resolved in the container; the container to be started and its related configuration parameters; the environment variables in the container; the configuration files to be loaded by the container; and the container's persistent storage, etc. After configuring the above parameters according to YAML programming standards, the deployment resource file is obtained.

[0108] Specifically, the steps for building a resource configuration file include:

[0109] Retrieve environment variable configuration parameters used to determine kyuubi environment variables, submission configuration parameters used to submit data analysis tasks, and Spark configuration parameters;

[0110] Obtain the shuttle configuration parameters for the simplified information aggregation service and the dynamic resource adjustment parameters for Spark;

[0111] Obtain the connection parameters used to determine the connection relationship between Spark and MinIO;

[0112] A resource configuration file is built based on environment variable configuration parameters, submission configuration parameters, Spark configuration parameters, shuttle configuration parameters, Spark resource dynamic adjustment parameters, and connection parameters.

[0113] In this embodiment, it should be noted that the Kyuubi container requires configuration of many parameters during startup, such as those related to Kyuubi itself, Spark, and MinIO. These mainly include: environment variable configuration parameters for determining Kyuubi environment variables, submission configuration parameters for submitting data analysis tasks, and Spark configuration parameters. Since these configurations exist in different file formats, in order to centralize these configuration files (kyuubi-en.sh, kyuubi-defauts.conf) and facilitate the loading of the Kyuubi container, we manage these configurations through the Kubernetes configMap resource object. By ultimately forming a configmap YAML file, these configuration files are unified together to constitute a resource configuration file.

[0114] It should be noted that, in order to avoid task failures due to the loss of shuffle data during the scaling up and down rebuilding of Spark executor pods, this embodiment uses a shuttle (a Spark remote shuffle service) to remotely store, manage, and optimize the data exchange process between map / reduce tasks in different stages during the runtime of Spark data analysis tasks. This improves Spark's computing efficiency while ensuring the stability of Spark compute pods during dynamic adjustments.

[0115] By integrating containerized Spark with the remote RSS service shuttle, this solution addresses the issue of task recalculation caused by shuffle data loss during Spark executor pod scaling (reducing the number of pods). This allows Spark's dynamic resource adjustment capabilities to function effectively after containerization. In this embodiment, pre-deployed shuttle configuration parameters are configured in a resource configuration file. When the Spark container starts, it can read the shuttle connection information through the corresponding configmap resource object in the resource configuration file, thereby processing Spark's map / reduce shuffle data in the remote shuttle and improving Spark's computing performance.

[0116] It should be noted that during the runtime of Spark distributed computing tasks, the resources required for different computing tasks may vary depending on the amount of data. To accelerate computing efficiency, the number of executor pods in the Spark cluster within the container can be automatically adjusted according to the size of the data and the resource requirements of the computing tasks. In other words, computing resources are dynamically adjusted. For containerized Spark, this embodiment configures the shuttle configuration parameters in the resource configuration file, along with Spark's dynamic resource adjustment parameters. This allows the Spark container to automatically scale up or down based on the amount of data to be computed by the data analysis task when it starts, maximizing the utilization of computing resources.

[0117] It should be noted that, in order to enable the Spark container to pull the files needed for this data analysis task from the external service minio into the Spark container when it starts up, and to participate in the data computation, the connection parameters used to determine the connection relationship between Spark and minio should be configured in the resource configuration file.

[0118] Specifically, the steps for building service resource files include:

[0119] Determine the port configuration parameters used to define the source port, destination port, and port mapping format of the kyuubi container;

[0120] A service resource file is constructed based on port configuration parameters. The resource type specified in the service resource file is service, and the namespace is kyuubi.

[0121] After starting the containers for the corresponding services (kyuubi-server, kyuubi-engine) in Kubernetes, these services can only be accessed within the container without further processing. Services outside the Kubernetes cluster will be unable to access kyuubi-server due to network issues. Services within the Kubernetes cluster cannot be accessed externally due to network restrictions. To address this issue, this embodiment uses Kubernetes service resources to define port and network protocols, exposing the service ports within the container to the outside of the Kubernetes cluster via NodePort or Ingress, enabling remote submission of Spark data analysis tasks to the Kubernetes cluster. Exposing service ports via NodePort specifically involves configuring the service resource file, specifying the resource type as "service," the namespace as "kyuubi," and defining the source port, target port, and port mapping parameters for the kyuubi container.

[0122] It's important to note that resource isolation in Kubernetes is achieved through namespaces. By creating a namespace in Kubernetes and assigning it specific CPU and memory resources, projects or tasks bound to that namespace can only use the resources within that namespace and cannot share or preempt resources from other namespaces. After creating the namespace, a service account and role need to be created under that namespace and bound together. All subsequent operations are performed using this service account in Kubernetes for pod creation and deletion. In this embodiment, a namespace is created for Spark, and service accounts and roles are bound accordingly to achieve resource isolation for different Spark analytics tasks and fine-grained management of related permissions.

[0123] This invention provides a processor for running a program, wherein the program executes the container-based IoT data batch processing method during runtime.

[0124] This invention provides a machine-readable storage medium storing a program that, when executed by a processor, implements the container-based IoT data batch processing method.

[0125] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0126] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0130] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0131] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0132] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0133] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A container-based IoT data batch processing method, characterized in that, include: Pull the kyuubi image from the preset kyuubi image server, and deploy kyuubi containerized in Kubernetes based on the kyuubi image; Submit data analysis tasks to the deployed kyuubi container via a JDBC client; The kyuubi container initiates a Spark container build request to the Kubernetes API server based on the downloaded Kubernetes client. Based on the Spark container build request, a Spark image is pulled from a preset Spark image server to perform containerized Spark deployment in Kubernetes based on the Spark image. The deployed Spark container retrieves stored data from the Hive data warehouse, and then executes the data analysis task based on the stored data using the Spark container. Before initiating a Spark container build request to the Kubernetes API server via the downloaded Kubernetes client through the kyuubi container, the process also includes: Execute the first script, which is used to install the k8s client and add the k8s cluster authentication certificate when the kyuubi container starts; The resource configuration file corresponding to the kyuubi container includes shuttle configuration parameters and Spark resource dynamic adjustment parameters. The shuttle configuration parameters are used to configure the remote shuffle service, and the Spark resource dynamic adjustment parameters are used to adjust the number of Spark executor containers.

2. The IoT data batch processing method according to claim 1, characterized in that, The steps for creating the Spark image include: Obtain and extract the Spark installation package; Obtain external system configuration files, including system configuration files corresponding to the Hadoop, Hive, and MinIO clusters; Determine the first directory to extract the Spark installation package; Copy the external system configuration file to the first decompressed directory as a dependency file; Add the environment variable containing the location of the external configuration file to the configuration script in the first decompressed directory; Obtain the Spark image build instruction file, which includes the system configuration file and configuration script in the first decompressed directory; The Spark image is obtained by executing the build instructions based on the Spark image build instruction file.

3. The IoT data batch processing method according to claim 1, characterized in that, The steps for creating the kyuubi image include: Obtain and extract the kyuubi installation package; Build the first script to install the Kubernetes client when the kyuubi container starts and to add certificates for Kubernetes cluster authentication; Determine the second extraction directory for the kyuubi installation package; Add the location and startup method of the first script to the kyuubi startup script in the second decompressed directory; Obtain the kyuubi image build instruction file, which includes the kyuubi startup script; The kyuubi image is obtained by executing the build instructions based on the kyuubi image build instruction file.

4. The IoT data batch processing method according to claim 1, characterized in that, Before pulling the kyuubi image from the preset kyuubi image server and deploying kyuubi containerized in Kubernetes based on the kyuubi image, the process also includes: Start the corresponding resource files for kyuubi in sequence. The resource files include deployment resource files, resource configuration files, and service resource files.

5. The IoT data batch processing method according to claim 4, characterized in that, The steps for constructing the deployment resource file include: Determine the namespace of the kyuubi container that needs to be started; Determine the name of the kyuubi container and the corresponding configuration parameters of the kyuubi container; The deployment resource file is constructed based on the namespace, the name, and the configuration parameters corresponding to the kyuubi container.

6. The IoT data batch processing method according to claim 4, characterized in that, The steps for constructing the resource configuration file include: Retrieve the environment variable configuration parameters used to determine the kyuubi environment variables, the submission configuration parameters used to submit data analysis tasks, and the Spark configuration parameters; Obtain the shuttle configuration parameters for the simplified information aggregation service and the dynamic resource adjustment parameters for Spark; Obtain the connection parameters used to determine the connection relationship between Spark and MinIO; The resource configuration file is constructed based on the environment variable configuration parameters, the submission configuration parameters, the Spark configuration parameters, the shuttle configuration parameters, the Spark resource dynamic adjustment parameters, and the connection parameters.

7. The IoT data batch processing method according to claim 4, characterized in that, The steps for constructing the service resource file include: Determine the port configuration parameters used to define the source port, target port, and port mapping format of the kyuubi container; The service resource file is constructed based on the port configuration parameters, wherein the resource type specified in the service resource file is service and the namespace is kyuubi.

8. A processor, characterized in that, It is configured to perform the container-based IoT data batch processing method according to any one of claims 1 to 7.

9. A machine-readable storage medium storing instructions thereon, characterized in that, This instruction is used to cause the machine to perform the container-based IoT data batch processing method as described in any one of claims 1 to 7.