K8s-based JUPter Notebook container arrangement method

Abstract

The invention relates to the technical field of containers and micro-service architecture, and discloses a K8s-based Juser Notebook container arrangement method, which comprises the following steps of: S1, packaging Notebook into a Docker mirror image file; s2, mounting a Notebook configuration file on the K8s cluster through a ConfigMap (Configuration Map); s3, deploying and defining the number of copies of the Notebook application through Deployment; S4, configuring a service network and an ingress network; S3, deploying and defining the number of copies of the Notebook application through Deployment; s5, a cloud storage mode is adopted, and calculation and storage are separated; and S6, K8s API automatic management is carried out. The deployment process is simplified, the maintenance and management difficulty is reduced, and the resource utilization rate is improved; a cloud storage mode is adopted, separation of calculation and storage is achieved, and environment configuration is simplified; the Ingress provides network functions such as flow management and reverse proxy, so that the Notebook service is easier to access and integrate; the K8s cluster can ensure the high availability of the Notebook service; and when a fault occurs, quick recovery is realized, service interruption time is reduced, monitoring and management are easy, and management and operation and maintenance efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of container and microservice architecture technology, specifically a JupyterNotebook container orchestration method based on Kubernetes. Background Technology

[0002] Container technology is a lightweight, portable software packaging and runtime environment. Its core goal is to achieve standardized encapsulation of application code, dependency libraries, and configurations. Containers isolate process resources through operating system-level virtualization technologies (such as Linux namespaces and control groups) to ensure that applications have completely consistent behavior in development, testing, and production environments.

[0003] Microservice architecture is a software design pattern that breaks down a monolithic application into a set of small services, each running in an independent process and collaborating through lightweight communication mechanisms (such as HTTP / RPC). Each microservice focuses on a specific business capability and has complete lifecycle autonomy.

[0004] Container technology provides an ideal runtime environment for microservice architectures, and the combination of the two forms the technological cornerstone of modern cloud-native applications.

[0005] Traditional container orchestration methods either deploy notebooks on a single machine for data analysis or manually deploy them across multiple machine clusters. However, single-machine deployment cannot meet the requirements for elastic scaling, while manual deployment across multiple machine clusters is complex and difficult to manage and maintain. Therefore, we need to propose a Jupyter Notebook container orchestration method based on Kubernetes. Summary of the Invention

[0006] The purpose of this invention is to provide a Jupyter Notebook container orchestration method based on Kubernetes (K8s). By leveraging the load balancing, high availability, and fault transfer features of K8s and the interactive computing environment of Jupyter Notebook, this method ensures environmental isolation between different users and can be applied to industry services for data analysis by different users, thereby solving the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a Jupyter Notebook container orchestration method based on K8s, comprising the following steps: S1. Package the Notebook into a Docker image file; S2. Mount the Notebook configuration file on the K8s cluster using ConfigMap; S3. Define the number of copies of the Notebook application through Deployment deployment, and manage and control the lifecycle and updates of multiple notebook pods; S4. Configure the service and ingress networks, configure the Layer 4 proxy Service and the Layer 7 proxy Ingress file, expose the access interface through Service, and provide traffic management through Ingress. S5. Adopts cloud storage to separate computing from storage; S6 and Kubernetes API automation management allow for declarative management of Notebook application deployment and lifecycle via the Kubernetes API.

[0008] Preferably, in step S1, Jupyter Notebook and its runtime environment are packaged into a container image using Docker container technology. This means that the Jupyter Notebook application code, all its dependent third-party libraries, system tools, and configuration files are packaged into an independent Docker image, thus creating a self-contained and portable runtime environment.

[0009] Preferably, in step S2, ConfigMap is an AP object provided by K8s, used to save non-confidential data in the form of key-value pairs. ConfigMap extracts the parameters of various configuration files and YAML definition files of Jupyter Notebook from the container image and manages them as independent K8s resources.

[0010] Preferably, ConfigMap supports two methods for injecting configuration data into the Pod: As environment variables, the key-value pairs in the ConfigMap are directly injected into the environment variables of the Pod container; As a volume mount, the entire configuration file or directory is mounted as a volume to a specified path in the container, completely replacing the original files in the container image.

[0011] Preferably, when a Pod starts, Kubernetes automatically mounts the configuration data defined in the ConfigMap to the specified container path. This means that the same Docker image file can exhibit different behaviors in different environments by using different ConfigMaps, without the need to rebuild the image.

[0012] Preferably, in step S3, the Jupyter Notebook is upgraded from a single Pod instance to a robust service consisting of multiple replicas with self-healing and rolling update capabilities using the Deployment controller in K8s.

[0013] Preferably, Deployment allows users to declaratively define the desired state of a Pod, including the number of Pod replicas that need to be run. The Deployment controller takes full control of the Pod's lifecycle, which includes Pod creation, scheduling, updating, rollback, and deletion. When the Notebook image version or configuration needs to be updated, Deployment supports rolling updates of the configuration, gradually replacing the old version of the Pod with the new version.

[0014] Preferably, in step S4, the service is used to define a logical set of Pods and access policies. The service dynamically associates backend Pods through a label selector and automatically distributes the request traffic arriving at the service to healthy backend Pod instances to achieve balanced distribution of requests. The Ingress, as the entry point of the K8s cluster, provides HTTP / HTTPS routing capabilities. The Ingress directs traffic to different backend services based on the domain name and URL path of the request, and handles HTTPS encryption and decryption uniformly at the Ingress level, reducing the burden on backend Pods.

[0015] Preferably, in step S5, the persistent data of the user's workspace and the persistent data of the analysis results are stored in an independent cloud storage service through the persistent volume and persistent declaration mechanism of K8s. According to the creation requirements of the notebook pod, the storage volume is automatically applied for and mounted from the cloud platform, and a computing container is created to automatically allocate independent storage space for the notebook pod.

[0016] Preferably, the K8s API submits a desired state file in YAML or JSON format to the server. After receiving the file, the API server drives the actual state of the cluster to converge to the desired state declared by the user. The K8s API provides a complete operation interface for applications from creation to destruction, including: creation and deployment, monitoring and querying, scaling up and down, updating and rolling back, deletion and cleanup. Furthermore, the K8s API automatically orchestrates the number of Jupyter Notebook containers according to the number of users, and manages, stores, and analyzes data for each user's container.

[0017] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention simplifies the deployment process, reduces maintenance and management difficulty, and improves resource utilization; it adopts cloud storage to separate computing and storage, simplifying environment configuration; Ingress provides network functions such as traffic management and reverse proxy, making Notebook services easier to access and integrate; 2. The K8s cluster of this invention can guarantee high availability of Notebook services; it can quickly recover from failures, reduce business interruption time, and is easy to monitor and manage, thereby improving management and operation efficiency. Attached Figure Description

[0018] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram illustrating the configuration of the ConfigMap of this invention; Figure 3 This is a schematic diagram of the Deployment workload configuration of the present invention; Figure 4 This is a screenshot of the notebook application of the intelligent analysis platform for structural monitoring data of this invention. Figure 5 This is an interface diagram of the notebook container automatically generated based on the number of users in this invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figure 1-5 This invention provides a technical solution: a Jupyter Notebook container orchestration method based on K8s, comprising the following steps: S1. Package the Notebook into a Docker image file; In step S1, Jupyter Notebook and its runtime environment are packaged into a container image using Docker container technology. This means that the Jupyter Notebook application code, all its dependent third-party libraries, system tools, and configuration files are packaged into an independent Docker image, thus creating a self-contained and portable runtime environment.

[0021] Once the image is built, it becomes an immutable entity. This ensures that exactly the same content runs when deployed in any environment, completely eliminating a range of problems caused by environmental differences.

[0022] Achieve environmental consistency, ensuring consistency between development, testing, and production environments. Lightweight packaging reduces resource consumption, improves deployment efficiency, and facilitates version management and image distribution.

[0023] Step S1 uses containerization technology to standardize and encapsulate the application and its runtime environment, laying a solid foundation for subsequent automated orchestration and management on the K8s cluster.

[0024] Docker images are composed of a series of read-only layers, such as: operating system layer, runtime environment layer (e.g., Python), dependency installation layer, and application code layer. This layered structure makes image building efficient, storage-saving, easy to reuse, and quick to distribute.

[0025] Step S1 achieves absolute consistency from development and testing to production environments. Images built by developers can run without modification on any Kubernetes cluster that supports Docker, significantly reducing debugging time during deployment and improving the reliability and speed of software delivery. Compared to traditional virtual machines, Docker containers share the host operating system kernel, eliminating the need to load a complete operating system for each application, resulting in smaller size and faster startup. Each image can be version-managed using tags, and the image repository can store, manage, and track different versions of images. Each Notebook instance runs in an independent container, with its own file system, process space, and network interface, isolated from other instances.

[0026] S2. Mount the Notebook configuration file on the K8s cluster using ConfigMap; In step S2, ConfigMap is an AP object provided by K8s, used to save non-confidential data in the form of key-value pairs. ConfigMap extracts the parameters of various configuration files and YAML definition files of Jupyter Notebook from the container image and manages them as independent K8s resources.

[0027] ConfigMap supports two methods for injecting configuration data into the Pod. The two injection methods are: As environment variables, the key-value pairs in the ConfigMap are directly injected into the environment variables of the Pod container; As a volume mount, the entire configuration file or directory is mounted as a volume to a specified path in the container, completely replacing the original files in the container image.

[0028] When a Pod starts, Kubernetes automatically mounts the configuration data defined in the ConfigMap to the specified container path. This means that the same Docker image file can exhibit different behaviors in different environments by using different ConfigMaps, without the need to rebuild the image.

[0029] It separates configuration from application, improves the flexibility of configuration management, facilitates version control and centralized management of configuration, supports dynamic configuration updates, and eliminates the need to redeploy applications.

[0030] Specifically, first, use the kubectl command or a declarative YAML file to create a ConfigMap (containing all the configuration information required by the Notebook service). Then, in the Pod template definition of the Deployment, declare a volume using the volumes field. The volume type is ConfigMap, and specify the name of the ConfigMap to reference. Finally, in the container definition, use the volumeMounts field to mount the declared volume to a specific directory inside the container (such as / var / app / middlewares / orchestrator / templates). When the Pod runs, the files defined in the ConfigMap will appear in that directory.

[0031] When the contents of a ConfigMap are updated, Kubernetes automatically synchronizes the updated files to the Pods that have mounted the ConfigMap. This addresses business scenarios that require frequent configuration adjustments and avoids service interruptions caused by a complete service restart.

[0032] S3. Define the number of copies of the Notebook application through Deployment deployment, and manage and control the lifecycle and updates of multiple notebook pods; In step S3, the Jupyter Notebook is upgraded from a single Pod instance to a robust service consisting of multiple replicas with self-healing and rolling update capabilities using the Deployment controller in Kubernetes.

[0033] Deployment allows users to declaratively define the desired state of a Pod, including the number of Pod replicas that need to run. If the number of Pod replicas is set to three, Kubernetes will continuously monitor to ensure that there are exactly three NotebookPod instances running at any given time. Even if a node fails, causing one of the Pods running on it to become invalid, Deployment can immediately recreate a new Pod on a healthy node to ensure that the total number of replicas meets expectations.

[0034] The Deployment controller takes full control of the Pod lifecycle, which includes Pod creation, scheduling, updating, rollback, and deletion. Users do not need to manually manage individual Pods; all operations are implemented by modifying the Deployment configuration, and Kubernetes automatically completes the actual scheduling and execution.

[0035] When the Notebook image version or configuration needs to be updated, Deployment supports rolling updates, gradually replacing older Pods with newer ones. Throughout this process, a certain number of older Pod replicas continue to provide service until all Pods have been replaced. This achieves zero-downtime service updates, completely transparent to users.

[0036] Automated replica management ensures high service availability, supports rolling updates and seamless upgrades, provides self-healing capabilities, and automatically replaces faulty nodes.

[0037] S4. Configure the service and ingress networks, configure the Layer 4 proxy Service and the Layer 7 proxy Ingress file, expose the access interface through Service, and provide traffic management through Ingress. In step S4, the service is used to define a logical set of Pods and access policies. The service dynamically associates backend Pods through a label selector and automatically distributes the request traffic arriving at the service to healthy backend Pod instances, achieving balanced distribution of requests. The Ingress, as the entry point of the Kubernetes cluster, provides HTTP / HTTPS routing capabilities. The Ingress directs traffic to different backend services based on the domain name and URL path of the request, and handles HTTPS encryption and decryption uniformly at the Ingress level, reducing the burden on backend Pods.

[0038] Service provides stable network endpoints for service discovery and load balancing, while Ingress offers advanced network features such as traffic management and reverse proxy, making Notebook services easier to access and integrate.

[0039] The virtual IP and DNS name provided by the Service offer a fixed endpoint for frontend access, decoupling the relationship between frontend access and backend Pod instances. When a Pod instance fails, the Service's health check mechanism automatically removes it from the load balancer pool, ensuring that traffic is only directed to healthy instances. Hostname-based routing via Ingress enables sharing among multiple Jupyter Notebook instances within the same cluster. SSL certificate management, access control, and rate limiting are centrally implemented at the Ingress level, avoiding the complexity of configuring external load balancers for each service and simplifying network management through declarative configuration. The Ingress controller serves as the aggregation point for all inbound traffic, facilitating unified access log collection, monitoring metric acquisition, and traffic analysis.

[0040] S5. Adopts cloud storage to separate computing from storage; In step S5, using Kubernetes' persistent volumes and persistent declaration mechanism, the user's workspace data and analysis results are persistently stored in an independent cloud storage service. Based on the notebook pod's creation requirements, the system automatically requests and mounts the storage volume from the cloud platform, creates a compute container, and automatically allocates independent storage space to the notebook pod. The lifecycle of the storage volume is decoupled from the Pod instance. Even if the notebook container is rebuilt due to version updates or node failures, the user's data remains securely stored in the associated persistent storage. Newly started Pods can remount the same storage volume, achieving seamless data inheritance.

[0041] It enables data persistence, ensures data security, separates computing and storage, improves system flexibility and scalability, supports dynamic storage volume allocation, and allocates storage resources on demand.

[0042] Compute and storage resources can scale independently. When the number of users surges, compute nodes can be rapidly expanded without being limited by storage capacity; when data volume grows, storage space can be expanded independently without affecting computing performance. This avoids resource waste caused by binding compute and storage together, truly achieving on-demand allocation.

[0043] S6 and Kubernetes API automation management allow for declarative management of Notebook application deployment and lifecycle via the Kubernetes API.

[0044] The Kubernetes API submits a desired state file in YAML or JSON format to the server. Upon receiving the file, the API server drives the actual cluster state to converge towards the user-declared desired state. The Kubernetes API provides a complete interface for operating applications from creation to termination, including: creation and deployment, monitoring and querying, scaling, updating and rolling back, deletion and cleanup. Furthermore, the Kubernetes API automatically orchestrates the number of Jupyter Notebook containers based on the number of users, managing, storing, and analyzing data for each user's container. An external orchestrator (custom controller or script) monitors the user request queue and then dynamically creates or deletes the corresponding Notebook Deployment instances by calling the Kubernetes API.

[0045] It provides standardized API interfaces to facilitate automated operation and maintenance, supports declarative configuration management, reduces operation and maintenance complexity, and facilitates integration with existing CI / CD pipelines.

[0046] The functions include: Creation and Deployment: Submitting configuration lists for resources such as Deployments, Services, and Ingresses; Monitoring and Querying: Obtaining real-time Pod running status, logs, resource usage, etc.; Scaling Up and Down: Dynamically modifying the number of Deployment replicas via API to achieve elastic scaling based on business load; Update and Rollback: Triggering rolling updates by updating the image version, or quickly rolling back to a previous version when problems occur; Deletion and Cleanup: Terminating the application and cleaning up related resources.

[0047] This invention utilizes Docker container technology to package Jupyter Notebook and its runtime environment into a container image, achieving lightweight application encapsulation. K8s deployment management uses Deployments on a K8s cluster to define the number of replicas of the notebook application and exposes access interfaces through services. API-driven management allows for declarative management of the Notebook application's deployment and lifecycle via the K8s API.

[0048] It's worth noting that this method configures the notebook YAML file as an environment variable, mounting it to the specified directory ` / var / app / middlewares / orchestrator / templates`. It utilizes Docker container technology to package the notebook container image, achieving lightweight application encapsulation, while the `resource` configuration allows for the limitation and allocation of container resources.

[0049] Based on actual business requirements, deployments are used to manage and control the lifecycle and updates of multiple notebook pods. The deployment's YAML file is configured to package and update the image file of the intelligent analysis platform for structural monitoring data, and to configure the created notebooks and the environment variable parameters used for structural monitoring data in the deployment's YAML file.

[0050] Configure the Layer 4 proxy service and the Layer 7 proxy ingress file. Run these three services through the Deployment, and users can access the structural monitoring data intelligent analysis platform (https: / / inotebook.anxinyun.cn / ).

[0051] The system automatically orchestrates the number of Jupyter Notebook containers based on the number of users. Kubernetes manages, stores, and analyzes each user's container.

[0052] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A K8s-based Jupyter Notebook container orchestration method, characterized in that, The method comprises the following steps: S1, packaging the Notebook into a Docker image file; S2, mounting the Notebook configuration file through ConfigMap on the K8s cluster; S3, deploying the number of replicas of the Notebook application through Deployment to manage and control the life cycle and update of multiple notebook pods; S4, configuring service and ingress network, configuring four-layer proxy service and seven-layer proxy ingress file, exposing access interface through service, and providing traffic management through ingress; S5, using cloud storage mode to separate calculation and storage; S6, K8s API automatic management, managing the deployment and life cycle of the Notebook application in a declarative manner through K8s API.

2. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: In step S1, the Docker container technology is used to package the Jupyter Notebook and its running environment into a container image, that is, the Jupyter Notebook application code, all third-party libraries, system tools and configuration files dependent thereon are packaged into an independent Docker image, that is, a self-contained and portable runtime environment is created.

3. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: In step S2, ConfigMap is an AP object provided by K8s, which is used to save non-confidential data in the form of key-value pairs, and ConfigMap separates various configuration files and YAML definition file parameters of the Jupyter Notebook from the container image and manages them as independent K8s resources.

4. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: ConfigMap supports two ways to inject configuration data into a Pod, namely: As an environment variable, the key-value pairs in the ConfigMap are directly injected into the environment variable of the container in the Pod; As a volume mount, the entire configuration file or directory is mounted as a volume to the specified path of the container, completely replacing the original file in the container image.

5. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: When the Pod starts, K8s automatically mounts the configuration data defined in the ConfigMap to the specified container path, that is, the same Docker image file behaves differently in different environments without the need to rebuild the image by matching different ConfigMap.

6. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: In step S3, the Jupyter Notebook is upgraded from a single Pod instance to a robust service composed of multiple replicas with self-healing and rolling update capabilities by using the Deployment controller in K8s.

7. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: Deployment allows users to declaratively define the desired state of the Pod, including the number of Pod replicas that need to be run, and the Deployment controller fully takes over the life cycle of the Pod, including the creation, scheduling, updating, rollback and deletion of the Pod; When the image version or configuration of Notebook needs to be updated, Deployment supports configuration rolling update, which replaces the old version of Pod with the new version of Pod gradually.

8. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: In step S4, service is used to define a logical set of a group of Pods and access strategy, service dynamically associates with backend Pods through label selector, and automatically distributes request traffic arriving at service to backend healthy Pod instances, achieving balanced allocation of requests; Ingress serves as the entry point of K8s cluster, providing HTTP / HTTPS routing capability, Ingress directs traffic to different backend services according to request domain name and URL path, and uniformly processes HTTPS encryption and decryption at Ingress level, reducing the burden of backend Pods.

9. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: In step S5, through the persistent volume and persistent declaration mechanism of K8s, the user's workspace data and persistent data of analysis results are stored in an independent cloud storage service, and according to the creation requirement of notebook pod, the cloud platform is automatically applied and mounted with storage volume, and the calculation container is created, and the notebook pod is automatically allocated with independent storage space.

10. The K8s-based Jupyter Notebook container orchestration method of claim 1, wherein: K8s API submits a YAML or JSON format desired state file to the server, and after the API server receives it, it drives the actual state of the cluster to converge to the desired state declared by the user; K8s API provides complete operation interfaces for the application from birth to death, including: creation and deployment, monitoring and query, scaling, update and rollback, deletion and cleaning; And K8s API automatically arranges the corresponding number of Jupyter Notebook container numbers according to the number of users, and K8s API manages, data storage and data analysis for each user container.

Images