Automatic modeling environment management method and system based on Kubernetes

Abstract

The invention relates to the technical field of information management, in particular to an automatic modeling environment management method and system based on Kubernetes, and the automatic modeling environment management method based on Kubernetes comprises the following steps: in Kubernetes, a user logs in, and safety verification is carried out; calling a preset resource prediction model to predict and output an optimal resource configuration file according to resource configuration selected when a task is submitted: calling a Kubernetes API to create a Deployment, binding a PVC to a dynamically generated storage path, and deploying a container; a user uploads a data set, and the data set is synchronized to a preset persistent storage module through a Sidear container. According to the method, a series of work such as safety reinforcement of a whole link, prediction and output of an optimal resource configuration file through a resource prediction model, high-performance storage isolation and efficient integration of tool chains can be realized.

Description

Technical Field

[0001] The present invention relates to the technical field of information management, and in particular to a Kubernetes-based automated modeling environment management method and system. Background Art

[0002] Cloud computing is a new technology, model, and business model that promotes on-demand provisioning of information technology capabilities and improves the utilization of information technology. It provides crucial foundational support for the development of the internet, big data, artificial intelligence, and other fields, empowering traditional enterprises to transform and upgrade, helping them focus on their core businesses and more quickly adapt to a volatile and competitive market. Kubernetes has become the de facto standard for distributed cluster management systems and public / private clouds. As a distributed operating system, it reduces IT operations and maintenance costs and shortens service delivery cycles. However, with the explosive growth of containers, traditional static resource scheduling algorithms are no longer suitable for current workloads. These static allocation methods can easily lead to resource waste and performance bottlenecks, and they also present numerous security issues. Independent authentication mechanisms can lead to permission loopholes and audit blind spots, while shared path designs can cause data aliasing and leakage. These require time-consuming and labor-intensive customization by users, necessitating an automated management system to improve efficiency and optimize resource allocation. Summary of the Invention

[0003] The technical problem to be solved by this invention is that the existing Kubernetes has problems such as resource rigidity, weak security, high storage risks and tool chain fragmentation during use, and is no longer suitable for the current working environment.

[0004] To solve the above technical problems, the first aspect of the present invention adopts the following technical solution: a Kubernetes-based automated modeling environment management method, comprising the following steps:

[0005] In Kubernetes, users log in and undergo security verification. After passing the verification, they submit tasks.

[0006] Based on the resource configuration selected when the task is submitted, the preset resource prediction model is called to predict and output the optimal resource configuration file:

[0007] Call the Kubernetes API to create a Deployment, bind the PVC to the dynamically generated storage path, and deploy the container;

[0008] The user uploads a dataset, records its metadata, and after verification, synchronizes it to the preset persistent storage module through the Sidecar container.

[0009] Monitor the container in real time, and when a container crash is detected, automatically restart the container through the preset configuration.

[0010] When the present invention is working, it can achieve full-link security reinforcement, and through a series of tasks such as predicting and outputting the optimal resource configuration file through the resource prediction model, high-performance storage isolation, and efficient integration of the tool chain, it effectively solves the problems of resource rigidity, low security, and complex operation and maintenance in the existing modeling environment, and reduces the labor intensity of operators. At the same time, it adopts intelligent resource scheduling, combines machine learning prediction with genetic algorithm optimization, and adopts dynamic scheduling to maximize resource utilization and improve economic benefits.

[0011] As a preference, when a user logs in and performs security verification, the following steps are taken:

[0012] A1: User logs in, performs user identity verification, obtains the user's account and password, and verifies them. If verification passes, proceeds to step A3. If multi-factor authentication is enabled, proceeds to step A2.

[0013] A2: Send the dynamic verification code to the verification device preset by the user for dynamic verification code verification. If the verification passes, proceed to step A3;

[0014] A3: Generate a token, sign the token through a key rotation mechanism, and store the token to use as an alternative verification source for session state.

[0015] Preferably, the following steps are also included: when a user initiates an operation request, the Envoy Sidecar proxy intercepts the request and sends an authentication request to the corresponding control plane, and the control plane verifies the user's token signature and its authority scope. The control plane is provided with a permission table that maps platform roles to container operation permissions through the RBAC model. After verification, the control plane returns the corresponding authentication result.

[0016] When the present invention works, it uses fine-grained permission control and the RBAC model to map platform roles to container operation permissions, so that users with different identities are isolated from each other, and ordinary users are limited to their own related projects. At the same time, a dynamic key rotation mechanism is adopted to prevent replay supply, thereby further improving security.

[0017] As a preferred option, the following steps are also included: collecting resource indicators of each node in the Kubernetes cluster in real time, storing them in the Prometheus time series database, extracting historical task resource demand characteristics, and training the resource prediction model in combination with the proximal strategy optimization algorithm.

[0018] As an advantage, the method further comprises the following steps:

[0019] Monitor Kubernetes' resource indicators. When a Kubernetes resource indicator continuously exceeds a preset utilization threshold for a preset time, horizontal expansion is performed to create a new Pod replica. The resource configuration of the new Pod is dynamically adjusted by the resource prediction model.

[0020] Monitor the resource indicators of the Pod. When it is detected that the Pod has insufficient resources, vertical expansion is performed and the resource quota of the Pod is adjusted. The new resource quota is dynamically adjusted by the resource prediction model.

[0021] When the present invention is working, it can realize dynamic allocation of resource quotas according to multi-dimensional resources by the resource prediction model, reducing the workload of manual allocation, while greatly improving the real-time performance of allocation, thereby facilitating the maximization of resource utilization, and further realizing automated lifecycle management, and can perform flexible horizontal expansion or vertical expansion according to the actual working status, which can effectively avoid resource waste or the occurrence of performance bottlenecks.

[0022] As a best practice, call the Kubernetes API to create a Deployment, bind the PVC to the dynamically generated storage path, and deploy the container using the following steps:

[0023] B1: Dynamically generate a storage path based on the preset path generation rules and the metadata when the task is submitted;

[0024] B2: Use the Kubernetes CSI driver to dynamically create a PVC, specifying the storage class, access mode, and storage capacity.

[0025] B3: Call the Kubernetes API to create a Deployment, define the Deployment, mount the PVC, and dynamically inject path parameters.

[0026] When working, the present invention can implement standardized path management based on path generation rules, with a high degree of automation, and can realize data traceability through storage paths. Each task or user obtains an independent path, which can avoid data aliasing and improve data security. At the same time, by binding PVC to the Deployment lifecycle, storage resources can be automatically released after the task is completed, reducing the operation and maintenance burden and complexity, and improving resource utilization.

[0027] As an advantage, the method further comprises the following steps:

[0028] Build layered images, including a base layer that provides a minimal infrastructure supporting GPU acceleration, a tool layer that inherits the machine learning toolchain and data processing components, and a business layer that adds enterprise-level functionality and security auditing capabilities. After building each layer of image, perform security scans, including vulnerability scans and compliance checks.

[0029] When a user customizes a kernel, the user uploads a private algorithm library, which is automatically compiled and integrated when building the image.

[0030] When the present invention is working, by building multi-layer images and integrating business systems when building multi-layer images, the adaptability of the business system can be improved, without the need to manually install tools such as data source connectors and algorithm libraries, thereby effectively improving integration and deployment efficiency.

[0031] Preferably, when a user uploads a dataset, records the metadata of the dataset, and after verification, synchronizes it to the preset persistent storage module through the Sidecar container, the following steps are taken:

[0032] C1: The user uploads a dataset through the control plane, triggering a file upload request and recording the dataset metadata in the database table.

[0033] C2: Use the preset JupyterLab plug-in to monitor user file operation events, intercept native file operations, and redirect them to the control plane API for permission verification. After the verification passes, proceed to step C3.

[0034] C3: The dataset is synchronized to the preset persistent storage module through the Sidecar container, where the storage path is generated according to the preset path generation rules.

[0035] As an advantage, the method further comprises the following steps:

[0036] Deploy Alluxio Master and Worker between the Kubernetes Pod and the persistent storage module to implement Alluxio cache acceleration and cache hot data in the Worker.

[0037] Perform network isolation and limit communication between containers through Kubernetes NetworkPolicy;

[0038] Use Ceph distributed storage to achieve data multi-copy redundancy, configure off-site data centers, generate snapshots according to the preset synchronization time and synchronize them to the off-site data center.

[0039] When the present invention is working, it can realize efficient storage of data sets, reduce I / O latency, and is suitable for large-scale data storage. At the same time, it realizes data multi-copy redundancy through Ceph distributed storage, ensuring data security, and can realize disaster recovery by configuring off-site data centers, further improving the safety factor. It adopts a namespace-based storage isolation strategy and network isolation method to prevent cross-user data leakage, and has a good isolation effect.

[0040] To solve the above technical problems, the second aspect of the present invention adopts the following technical solution: a Kubernetes-based automated modeling environment management system, applying the above-mentioned Kubernetes-based automated modeling environment management method, comprising:

[0041] User management module, used for user login, security verification and authority allocation;

[0042] Dynamic resource scheduling module, used to predict the optimal resource profile based on the resource configuration selected by the user;

[0043] The persistent storage module is used to dynamically generate storage paths and provide data redundancy and disaster recovery.

[0044] Customized image module, used to build layered images and support custom kernels for automatic compilation and integration when building images;

[0045] The user management module, dynamic resource scheduling module, persistent storage module and customized mirror module are all deployed in Kubernetes.

[0046] The beneficial technical effects of the present invention include:

[0047] 1. The present invention can achieve full-link security reinforcement. Through a series of tasks such as predicting and outputting optimal resource configuration files through resource prediction models, high-performance storage isolation, and efficient integration of tool chains, it effectively solves problems such as resource rigidity, low security, and complex operation and maintenance in existing modeling environments, reducing the labor intensity of operators. At the same time, it adopts intelligent resource scheduling, combines machine learning prediction with genetic algorithm optimization, and adopts dynamic scheduling to maximize resource utilization and improve economic benefits.

[0048] 2. The present invention uses fine-grained permission control and maps platform roles to container operation permissions through the RBAC model, so that users with different identities are isolated from each other, and ordinary users are limited to their own related projects. At the same time, a dynamic key rotation mechanism is adopted to prevent replay supply, thereby further improving security.

[0049] 3. The present invention can realize dynamic allocation of resource quotas based on multi-dimensional resources through a resource prediction model, reducing the workload of manual allocation while greatly improving the real-time nature of allocation, thereby facilitating the maximization of resource utilization. Furthermore, it can realize automated lifecycle management and can perform flexible horizontal or vertical expansion according to actual working status, which can effectively avoid resource waste or the occurrence of performance bottlenecks.

[0050] 4. The present invention can implement standardized path management based on path generation rules, with a high degree of automation, and can realize data traceability through storage paths. Each task or user obtains an independent path, which can avoid data aliasing and improve data security. At the same time, by binding PVC to the Deployment lifecycle, storage resources can be automatically released after the task is completed, reducing the operation and maintenance burden and complexity, and improving resource utilization.

[0051] 5. The present invention can improve the adaptability of business systems by building multi-layer images and integrating business systems when building multi-layer images. There is no need to manually install tools such as data source connectors and algorithm libraries, which effectively improves integration and deployment efficiency.

[0052] 6. The present invention can realize efficient storage of data sets, reduce I / O latency, and is suitable for large-scale data storage. At the same time, it realizes data multi-copy redundancy through Ceph distributed storage, ensuring data security. By configuring off-site data centers, it can realize disaster recovery and further improve the safety factor. It adopts a namespace-based storage isolation strategy and network isolation method to prevent cross-user data leakage, and has a good isolation effect.

[0053] Other features and advantages of the present invention will be disclosed in detail in the following specific embodiments and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described below with reference to the accompanying drawings:

[0055] Figure 1 A workflow diagram for an automated modeling environment management method based on Kubernetes;

[0056] Figure 2 A workflow diagram for security verification in a Kubernetes-based automated modeling environment management method;

[0057] Figure 3 A workflow diagram for deploying containers in a Kubernetes-based automated modeling environment management method;

[0058] Figure 4A workflow diagram for storing datasets in a Kubernetes-based automated modeling environment management approach. DETAILED DESCRIPTION

[0059] The following is an explanation and description of the technical solutions of the embodiments of the present invention in conjunction with the drawings of the embodiments of the present invention. However, the following embodiments are only preferred embodiments of the present invention and are not exhaustive. Based on the embodiments in the implementation manner, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

[0060] In the following description, terms such as "inside", "outside", "up", "down", "left", "right", etc. that indicate directions or positional relationships are only used to facilitate the description of the embodiments and simplify the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operate in a specific direction. Therefore, they should not be understood as limiting the present invention.

[0061] Example 1:

[0062] See also Figure 1 This embodiment discloses a Kubernetes-based automated modeling environment management method, comprising the following steps:

[0063] In Kubernetes, users log in and undergo security verification. After passing the verification, they submit tasks.

[0064] Based on the resource configuration selected when the task is submitted, the preset resource prediction model is called to predict and output the optimal resource configuration file:

[0065] Call the Kubernetes API to create a Deployment, bind the PVC to the dynamically generated storage path, and deploy the container;

[0066] The user uploads a dataset, records its metadata, and after verification, synchronizes it to the preset persistent storage module through the Sidecar container.

[0067] Monitor the container in real time. When a container crash is detected, the container is automatically restarted through the preset configuration. Preferably, the number of automatic restarts can be set. When the restart fails continuously, an alarm is triggered and the operation and maintenance personnel are notified. In specific implementation, when data corruption is detected, it can also automatically recover from the off-site copy. When working, Prometheus collects cluster indicators, and Grafana visualizes resource utilization and abnormal events, which can achieve real-time monitoring and automatically recover to a healthy state without human intervention.

[0068] When this embodiment is working, it can achieve full-link security reinforcement, and through a series of tasks such as predicting and outputting the optimal resource configuration file through the resource prediction model, high-performance storage isolation, and efficient integration of the tool chain, it effectively solves the problems of resource rigidity, low security, and complex operation and maintenance in the existing modeling environment, and reduces the labor intensity of operators. At the same time, it adopts intelligent resource scheduling, combined with machine learning prediction and genetic algorithm optimization, and adopts dynamic scheduling to maximize resource utilization and improve economic benefits.

[0069] Please refer to Figure 2 In this embodiment, when a user logs in and performs security verification, the following steps are taken:

[0070] A1: User logs in, performs user identity verification, obtains the user's account and password, and verifies them. If verification passes, proceeds to step A3. If multi-factor authentication is enabled, proceeds to step A2.

[0071] A2: Send the dynamic verification code to the verification device preset by the user for dynamic verification code verification. If the verification passes, proceed to step A3;

[0072] A3: Generate a token, sign it through a key rotation mechanism, and store it as a backup verification source for session state. In specific implementation, the token includes the user, role, permission scope, and validity period. It is signed using HMAC-SHA256, and the key is rotated every 24 hours to prevent replay attacks.

[0073] Preferably, the following steps are also included: when a user initiates an operation request, the Envoy Sidecar proxy intercepts the request and sends an authentication request to the corresponding control plane, and the control plane verifies the user's token signature and its authority scope. The control plane is provided with a permission table that maps platform roles to container operation permissions through the RBAC model. After verification, the control plane returns the corresponding authentication result. When performing permission verification, the following steps are adopted: the user requests to access the container with a token, the control plane calls the authentication interface, verifies the validity of the token and the authority scope, and if successful, the verification is passed. When the permission is insufficient, HTTP 403 is returned. It can also record sys_user_oper_log, field: `error_msg = Permission denied`, and encrypt all operation logs and store them in Elasticsearch to support subsequent real-time retrieval and traceability.

[0074] When this embodiment works, through fine-grained permission control, the platform roles are mapped to container operation permissions through the RBAC model, so that users with different identities are isolated from each other, and ordinary users are limited to their own related projects. At the same time, a dynamic key rotation mechanism is adopted to prevent replay supply, thereby further improving security.

[0075] Preferably, step A3 further includes the following steps: identifying risks of operations, dynamically adjusting the session timeout period according to the operational risks, and re-authenticating the user when the session times out, thereby improving the security factor in a zero-trust environment.

[0076] Example 2:

[0077] This embodiment provides an automated modeling environment management method based on Kubernetes. The similarities with other embodiments are not repeated here, and the differences are described in detail below.

[0078] In this embodiment, based on the resource configuration selected when the task is submitted, the resource configuration includes various required resources, such as CPU, memory, GPU, etc. The preset resource prediction model is called to predict and output the optimal resource configuration file. In the specific implementation, the optimal resource configuration file can be predicted and output based on the PPO algorithm. When working, the current cluster resource utilization, task queue characteristics, such as the number of queued tasks and priority, and user historical operation data are input, and the Kubernetes resource configuration file, such as 4-core CPU, 16GB memory and 1 GPU, is generated by the resource prediction model.

[0079] In this embodiment, the following steps are also included: collecting resource indicators of each node in the Kubernetes cluster in real time, storing them in the Prometheus time series database, extracting historical task resource demand characteristics, and training the resource prediction model in combination with the proximal policy optimization algorithm. In the specific implementation, the state space and action space are first defined. The state space includes the current cluster resource utilization and task queue characteristics, such as the number of queued tasks and priority. The action space can also pre-allocate resource allocation plans. The reward function during training can use the following formula:

[0080] R = α × resource utilization - β × task waiting time

[0081] Among them, R is the reward value, α and β are adjustment coefficients. When working, α can be taken as 0.6 and β can be taken as 0.4 to facilitate balanced resource allocation.

[0082] Preferably, the method further comprises the following steps:

[0083] Monitor Kubernetes' resource indicators. When a Kubernetes resource indicator exceeds a preset utilization threshold for a period of time, horizontal expansion is performed, creating a new Pod replica. The resource configuration of the new Pod is dynamically adjusted by the resource prediction model. For example, if the Pod CPU utilization exceeds 80% for 5 minutes, capacity expansion is automatically triggered from 10 Pods to 20 Pods.

[0084] Monitor the various resource indicators of the Pod. When it is detected that the Pod has insufficient resources, perform vertical expansion and adjust the Pod's resource quota. The new resource quota is dynamically adjusted by the resource prediction model. For example, when insufficient memory is detected, the Pod's memory limit is dynamically adjusted from 16Gi to 32Gi. Preferably, a sliding window algorithm can be used to set an appropriate window size, predict resource requirements, and combine genetic algorithms to optimize resource allocation.

[0085] When this embodiment is working, it can realize dynamic allocation of resource quotas according to multi-dimensional resources by the resource prediction model, reducing the workload of manual allocation, while greatly improving the real-time performance of allocation, thereby facilitating the maximization of resource utilization, and further realizing automated lifecycle management, and can perform elastic horizontal expansion or vertical expansion according to the actual working status, which can effectively avoid resource waste or performance bottlenecks.

[0086] Preferably, it also includes GPU compatibility detection and automatic matching of NVIDIA driver versions. For example, Ampere architecture GPUs are bound to CUDA 11+ to improve compatibility.

[0087] Example 3:

[0088] See also Figure 3 This embodiment provides an automated modeling environment management method based on Kubernetes. The similarities with other embodiments are not repeated here, and the differences are described in detail below.

[0089] In this example, the Kubernetes API is called to create a Deployment, and the PVC is bound to the dynamically generated storage path. When deploying the container, the following steps are taken:

[0090] B1: Dynamically generate a storage path based on the preset path generation rules and the metadata when the task is submitted. For example, the storage path can be generated based on the user ID or task ID. The specific code is as follows:

[0091] ```Python

[0092] path=f" / nfs / user_{user_id} / task_{task_id} / "

[0093] ```;

[0094] B2: Use the Kubernetes CSI driver to dynamically create a PVC, specifying the storage class, access mode, and storage capacity. Some code references are as follows:

[0095]

[0096] B3: Call the Kubernetes API to create a Deployment, define the Deployment, mount the PVC, and dynamically inject path parameters.

[0097] When this embodiment is working, it can implement standardized path management based on path generation rules, with a high degree of automation. It can also realize data traceability through storage paths. Each task or user obtains an independent path, which can avoid data aliasing and improve data security. At the same time, by binding PVC to the Deployment lifecycle, storage resources can be automatically released after the task is completed, reducing the operation and maintenance burden and complexity and improving resource utilization.

[0098] As a further improvement to this embodiment, if cluster resources are insufficient, the cloud vendor API, such as AWS EB2 AutoScaling Group, is triggered to automatically scale out nodes to avoid performance bottlenecks.

[0099] During the specific implementation, the following steps are also included:

[0100] Building a layered image includes building a base layer that builds a minimal basic environment that supports GPU acceleration, a tool layer that inherits the machine learning tool chain and data processing components, and a business layer that adds enterprise-level functions and security audit capabilities. After building each layer of the image, a security scan is performed, which includes vulnerability scanning and compliance checks. In the specific implementation, the base image is equipped with Ubuntu 20.04, Python 3.8 and CUDA 11.6, which can be used to support NVIDIA GPU acceleration. The working layer is pre-installed with MLflow, i.e. model tracking, Airflow, i.e. workflow orchestration, and Kafka-Python, i.e. data stream processing. The business layer integrates enterprise data lake connectors such as AWS S3, Snowflake and security audit SDK.

[0101] When a user customizes a kernel, the user uploads a private algorithm library, such as a PyTorch custom operator, which is automatically compiled and integrated when building the image.

[0102] When this embodiment is working, by building a multi-layer image and integrating the business system when building the multi-layer image, the adaptability of the business system can be improved. There is no need to manually install tools such as data source connectors and algorithm libraries, which effectively improves the integration and deployment efficiency.

[0103] When working, the code triggers the Jenkins pipeline to automatically build the image and push it to the private repository. Optimally, Trivy can be integrated to detect CVE vulnerabilities and block the deployment of high-risk images. During specific implementation, a lightweight design can be used to compress the base image size to less than 500MB, and Alpine Linux and minimized Python environments can be used.

[0104] Example 4:

[0105] See also Figure 4 This embodiment provides an automated modeling environment management method based on Kubernetes. The similarities with other embodiments are not repeated here, and the differences are described in detail below.

[0106] In this embodiment, when a user uploads a dataset, records the metadata of the dataset, and after verification, synchronizes it to the preset persistent storage module through the Sidecar container, the following steps are taken:

[0107] C1: The user uploads a dataset through the control plane, triggering a file upload request and recording the dataset metadata in a database table. In specific implementations, the control plane records the dataset metadata in the `notebook_file` table, including the file name, user ID, storage path, and permissions.

[0108] C2: Use the preset JupyterLab plug-in to monitor user file operation events, intercept native file operations, and redirect them to the control plane API for permission verification. After the verification passes, proceed to step C3.

[0109] C3: The dataset is synchronized to the preset persistent storage module through the Sidecar container. The storage path is generated according to the preset path generation rules. An independent storage path is generated based on the user ID and container ID to ensure data isolation. In specific implementation, storage quotas can also be dynamically allocated. For example, the disk size is defined in the `cs_resource` table, and storage resources are automatically bound through Kubernetes PVC to achieve elastic expansion and contraction of storage resources. During operation, the PVC capacity can be dynamically adjusted according to the `disk` field value.

[0110] Preferably, the method further comprises the following steps:

[0111] Deploy Alluxio Master and Worker between the Kubernetes Pod and the persistent storage module to implement Alluxio cache acceleration and cache hot data in the Worker.

[0112] Perform network isolation and limit communication between containers through Kubernetes NetworkPolicy;

[0113] Ceph distributed storage is used to achieve multi-copy data redundancy, configure off-site data centers, generate snapshots according to the preset synchronization time and synchronize them to the off-site data centers. During operation, when the main data center fails and services need to be restored quickly, the disaster recovery system can load snapshots from off-site storage, such as AWS S3, and rebuild the Kubernetes cluster within 30 minutes. User session data is restored from the off-site copy of the Redis Cluster, which can shorten the service interruption time to within 2 minutes.

[0114] When this embodiment is working, it can achieve efficient storage of data sets, reduce I / O latency, and is suitable for large-scale data storage. At the same time, it realizes data multi-copy redundancy through Ceph distributed storage, ensuring data security, and can achieve disaster recovery by configuring off-site data centers, further improving the safety factor. It adopts a namespace-based storage isolation strategy and network isolation method to prevent cross-user data leakage, and has a good isolation effect.

[0115] Example 4:

[0116] This embodiment provides an automated modeling environment management system based on Kubernetes, applying the automated modeling environment management method based on Kubernetes described in the above embodiment, including:

[0117] User management module, used for user login, security verification and authority allocation;

[0118] Dynamic resource scheduling module, used to predict the optimal resource profile based on the resource configuration selected by the user;

[0119] The persistent storage module is used to dynamically generate storage paths and provide data redundancy and disaster recovery.

[0120] Customized image module, used to build layered images and support custom kernels for automatic compilation and integration when building images;

[0121] The user management module, dynamic resource scheduling module, persistent storage module and customized mirror module are all deployed in Kubernetes. When working, they work together through standardized interfaces to support high concurrency, high availability and elastic expansion.

[0122] In specific implementation, the persistent storage module can adopt the NFS solution to support cross-platform sharing. By adopting the technical solution in the above embodiment, while ensuring universality, it can achieve multi-level directory isolation, avoid multi-user data aliasing under shared storage, and have a high safety factor.

[0123] The beneficial technical effects of this embodiment include: the present invention can achieve full-link security reinforcement, output the optimal resource configuration file through resource prediction model prediction, high-performance storage isolation, and efficient integration of tool chains, etc., effectively solving the problems of resource rigidity, low security and complex operation and maintenance in the existing modeling environment, reducing the labor intensity of operators, and at the same time adopting intelligent resource scheduling, combining machine learning prediction and genetic algorithm optimization, and adopting dynamic scheduling to maximize resource utilization and improve economic benefits.

[0124] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Those skilled in the art will understand that the present invention includes, but is not limited to, the contents described in the drawings and the above specific embodiments. Any modifications that do not deviate from the functional and structural principles of the present invention are intended to be included within the scope of the claims.

Claims

1. A Kubernetes-based automated modeling environment management method, characterized in that: The following steps are involved: In Kubernetes, users log in and undergo security verification. After passing the verification, they submit tasks. Based on the resource configuration selected when the task is submitted, the preset resource prediction model is called to predict and output the optimal resource configuration file: Call the Kubernetes API to create a Deployment, bind the PVC to the dynamically generated storage path, and deploy the container; The user uploads a dataset, records its metadata, and after verification, synchronizes it to the preset persistent storage module through the Sidecar container. Monitor the container in real time, and when a container crash is detected, automatically restart the container through the preset configuration.

2. The method for managing an automated modeling environment based on Kubernetes according to claim 1, wherein: When a user logs in and performs security verification, the following steps are taken: A1: User logs in, performs user identity verification, obtains the user's account and password, and verifies them. If verification passes, proceeds to step A3. If multi-factor authentication is enabled, proceeds to step A2. A2: Send the dynamic verification code to the verification device preset by the user for dynamic verification code verification. If the verification passes, proceed to step A3; A3: Generate a token, sign the token through a key rotation mechanism, and store the token to use as an alternative verification source for session state.

3. The method for managing an automated modeling environment based on Kubernetes according to claim 2, wherein: It also includes the following steps: when a user initiates an operation request, the Envoy Sidecar proxy intercepts the request and sends an authentication request to the corresponding control plane. The control plane verifies the user's token signature and its permission scope. The control plane is provided with a permission table that maps platform roles to container operation permissions through the RBAC model. After verification, the control plane returns the corresponding authentication result.

4. The method for managing an automated modeling environment based on Kubernetes according to claim 2, wherein: The following steps are also included: collecting resource indicators of each node in the Kubernetes cluster in real time, storing them in the Prometheus time series database, extracting historical task resource demand characteristics, and training the resource prediction model in combination with the proximal strategy optimization algorithm.

5. The method for managing an automated modeling environment based on Kubernetes according to claim 1, wherein: The following steps are also included: Monitor Kubernetes' resource indicators. When a Kubernetes resource indicator continuously exceeds a preset utilization threshold for a preset time, horizontal expansion is performed to create a new Pod replica. The resource configuration of the new Pod is dynamically adjusted by the resource prediction model. Monitor the resource indicators of the Pod. When it is detected that the Pod has insufficient resources, vertical expansion is performed and the resource quota of the Pod is adjusted. The new resource quota is dynamically adjusted by the resource prediction model.

6. The method for managing an automated modeling environment based on Kubernetes according to claim 1, wherein: Call the Kubernetes API to create a Deployment, bind the PVC to the dynamically generated storage path, and deploy the container using the following steps: B1: Dynamically generate a storage path based on the preset path generation rules and the metadata when the task is submitted; B2: Use the Kubernetes CSI driver to dynamically create a PVC, specifying the storage class, access mode, and storage capacity. B3: Call the Kubernetes API to create a Deployment, define the Deployment, mount the PVC, and dynamically inject path parameters.

7. The method for managing an automated modeling environment based on Kubernetes according to claim 1, wherein: The following steps are also included: Build layered images, including a base layer that provides a minimal infrastructure supporting GPU acceleration, a tool layer that inherits the machine learning toolchain and data processing components, and a business layer that adds enterprise-level functionality and security auditing capabilities. After building each layer of image, perform security scans, including vulnerability scans and compliance checks. When a user customizes a kernel, the user uploads a private algorithm library, which is automatically compiled and integrated when building the image.

8. The method for managing an automated modeling environment based on Kubernetes according to claim 1, wherein: When a user uploads a dataset, records the metadata of the dataset, and after verification, synchronizes it to the preset persistent storage module through the Sidecar container, the following steps are taken: C1: The user uploads a dataset through the control plane, triggering a file upload request and recording the dataset metadata in the database table. C2: Use the preset JupyterLab plug-in to monitor user file operation events, intercept native file operations, and redirect them to the control plane API for permission verification. After the verification passes, proceed to step C3. C3: The dataset is synchronized to the preset persistent storage module through the Sidecar container, where the storage path is generated according to the preset path generation rules.

9. The method for managing an automated modeling environment based on Kubernetes according to claim 1, wherein: The following steps are also included: Deploy Alluxio Master and Worker between the Kubernetes Pod and the persistent storage module to implement Alluxio cache acceleration and cache hot data in the Worker. Perform network isolation and limit communication between containers through Kubernetes NetworkPolicy; Use Ceph distributed storage to achieve data multi-copy redundancy, configure off-site data centers, generate snapshots according to the preset synchronization time and synchronize them to the off-site data center.

10. The Kubernetes-based automated modeling environment management system according to claim 1, wherein the Kubernetes-based automated modeling environment management method according to any one of claims 1 to 9 is applied, characterized in that: include: User management module, used for user login, security verification and authority allocation; Dynamic resource scheduling module, used to predict the optimal resource profile based on the resource configuration selected by the user; The persistent storage module is used to dynamically generate storage paths and provide data redundancy and disaster recovery. Customized image module, used to build layered images and support custom kernels for automatic compilation and integration when building images; The user management module, dynamic resource scheduling module, persistent storage module and customized mirror module are all deployed in Kubernetes.

Images