Large model deployment and service method and device for realizing intelligent customer service based on container arrangement, equipment and medium

Abstract

The invention discloses a large model deployment and service method and device for realizing intelligent customer service based on container arrangement, equipment and a medium, and relates to the field of artificial intelligence, and the method comprises the steps: constructing a standardized container mirror image which comprises an inference engine, a model acceleration library and a model encryption and decryption assembly and is adaptive to an intelligent customer service scene; establishing a graphic processing unit resource management system in the container arrangement cluster; storing model data of the intelligent customer service model to a target storage system, and determining a mounting mechanism corresponding to the model data; generating a container arrangement and deployment file based on a graphic processing unit resource management system, the operation demand of the standardized container mirror image and a mounting mechanism corresponding to the model data; and submitting the container arrangement deployment file to a container arrangement platform, so that the container arrangement platform carries out intelligent customer service model deployment, and the deployed intelligent customer service model is utilized to receive a user request and return a corresponding target response. According to the method, the deployment complexity is reduced, and the model deployment efficiency and stability are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a large model deployment and service method, device and medium for intelligent customer service based on container orchestration. BACKGROUND

[0002] In the process of digital transformation of enterprise services, intelligent customer service systems have become the core support for improving customer service efficiency and reducing labor costs. They need to be based on large language models (LLMs) to realize complex user intent understanding, multi-round dialogue interaction, domain knowledge question answering and other core functions. Especially in scenarios such as e-commerce promotions, after-sales peaks, and holiday consultation surges, high requirements are placed on the response speed, service stability and concurrent processing capacity of the model. The model deployment of traditional intelligent customer service systems relies on fixed physical servers or virtual machines, which has strong environmental dependence, low resource utilization, low deployment efficiency, and lacks flexibility.

[0003] In recent years, containerization technologies such as Docker and container orchestration systems such as Kubernetes have been gradually introduced into intelligent customer service deployment scenarios. However, in the face of the unique deployment needs of LLMs in intelligent customer service scenarios, existing technologies still face multiple challenges: 1. Large model size: the number of model parameters can reach tens of billions or even hundreds of billions, the model weight file is large, and the loading time is long, which puts higher requirements on storage and network bandwidth; 2. High computing power demand: large model training and inference often require multiple GPUs or multi-node parallel computing, how to efficiently allocate GPU resources and implement task scheduling becomes a key issue; 3. Environment consistency problem: there may be differences in operating systems and library versions between different nodes, affecting the consistency of model deployment; 4. Elastic scaling and high availability: traditional deployment methods are difficult to quickly scale in the face of sudden traffic or model request surges; 5. Low resource utilization: idle resources in the cluster are not effectively utilized, resulting in resource waste. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a large model deployment and service method, device and medium for intelligent customer service based on container orchestration, which reduces the deployment complexity and improves the model deployment efficiency and stability in the intelligent customer service scenario through containerization encapsulation, efficient GPU resource scheduling and automatic container orchestration deployment.

[0005] In a first aspect, the present application provides a large model deployment and service method for intelligent customer service based on container orchestration, comprising:

[0006] A standardized container image is constructed that includes an inference engine, a model acceleration library, and model encryption / decryption components, and is adapted to intelligent customer service scenarios; the standardized container image is used to provide a unified and secure operating environment for service instances;

[0007] A graphics processing unit resource management system is established in the container orchestration cluster; the graphics processing unit resource management system is used to convert physical graphics processing units into logical resources that can be intelligently scheduled;

[0008] The model data of the intelligent customer service model is saved to the target storage system, and the mounting mechanism corresponding to the model data is determined. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism is that when the service instance starts, the required target model data is downloaded and verified from the target storage system by initializing the container. The shared storage volume that supports multi-node read and write is mounted to the specified directory of the service instance through the persistent storage declaration mechanism of the container orchestration platform, so as to realize the reliable distribution and multi-instance sharing of the target model data.

[0009] Based on the graphics processing unit resource management system, the operational requirements of the standardized container image, and the mounting mechanism corresponding to the model data, a container orchestration and deployment file is generated; the container orchestration and deployment file includes the resource allocation strategy of the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs;

[0010] The container orchestration deployment file is submitted to the container orchestration platform so that the container orchestration platform can deploy the intelligent customer service model, so as to receive user requests and return the corresponding target response using the deployed intelligent customer service model.

[0011] Optionally, establishing a graphics processing unit resource management system includes:

[0012] The graphics processing unit (GPU) device management plugin identifies and collects the GPU resource information corresponding to each node in the container orchestration cluster, and constructs a manageable resource pool based on the GPU resource information; the GPU resource information includes the hardware model, video memory capacity, and operating status of the GPU.

[0013] Based on the resource information of the graphics processing unit, add corresponding type identifiers to each node to establish a resource classification system within the resource pool;

[0014] Configure resource scheduling strategies according to the resource classification system; the resource scheduling strategies are used to realize the allocation mode of graphics processing units being shared among multiple service instances or exclusively used by a single service instance according to the computing requirements of intelligent customer service reasoning tasks.

[0015] Optionally, saving the model data of the intelligent customer service model to the target storage system includes:

[0016] Save the trained model data, including the weight file, parameter configuration file, and word segmenter vocabulary of the intelligent customer service model, to the target storage system.

[0017] A unique checksum is generated for the model data so that checksum verification can be performed in subsequent operations on the model data based on the checksum.

[0018] The model data is judged according to a preset sensitivity judgment standard. For model data that meets the preset sensitivity, the encrypted storage component of the container orchestration platform is used to encrypt and store the model data that meets the preset non-sensitivity. For model data that meets the preset non-sensitivity, the shared configuration component of the container orchestration platform is used to manage the model data that meets the preset non-sensitivity.

[0019] Optionally, the step of downloading and verifying the required target model data from the target storage system by initializing the container includes:

[0020] The initialization container is used to obtain the latest version of the target model data from the target storage system through the application programming interface; the initialization container supports breakpoint resume technology to transmit model data in the event of network instability;

[0021] The acquired target model data is subjected to hash value verification to obtain the verification result. If the verification result indicates that the verification has failed, the initialization container is used to reacquire the latest version of the target model data until the verification result indicates that the verification has succeeded.

[0022] Optionally, the deployment of the intelligent customer service model, to receive user requests and return corresponding target responses using the deployed intelligent customer service model, includes:

[0023] Based on the resource allocation strategy and graphics processing unit resource management system in the container orchestration deployment file, select a target node that meets the requirements, create an intelligent customer service instance on the target node and complete model loading, so that the intelligent customer service instance can receive user requests, perform model inference calculations to generate corresponding target responses, and return the target responses to the user.

[0024] Optionally, the method for deploying and providing services for large-scale intelligent customer service models based on container orchestration further includes:

[0025] Configure a health check probe during service operation to continuously monitor the running status and model loading of the intelligent customer service instance.

[0026] If an instance anomaly is determined based on the running status of the intelligent customer service instance and the model loading status, the intelligent customer service instance is controlled to stop processing user requests and perform fault recovery operations.

[0027] Optionally, the process of adjusting instance size using dynamic scaling rules includes:

[0028] Real-time collection of performance metrics data for intelligent customer service; the performance metrics data include service request volume, response latency, and computing resource utilization.

[0029] When the service request volume or response latency reaches the preset high load threshold, the computing resource utilization meets the preset high occupancy condition, and the number of service instances is lower than the preset upper limit, the container orchestration platform automatically increases the number of service instances according to preset rules. When the service request volume drops to the preset low load threshold, the computing resource utilization meets the preset low occupancy condition, and the number of service instances is higher than the preset lower limit, the container orchestration platform automatically reduces the number of service instances according to preset rules.

[0030] Secondly, this application provides a large-scale deployment and service device for intelligent customer service based on container orchestration, including:

[0031] The image building module is used to build standardized container images that include an inference engine, a model acceleration library, and model encryption / decryption components, and are adapted to intelligent customer service scenarios; the standardized container images are used to provide a unified and secure operating environment for service instances;

[0032] The management system construction module is used to establish a graphics processing unit resource management system in a container orchestration cluster; the graphics processing unit resource management system is used to convert physical graphics processing units into logical resources that can be intelligently scheduled.

[0033] The data storage module is used to save the model data of the intelligent customer service model to the target storage system and determine the mounting mechanism corresponding to the model data. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism is that when the service instance starts, the required target model data is downloaded from the target storage system by initializing the container and verified. The shared storage volume that supports multi-node read and write is mounted to the specified directory of the service instance through the persistent storage declaration mechanism of the container orchestration platform, so as to realize the reliable distribution and multi-instance sharing of the target model data.

[0034] The file generation module is used to generate container orchestration and deployment files based on the graphics processing unit resource management system, the runtime requirements of the standardized container image, and the mounting mechanism corresponding to the model data. The container orchestration and deployment files include the resource allocation strategy of the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs.

[0035] The service module is used to submit the container orchestration deployment file to the container orchestration platform so that the container orchestration platform can deploy the intelligent customer service model, so as to receive user requests and return the corresponding target response using the deployed intelligent customer service model.

[0036] Thirdly, this application provides an electronic device, comprising:

[0037] Memory, used to store computer programs;

[0038] A processor is used to execute the computer program to implement the aforementioned method for deploying and serving large-scale intelligent customer service models based on container orchestration.

[0039] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned method for deploying and serving large-scale intelligent customer service models based on container orchestration.

[0040] This application constructs a standardized container image that includes an inference engine, a model acceleration library, and model encryption / decryption components, and is adapted to intelligent customer service scenarios. This standardized container image provides a unified and secure operating environment for service instances. A graphics processing unit resource management system is established within the container orchestration cluster. This system converts physical graphics processing units into intelligently schedulable logical resources. The model data of the intelligent customer service model is saved to a target storage system, and the mounting mechanism corresponding to the model data is determined. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism involves downloading and verifying the required target model data from the target storage system by initializing the container when the service instance starts. The persistent storage declaration mechanism of the container orchestration platform mounts a shared storage volume that supports multi-node read / write operations to a specified directory of the service instance to achieve reliable distribution and multi-instance sharing of target model data. Based on the graphics processing unit resource management system, the operational requirements of the standardized container image, and the mounting mechanism corresponding to the model data, a container orchestration deployment file is generated. This file includes the resource allocation strategy for the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs. The container orchestration deployment file is then submitted to the container orchestration platform so that the platform can deploy the intelligent customer service model to receive user requests and return corresponding target responses. As shown above, this application stores model data in an independent target storage system. When a service instance starts, it downloads and verifies the model data by initializing the container. Then, through the persistent storage declaration mechanism of the container orchestration platform, it mounts a shared storage volume that supports multi-node read / write to the same directory of all service instances, achieving "download once, mount and share multiple times" for model data. This reduces network bandwidth pressure and avoids the lengthy waiting and wasted time caused by each instance independently downloading the model, thus solving the deployment bottleneck caused by the large size of the model. Facing the demand for large-scale model inference requiring significant GPU computing power, this application constructs a GPU resource management system within the container orchestration cluster. This system can convert physical graphics processing units into intelligently schedulable logical resources, enabling intelligent customer service tasks to be intelligently scheduled to run on the most suitable GPU nodes. This ensures that high-computing-power tasks can obtain stable and matched computing resources, solving the key problems of high computing power requirements and difficult resource allocation. All dependencies required by the intelligent customer service model, including the inference engine, model acceleration library, and security components, are encapsulated into a standardized container image. This ensures that the model's operating environment remains completely consistent throughout the entire process from development to testing to production, fundamentally solving the deployment dilemma caused by strong environment dependencies.The container orchestration and deployment files clearly define the resource allocation strategy, service access interface, and dynamic scaling rules for automatically adjusting the number of service instances based on actual needs. During off-peak periods, the number of instances is automatically reduced to release resources, and during peak periods, the system can quickly scale up, thereby revitalizing idle resources in the cluster and significantly improving the average resource utilization of the entire infrastructure. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 This application discloses a flowchart of a method for deploying and serving a large-scale intelligent customer service model based on container orchestration.

[0043] Figure 2 This is a schematic diagram of a large-scale model deployment and service device for intelligent customer service based on container orchestration disclosed in this application;

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device disclosed in this application. Detailed Implementation

[0045] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] To address the unique deployment requirements of models in intelligent customer service scenarios, existing technologies still face the following challenges: 1. Large model weight files place higher demands on storage and network bandwidth; 2. Efficiently allocating GPU resources and implementing task scheduling are crucial; 3. Differences in operating systems and library versions between different nodes may affect the consistency of model deployment; 4. Traditional deployment methods struggle to scale quickly in the face of sudden traffic surges or model request spikes; 5. Ineffective utilization of idle resources in the cluster leads to resource waste. To address these challenges, this application provides a method for deploying and serving large models for intelligent customer service based on container orchestration. Through containerized encapsulation, efficient GPU resource scheduling, and automated deployment via container orchestration, it achieves high environmental consistency, high resource utilization, high deployment efficiency, and strong elastic scalability for large model deployments, reducing deployment complexity and improving the efficiency and stability of model deployment in intelligent customer service scenarios.

[0047] See Figure 1 As shown in the figure, this application discloses a method for deploying and serving a large-scale intelligent customer service model based on container orchestration, including:

[0048] Step S11: Construct a standardized container image that includes an inference engine, a model acceleration library, and model encryption / decryption components, and is adapted to the intelligent customer service scenario; the standardized container image is used to provide a unified and secure operating environment for service instances.

[0049] In this embodiment, to achieve efficient, stable, and secure AI inference services, a complete AI inference runtime environment can be encapsulated using Docker technology. For example, mainstream inference engines such as TensorFlow Serving and PyTorch Serving can be integrated into the runtime environment using Docker technology to efficiently handle various complex inference tasks, providing a solid foundation for the rapid deployment and operation of models. Advanced model acceleration libraries such as ONNX Runtime and TensorRT are also installed to improve the inference speed and efficiency of the models.

[0050] Meanwhile, during the image building process, a multi-stage build process can be adopted. Through refined layering and optimization strategies, unnecessary files and dependencies can be gradually eliminated, effectively reducing the image size. A smaller image size not only reduces storage resource consumption but also speeds up image transmission and deployment, improving the overall system response efficiency. Furthermore, building an image with model format compatibility enables seamless loading of various common model formats such as ONNX, TorchScript, and SavedModel. Whether the model is exported from different development frameworks or has undergone various optimization processes, it can run smoothly in this standardized inference framework image without the need for cumbersome format conversions and adaptations, greatly improving the flexibility and convenience of model deployment.

[0051] In addition, considering the security of the inference service, a model encryption / decryption component and an access control module can be built into the image. The model encryption / decryption component can encrypt sensitive model data to prevent unauthorized access and tampering during transmission and storage. The access control module restricts access to the inference service through a strict permission management mechanism, ensuring that only authorized users or systems can use the inference service, thereby guaranteeing the security of the inference service and the confidentiality of data.

[0052] Step S12: Establish a graphics processing unit resource management system in the container orchestration cluster; the graphics processing unit resource management system is used to convert physical graphics processing units into logical resources that can be intelligently scheduled.

[0053] In this embodiment, a graphics processing unit (GPU) resource management system is established within the container orchestration cluster. Specifically, this includes: identifying and collecting GPU resource information for each node in the container orchestration cluster through a GPU device management plugin; and constructing a manageable resource pool based on this GPU resource information. The GPU resource information includes, but is not limited to, the hardware model, memory capacity, and operating status of the GPU. For example, in the system environment, the NVIDIA Device Plugin component is installed to implement GPU resource pooling. The installation of this component lays the foundation for flexible allocation and management of GPU resources, enabling dynamic allocation and adjustment of GPU resources according to actual needs, thus improving GPU resource utilization efficiency. Then, corresponding type identifiers are added to each node based on the GPU resource information to establish a resource classification system within the resource pool. Next, a resource scheduling strategy is configured according to the resource classification system. This resource scheduling strategy is used to implement an allocation mode where GPUs are shared among multiple service instances or exclusively used by a single service instance, based on the computational requirements of the intelligent customer service inference task. A well-configured resource scheduling strategy can ensure that GPU resources are optimally matched with the corresponding tasks and models during the allocation process, thereby fully leveraging the performance advantages of the GPU and avoiding resource waste and performance bottlenecks.

[0054] Meanwhile, during actual model execution, the CUDA_VISIBLE_DEVICES environment variable allows for precise control over the model's device binding. This environment variable accurately specifies which GPU devices the model will run on, providing a clear device reference and ensuring the model's stability and efficiency. To further improve inference performance, the Mixed Precision inference mode can be applied to the system. This inference mode effectively reduces computational load and memory consumption while maintaining computational accuracy, accelerating inference speed and enabling the model to run more efficiently when processing large-scale data, providing users with faster response and a better experience.

[0055] Furthermore, to monitor the GPU's operational status in real time, the Kubelet GPU monitoring module can be integrated. This module can collect key metrics such as memory usage and GPU utilization in real time, providing timely feedback of this important information to system administrators. By monitoring these metrics in real time, administrators can promptly identify GPU resource usage, enabling them to make appropriate adjustments and optimizations to ensure stable system operation and high performance.

[0056] Step S13: Save the model data of the intelligent customer service model to the target storage system and determine the mounting mechanism corresponding to the model data. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism is that when the service instance starts, the required target model data is downloaded from the target storage system by initializing the container and verified. The shared storage volume that supports multi-node read and write is mounted to the specified directory of the service instance through the persistent storage declaration mechanism of the container orchestration platform, so as to realize the reliable distribution and multi-instance sharing of the target model data.

[0057] In this embodiment, to achieve efficient management and secure storage of model data, the model data of the intelligent customer service model can be saved to the target storage system. Specifically, this can include saving the trained model data, including the weight file, parameter configuration file, and word segmenter vocabulary of the intelligent customer service model, to the target storage system. A unique checksum is generated for the model data so that verification can be performed on the model data in subsequent operations. Then, the model data is judged according to a preset sensitivity judgment standard. For model data that meets the preset sensitivity, the encrypted storage component of the container orchestration platform is used to encrypt and store the model data that meets the preset non-sensitivity standard. For model data that meets the preset non-sensitivity standard, the shared configuration component of the container orchestration platform is used to manage the model data that meets the preset non-sensitivity standard.

[0058] For example, crucial model data such as quantized and compressed model weight files, Beam Search parameter configurations, and the word segmenter vocabulary are uniformly stored in the highly reliable and stable MinIO object storage system. An MD5 checksum mechanism is introduced, generating a unique MD5 checksum for each piece of model data to rigorously verify it at every stage of file storage and transmission. Whether writing data or reading or transmitting it later, the system compares it against the pre-generated MD5 checksum. If a mismatch is found, it indicates that the data has been corrupted or tampered with, allowing for timely measures to ensure the security and integrity of core model data.

[0059] For configuration file management, a hybrid management model of ConfigMap + Secrets can be adopted. This model can classify and manage parameters according to their sensitivity. Parameters involving sensitive information such as system security and privacy are encrypted and stored using Kubernetes Secrets, while non-sensitive configuration file parameters are managed using ConfigMaps. This facilitates sharing and use among different nodes in the cluster, improving the efficiency and flexibility of configuration management.

[0060] It should be noted that downloading and verifying the required target model data from the target storage system through the initialization container can include: using the initialization container to obtain the latest version of the target model data from the target storage system via the application programming interface; wherein, the initialization container supports breakpoint resumption technology to transmit model data in the event of network instability. The obtained target model data is then subjected to hash value verification to obtain the verification result. If the verification result indicates a failure, the initialization container is used to re-obtain the latest version of the target model data until the verification result indicates a successful verification.

[0061] Meanwhile, to meet security requirements in different environments, a dual authentication mechanism of HTTPS / private CA certificates can be applied during the process of retrieving target model data from the target storage system. HTTPS (Hypertext Transfer Protocol Secure) encrypts data transmission, preventing data theft or tampering during transmission. For scenarios with even higher security requirements, a private CA certificate (CA, Certificate Authority) is used for identity authentication and encrypted communication, further enhancing system security and ensuring that the model retrieval task proceeds smoothly in a secure and reliable environment.

[0062] In a Kubernetes cluster environment, this embodiment employs a persistent storage mapping scheme to achieve efficient management and flexible utilization of storage resources. First, dynamic PV / PVC resources (PV, Persistent Volume; PVC, Persistent Volume Claim) are defined in the Kubernetes cluster. By defining PVCs, users can specify parameters such as required storage capacity and access mode according to their needs. In this way, the system can dynamically allocate and manage storage resources based on user requirements, improving the utilization and flexibility of storage resources.

[0063] The defined dynamic PV / PVC resources are mounted to the / models directory of the inference container using the ReadWriteMany access mode. The ReadWriteMany access mode allows multiple nodes to perform read and write operations on storage resources simultaneously. By mounting the storage resources to the specified directory of the inference container, the application within the container can access and use the model data stored in the persistent volume, ensuring the smooth operation of the model loading and inference process.

[0064] Meanwhile, to enable automatic scaling of storage resources, StorageClass can be configured. StorageClass is a mechanism in Kubernetes used to manage storage resources, defining the categories and attributes of storage resources, including storage type and reclamation policies. By properly configuring StorageClass, the system can automatically adjust the size of storage resources according to actual storage needs, and automatically expand storage capacity when storage resources are insufficient to meet the needs of business growth.

[0065] Furthermore, to further improve the system's I / O (Input / Output) performance, EmptyDir temporary storage can be used to cache frequently accessed data. Cached in EmptyDir during system operation reduces the number of accesses to persistent storage, improving data read and write speeds. This tiered storage architecture fully leverages the advantages of different storage media, ensuring both data security and durability while enhancing system I / O performance, thus guaranteeing efficient model inference.

[0066] Step S14: Generate a container orchestration and deployment file based on the graphics processing unit resource management system, the operational requirements of the standardized container image, and the mounting mechanism corresponding to the model data; the container orchestration and deployment file includes the resource allocation strategy of the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs.

[0067] In this embodiment, the Deployment strategy can be clearly and comprehensively defined by writing a Deployment YAML file. During this process, parameters related to resources.requests and resources.limits are set. This precise configuration method enables accurate control over CPU (Central Processing Unit) and GPU resource quotas, ensuring the rational allocation and efficient utilization of resources.

[0068] The process of adjusting instance size using dynamic scaling rules can include: real-time collection of performance metrics data for the intelligent customer service; these performance metrics include, but are not limited to, service request volume, response latency, and computing resource utilization. When the service request volume or response latency reaches a preset high load threshold, the computing resource utilization meets a preset high utilization condition, and the number of service instances is lower than a preset upper limit, the container orchestration platform automatically increases the number of service instances according to preset rules. When the service request volume drops to a preset low load threshold, the computing resource utilization meets a preset low utilization condition, and the number of service instances is higher than a preset lower limit, the container orchestration platform automatically reduces the number of service instances according to preset rules.

[0069] Furthermore, to effectively expose services, a Service can be used to publicly display RESTful API endpoints, enabling them to be accessed and invoked by external systems or clients. Simultaneously, an Istio gateway can be configured to implement traffic coloring and canary releases. Traffic coloring clearly identifies and distinguishes traffic from different sources or characteristics, while canary releases gradually push new versions of the service to users without affecting overall system stability, effectively reducing release risks.

[0070] Step S15: Submit the container orchestration deployment file to the container orchestration platform so that the container orchestration platform can deploy the intelligent customer service model to receive user requests and return the corresponding target response.

[0071] In this embodiment, after receiving the container orchestration deployment file, the container orchestration platform can select a target node that meets the requirements according to the resource allocation strategy and graphics processing unit resource management system in the container orchestration deployment file, create an intelligent customer service instance on the target node and complete the model loading, so that the intelligent customer service instance can receive user requests, perform model inference calculations to generate the corresponding target response, and return the target response to the user.

[0072] This includes configuring health check probes during service operation to continuously monitor the running status and model loading of the intelligent customer service instance. If an instance anomaly is determined based on the running status and model loading of the intelligent customer service instance, the instance will be controlled to stop processing user requests and perform fault recovery operations.

[0073] For example, the written YAML configuration file is submitted to a GitOps repository, leveraging ArgoCD to achieve declarative continuous delivery. In the Kubernetes cluster, the K8s scheduler selects compute nodes with A100 / V100 GPUs based on node labels to deploy the relevant applications. Simultaneously, by employing Taints and Tolerations mechanisms, specific GPU resources are ensured to be dedicated to their intended use, avoiding unreasonable resource consumption and conflicts. To ensure stable service operation, Liveness Probe is enabled, sending health check requests to applications at set intervals. If an application anomaly is detected, appropriate measures are taken promptly for recovery or restart. Readiness Probe monitors the model's loading status; when the model is successfully loaded and ready to provide services, the corresponding Pod (container group) is included in the load balancing scope. Furthermore, Pod Disruption Budget is used to limit the number of interrupted Pods even during cluster node maintenance, upgrades, or other events that may cause Pod interruptions, thereby ensuring overall service high availability and ensuring that business operations are not significantly affected, allowing for continuous and stable service provision.

[0074] As shown above, this embodiment encapsulates all dependencies required by the intelligent customer service model into a standardized container image through containerization, ensuring a completely consistent model runtime environment throughout the entire process from development to testing to production, thus resolving deployment challenges caused by strong environment dependencies. Leveraging Kubernetes' resource scheduling capabilities, CPU / GPU resources are allocated rationally to avoid resource waste. The container orchestration deployment file clearly defines the resource allocation strategy for the model service, service access interfaces, and dynamic scaling rules that automatically adjust the number of service instances based on actual needs. During off-peak periods, the number of instances is automatically reduced to release resources, while rapid scaling is performed during peak periods, significantly improving the average resource utilization of the entire infrastructure and meeting the unique deployment requirements of large models in intelligent customer service scenarios.

[0075] See Figure 2 As shown in the embodiments of this application, a large-scale model deployment and service device for intelligent customer service based on container orchestration is also disclosed, including:

[0076] The image building module 11 is used to build a standardized container image that includes an inference engine, a model acceleration library, and model encryption and decryption components, and is adapted to the intelligent customer service scenario; the standardized container image is used to provide a unified and secure operating environment for service instances.

[0077] The management system construction module 12 is used to establish a graphics processing unit resource management system in the container orchestration cluster; the graphics processing unit resource management system is used to convert physical graphics processing units into logical resources that can be intelligently scheduled.

[0078] The data storage module 13 is used to save the model data of the intelligent customer service model to the target storage system and determine the mounting mechanism corresponding to the model data. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism is that when the service instance starts, the required target model data is downloaded from the target storage system by initializing the container and verified. The shared storage volume that supports multi-node read and write is mounted to the specified directory of the service instance through the persistent storage declaration mechanism of the container orchestration platform, so as to realize the reliable distribution and multi-instance sharing of the target model data.

[0079] The file generation module 14 is used to generate container orchestration and deployment files based on the graphics processing unit resource management system, the runtime requirements of the standardized container image, and the mounting mechanism corresponding to the model data; the container orchestration and deployment files include the resource allocation strategy of the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs;

[0080] Service module 15 is used to submit the container orchestration deployment file to the container orchestration platform so that the container orchestration platform can deploy the intelligent customer service model to receive user requests and return the corresponding target response.

[0081] In some specific embodiments, the management system construction module 12 includes:

[0082] The resource pool construction unit is used to identify and collect the graphics processing unit resource information corresponding to each node in the container orchestration cluster through the graphics processing unit device management plugin, and construct a manageable resource pool based on the graphics processing unit resource information; the graphics processing unit resource information includes the hardware model, video memory capacity and operating status of the graphics processing unit;

[0083] The system establishment unit is used to add corresponding type identifiers to each node based on the resource information of the graphics processing unit, so as to establish a resource classification system in the resource pool;

[0084] The strategy configuration unit is used to configure resource scheduling strategies according to the resource classification system; the resource scheduling strategy is used to realize the allocation mode of the graphics processing unit being shared among multiple service instances or exclusively used by a single service instance according to the computing requirements of the intelligent customer service reasoning task.

[0085] In some specific embodiments, the data storage module 13 includes:

[0086] The data storage unit is used to save the trained model data, including the weight file, parameter configuration file, and word segmenter vocabulary of the intelligent customer service model, to the target storage system.

[0087] A check code generation unit is used to generate a unique check code for the model data, so as to perform check code verification in subsequent operations on the model data based on the check code.

[0088] The data management unit is used to judge the model data according to a preset sensitivity judgment standard. For model data that meets the preset sensitivity, the encrypted storage component of the container orchestration platform is used to encrypt and store the model data that meets the preset sensitivity. For model data that meets the preset non-sensitivity, the shared configuration component of the container orchestration platform is used to manage the model data that meets the preset non-sensitivity.

[0089] In some specific embodiments, the large-scale deployment and service device for intelligent customer service based on container orchestration further includes:

[0090] The data acquisition unit is used to acquire the latest version of the target model data from the target storage system through the application programming interface using an initialization container; the initialization container supports breakpoint resume technology so as to transmit model data in the event of network instability;

[0091] The verification unit is used to perform hash value verification on the acquired target model data and obtain the verification result. If the verification result indicates that the verification failed, the initialization container is used to re-acquire the latest version of the target model data until the verification result indicates that the verification was successful.

[0092] In some specific embodiments, the service module 15 includes:

[0093] The service unit is used to select a target node that meets the requirements according to the resource allocation strategy and graphics processing unit resource management system in the container orchestration deployment file, create an intelligent customer service instance on the target node and complete model loading, so that the intelligent customer service instance can receive user requests, perform model inference calculations to generate corresponding target responses, and return the target responses to the user terminal.

[0094] In some specific embodiments, the large-scale deployment and service device for intelligent customer service based on container orchestration further includes:

[0095] The monitoring unit is used to configure health check probes during service operation and to continuously monitor the running status and model loading of the intelligent customer service instance using the health check probes.

[0096] The fault handling unit is used to control the intelligent customer service instance to stop processing user requests and perform fault recovery operations if it is determined that there is an instance abnormality based on the running status of the intelligent customer service instance and the model loading status.

[0097] In some specific embodiments, the large-scale deployment and service device for intelligent customer service based on container orchestration further includes:

[0098] The data acquisition unit is used to collect performance indicator data of the intelligent customer service in real time; the performance indicator data includes service request volume, response latency, and computing resource utilization.

[0099] The instance adjustment unit is used to automatically increase the number of service instances according to preset rules through the container orchestration platform when the service request volume or response latency reaches a preset high load threshold, the computing resource utilization meets a preset high occupancy condition, and the number of service instances is lower than the preset upper limit. When the service request volume drops to a preset low load threshold, the computing resource utilization meets a preset low occupancy condition, and the number of service instances is higher than the preset lower limit, the unit automatically reduces the number of service instances according to preset rules through the container orchestration platform.

[0100] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0101] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the large-scale deployment and service method for intelligent customer service based on container orchestration disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be a computer.

[0102] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0103] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0104] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs capable of performing the large-scale deployment and service method for intelligent customer service based on container orchestration disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0105] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for large-scale deployment and service of intelligent customer service based on container orchestration. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0106] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0107] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0109] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for deploying and serving large-scale intelligent customer service models based on container orchestration, characterized in that, include: Build a standardized container image that includes an inference engine, a model acceleration library, and model encryption / decryption components, and is adapted to intelligent customer service scenarios; The standardized container image is used to provide a unified and secure operating environment for service instances; A graphics processing unit resource management system is established in the container orchestration cluster; the graphics processing unit resource management system is used to convert physical graphics processing units into logical resources that can be intelligently scheduled; The model data of the intelligent customer service model is saved to the target storage system, and the mounting mechanism corresponding to the model data is determined. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism is that when the service instance starts, the required target model data is downloaded and verified from the target storage system by initializing the container. The shared storage volume that supports multi-node read and write is mounted to the specified directory of the service instance through the persistent storage declaration mechanism of the container orchestration platform, so as to realize the reliable distribution and multi-instance sharing of the target model data. Based on the graphics processing unit resource management system, the operational requirements of the standardized container image, and the mounting mechanism corresponding to the model data, a container orchestration and deployment file is generated; the container orchestration and deployment file includes the resource allocation strategy of the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs; The container orchestration deployment file is submitted to the container orchestration platform so that the container orchestration platform can deploy the intelligent customer service model, so as to receive user requests and return the corresponding target response using the deployed intelligent customer service model.

2. The method for deploying and serving a large-scale intelligent customer service model based on container orchestration as described in claim 1, characterized in that, The establishment of the graphics processing unit resource management system includes: The graphics processing unit (GPU) device management plugin identifies and collects the GPU resource information corresponding to each node in the container orchestration cluster, and constructs a manageable resource pool based on the GPU resource information; the GPU resource information includes the hardware model, video memory capacity, and operating status of the GPU. Based on the resource information of the graphics processing unit, add corresponding type identifiers to each node to establish a resource classification system within the resource pool; Configure resource scheduling strategies according to the resource classification system; the resource scheduling strategies are used to realize the allocation mode of graphics processing units being shared among multiple service instances or exclusively used by a single service instance according to the computing requirements of intelligent customer service reasoning tasks.

3. The method for deploying and serving large-scale intelligent customer service models based on container orchestration according to claim 1, characterized in that, Saving the model data of the intelligent customer service model to the target storage system includes: Save the trained model data, including the weight file, parameter configuration file, and word segmenter vocabulary of the intelligent customer service model, to the target storage system. A unique checksum is generated for the model data so that checksum verification can be performed in subsequent operations on the model data based on the checksum. The model data is judged according to a preset sensitivity judgment standard. For model data that meets the preset sensitivity, the encrypted storage component of the container orchestration platform is used to encrypt and store the model data that meets the preset non-sensitivity. For model data that meets the preset non-sensitivity, the shared configuration component of the container orchestration platform is used to manage the model data that meets the preset non-sensitivity.

4. The method for deploying and serving a large-scale intelligent customer service model based on container orchestration as described in claim 1, characterized in that, The step of downloading and verifying the required target model data from the target storage system by initializing the container includes: The initialization container is used to obtain the latest version of the target model data from the target storage system through the application programming interface; the initialization container supports breakpoint resume technology to transmit model data in the event of network instability; The acquired target model data is subjected to hash value verification to obtain the verification result. If the verification result indicates that the verification has failed, the initialization container is used to reacquire the latest version of the target model data until the verification result indicates that the verification has succeeded.

5. The method for deploying and serving large-scale intelligent customer service models based on container orchestration according to claim 1, characterized in that, The deployment of the intelligent customer service model, to receive user requests and return corresponding target responses using the deployed intelligent customer service model, includes: Based on the resource allocation strategy and graphics processing unit resource management system in the container orchestration deployment file, select a target node that meets the requirements, create an intelligent customer service instance on the target node and complete model loading, so that the intelligent customer service instance can receive user requests, perform model inference calculations to generate corresponding target responses, and return the target responses to the user.

6. The method for deploying and serving a large-scale intelligent customer service model based on container orchestration according to claim 1, characterized in that, Also includes: Configure a health check probe during service operation to continuously monitor the running status and model loading of the intelligent customer service instance. If an instance anomaly is determined based on the running status of the intelligent customer service instance and the model loading status, the intelligent customer service instance is controlled to stop processing user requests and perform fault recovery operations.

7. The method for deploying and serving large-scale intelligent customer service models based on container orchestration according to any one of claims 1 to 6, characterized in that, The process of adjusting instance size using dynamic scaling rules includes: Real-time collection of performance metrics data for intelligent customer service; the performance metrics data include service request volume, response latency, and computing resource utilization. When the service request volume or response latency reaches the preset high load threshold, the computing resource utilization meets the preset high occupancy condition, and the number of service instances is lower than the preset upper limit, the container orchestration platform automatically increases the number of service instances according to preset rules. When the service request volume drops to the preset low load threshold, the computing resource utilization meets the preset low occupancy condition, and the number of service instances is higher than the preset lower limit, the container orchestration platform automatically reduces the number of service instances according to preset rules.

8. A large-scale model deployment and service device for intelligent customer service based on container orchestration, characterized in that, include: The image building module is used to build standardized container images that include an inference engine, a model acceleration library, and model encryption / decryption components, and are adapted to intelligent customer service scenarios. The standardized container image is used to provide a unified and secure operating environment for service instances; The management system construction module is used to establish a graphics processing unit resource management system in a container orchestration cluster; the graphics processing unit resource management system is used to convert physical graphics processing units into logical resources that can be intelligently scheduled. The data storage module is used to save the model data of the intelligent customer service model to the target storage system and determine the mounting mechanism corresponding to the model data. The model data includes model weight files, parameter configuration files, and tool configurations. The mounting mechanism is that when the service instance starts, the required target model data is downloaded from the target storage system by initializing the container and verified. The shared storage volume that supports multi-node read and write is mounted to the specified directory of the service instance through the persistent storage declaration mechanism of the container orchestration platform, so as to realize the reliable distribution and multi-instance sharing of the target model data. The file generation module is used to generate container orchestration and deployment files based on the graphics processing unit resource management system, the runtime requirements of the standardized container image, and the mounting mechanism corresponding to the model data. The container orchestration and deployment files include the resource allocation strategy of the model service, the service access interface, and dynamic scaling rules that automatically adjust the number of service instances according to actual needs. The service module is used to submit the container orchestration deployment file to the container orchestration platform so that the container orchestration platform can deploy the intelligent customer service model, so as to receive user requests and return the corresponding target response using the deployed intelligent customer service model.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the large-scale deployment and service method for intelligent customer service based on container orchestration as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs, which, when executed by a processor, implement the large-scale model deployment and service method for intelligent customer service based on container orchestration as described in any one of claims 1 to 7.

Images