AI model full life cycle management method and system based on cloud native

Abstract

The invention relates to the cross technical field of artificial intelligence operation and maintenance and cloud native computing, in particular to an AI model full life cycle management method and system based on cloud native, and the method comprises the steps: constructing a containerized interactive development environment, and generating an invariable OCI mirror image containing a model code and a dependency library; a versioning model registration center of a three-layer storage architecture is adopted, and strong consistency management of model files and metadata is achieved; an automatic CI / CD assembly line is driven through a declarative workflow engine, and event triggering and timing scheduling are supported; the reasoning service deployment module based on the service grid realizes gray release and elastic capacity expansion and contraction; a multi-dimensional monitoring index system is established, and model performance attenuation and data drift are detected in real time; and designing an adaptive resource scheduling algorithm, and dynamically optimizing heterogeneous computing resource allocation. The resources can be dynamically adjusted according to the actual load, and high performance and high availability of the model service are ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence operation and maintenance and cloud native computing, in particular to an AI model full life cycle management method and system based on cloud native. BACKGROUND

[0002] With the in-depth industrial application of artificial intelligence technology, the full life cycle management of AI models has become a key link to support their large-scale landing. However, existing solutions still face a series of severe technical challenges in the cloud native environment, which restricts the overall efficiency of AI research and operation. Heterogeneous tool chains are usually used in different stages of model development, training and deployment. The running environments of these links are mutually isolated, and there is a lack of unified encapsulation and delivery standards, resulting in serious version conflicts and environment inconsistency risks between software dependencies, CUDA versions and system libraries.

[0003] Traditional MLOps tools mainly focus on experiment tracking and model registration, and lack systematic monitoring capabilities for production environments. Specifically, it is difficult to collect and analyze performance indicators of online inference services in real time, and it is also difficult to effectively detect key issues such as model accuracy decline and data distribution drift that affect service quality, resulting in model degradation that cannot be discovered and intervened in a timely manner. At the inference service level, GPU resources often use static allocation strategies and cannot be scaled based on real-time load. At the training task level, there is a lack of comprehensive consideration and dynamic scheduling strategies for heterogeneous resources, resulting in long task queuing time and low overall resource utilization.

[0004] Therefore, there is an urgent need in the art for a cloud native AI model full life cycle management solution that can break through environmental isolation, achieve full-link observability, support intelligent resource scheduling, and have cross-cluster collaborative management capabilities. SUMMARY

[0005] To solve the above problems, the present application provides an AI model full life cycle management method and system based on cloud native, specifically relates to a machine learning model development, deployment, monitoring, version control and resource scheduling system in a containerized environment, suitable for hybrid cloud infrastructure in the Kubernetes ecosystem.

[0006] In a first aspect, the present application provides an AI model full life cycle management method based on cloud native, which is cooperatively executed by a plurality of microservices deployed on a Kubernetes container orchestration platform, comprising: S1, building a model based on a multi-tenant interactive development platform, and automatically encapsulating the model code, dependent library and runtime environment into an immutable OCI image, and pushing it to a container image repository; S2, listen to code commit events from the version control warehouse, timing scheduling tasks, or retraining trigger alarm events generated by model performance monitoring; S3, in response to the event, drive the automatic CI / CD pipeline through the workflow engine; wherein, in response to the code commit event or the retraining trigger alarm event, the pipeline at least includes the task of pulling the corresponding image from the container image warehouse and executing model training, and outputting the trained model file; S4, store the model file and metadata in the versioned model registry center, and bind the model version number with the source code commit record in the version control warehouse; S5, based on the service mesh, deploy the model inference service obtained from the registry center, perform a gray release strategy, and perform elastic scaling of the inference service based on real-time collected service quality indicators; S6, a multi-dimensional monitoring index system is established for the deployed model inference service, and a composite health score of the model is calculated based on the monitoring indicators; when the composite health score is lower than a preset threshold, the model is automatically rolled back to the last stable version, and the retraining trigger alarm event is generated; Wherein, for the model training task in S3 and the model inference service in S5, the resource scheduling step is performed: based on a dynamic weight factor algorithm, the heterogeneous computing resources are scheduled, and the algorithm at least calculates the node score by comprehensively considering the CPU utilization, available GPU memory and available network bandwidth of the node.

[0007] Through the workflow engine, the automatic CI / CD pipeline is driven, and in response to the code commit event, the timing scheduling task or the retraining trigger alarm event generated by the model performance monitoring, the full life cycle automatic management of the model from development, training, deployment to monitoring is realized, the manual intervention is reduced, and the management efficiency and quality are improved. For the model training task and the model inference service, the resource scheduling based on the dynamic weight factor algorithm is performed, the node score is calculated by comprehensively considering the CPU utilization, available GPU memory and available network bandwidth of the node, and the heterogeneous computing resources can be reasonably allocated, the resource utilization is improved, and the cost of model training and inference is reduced.

[0008] As a further limitation of the technical scheme of the application, the step of automatically encapsulating the model code, dependent library and runtime environment into an immutable OCI image in S1 and pushing it to the container image warehouse comprises: S11, parse the model code, automatically identify the Python dependent package and version required by the model code, and generate a standardized dependent list file; S12, dynamically generate a Dockerfile based on a predefined base image template matched with the interactive development platform kernel; the Dockerfile contains instructions for copying the dependency manifest file, model code and runtime configuration into the image; S13, execute a docker build command or use a Kaniko tool to complete the construction of an OCI image in a container, and perform SHA-256 hash operation on the successfully constructed image to generate a unique digital fingerprint; S14, add a metadata tag containing the model project name, code version identifier, base image information and the digital fingerprint to the image, and push the image with the added metadata tag to a container image repository.

[0009] The construction operation of the OCI image simplifies the complexity of image construction, reduces the burden of the developer, and improves the efficiency and accuracy of image construction. The SHA-256 hash operation on the successfully constructed image generates a unique digital fingerprint, and the metadata tag containing the model project name, code version identifier, base image information and digital fingerprint is added to the image, which can ensure the uniqueness and traceability of the image, facilitate the quick positioning and identification of the image in the subsequent model development, deployment and operation process, and improve the manageability and security of the system.

[0010] As a further limitation of the technical scheme of the application, the step S2 comprises: S21, in the event bus or workflow engine, register corresponding listeners for different types of event sources, and configure trigger conditions and parameters; S22, through the listener, in parallel: Listen to the code commit event of the Webhook push of the version control repository; Listen to the timing scheduling event generated by the Kubernetes CronJob; Listen to the retraining trigger alarm event from the monitoring and management center based on the composite health score; S23, analyze and verify the original load of the listened event, and automatically fill in the context information, wherein for the code commit event, at least extract the code repository address, commit hash and branch information; for the retraining trigger alarm event, at least extract the service name of the alarm source, the health score at the time of triggering and the related monitoring indicators; S24, dispatch and trigger the corresponding automatic CI / CD pipeline in the workflow engine with the analyzed and filled event as input parameters; wherein, route the code commit event to the CI / CD pipeline for new model training and deployment; route the retraining trigger alarm event to the CI / CD pipeline for model retraining and version change.

[0011] The parsed and verified event raw load and the automatically filled context information can accurately extract the key information of the event, provide accurate input parameters for subsequent workflow triggering and processing, and improve the flexibility and accuracy of event processing. The precise triggering and routing of the workflow are realized, different types of events can trigger the corresponding workflow for processing, and the automation degree and work efficiency of the system are improved.

[0012] As a further limitation of the technical scheme of the application, in S22, the code submission event of the webhook push of the version control warehouse is monitored, which includes: The webhook is set to point to the receiving endpoint of the workflow engine; When the push or merge request event to a specific branch occurs in the version control warehouse, an HTTP POST request containing the submission details is sent to the receiving endpoint to trigger the monitoring.

[0013] The code submission event can be quickly responded to, ensuring that the model update can be timely processed in the automatic CI / CD pipeline, improving the efficiency of model development and update.

[0014] As a further limitation of the technical scheme of the application, in S3, the steps of driving the automatic CI / CD pipeline by the workflow engine in response to the monitored event include: S31, in response to the monitored event, the workflow engine instantiates a predefined declarative workflow template according to the event type, and binds the parameters parsed in the event load as workflow variables; S32, the workflow first performs an environment preparation task, pulls the immutable OCI image associated with the current code submission hash generated by S1 from the container image warehouse, and creates a task execution environment based on the image; S33, according to the definition of directed acyclic graph, the following at least two stages of tasks are scheduled in sequence or in parallel: Data verification stage: running data quality checking tools in the environment to verify the mode and distribution of input data set; Model training stage: starting a distributed training framework in the environment, calling the model code submitted in S1 for training, and outputting the model file; Model verification stage: loading the trained model file in the environment, calculating the performance indicators on the reserved test set, and comparing with the preset threshold; S34, if the model verification is passed, the trained model file and evaluation indicators are prepared as output to be registered in the versioned model registration center, and the pipeline success state is reported to the monitoring system.

[0015] The corresponding workflow template can be instantiated flexibly according to different event types, improving the automation and flexibility of the workflow. The environment preparation task pulls an immutable OCI image associated with the current code commit hash from a container image repository, and creates a task execution environment based on the image, ensuring the consistency and stability of the task execution environment.

[0016] According to the definition of the directed acyclic graph, the tasks of the data verification phase, the model training phase and the model verification phase are scheduled in sequence or in parallel, realizing the standardized process of model training and verification. The data quality checking tool verifies the mode and distribution of the input data set, which can ensure the quality of the input data; the distributed training framework calls the submitted model code for training, and compares the performance indicators calculated on the reserved test set with the preset threshold, which can ensure the training quality and performance of the model, and improve the reliability and stability of the model.

[0017] As a further limitation of the technical solution of the application, the step S4 includes: S41, receiving the pipeline output from S3, including the trained model file, the performance evaluation indicators calculated in the verification phase, and the data set fingerprint used for training; S42, automatically generating a semantic version number of the model based on the version control strategy, and automatically incrementing the version number according to the code change type; S43, uploading the model file to the object storage layer, and storing the version number, performance evaluation indicators, data set fingerprint, model file storage path and associated code commit record as a piece of metadata in the relational database of the metadata layer, and establishing a bidirectional index between the two; S44, synchronously recording the identifier of the base OCI image used by the model training task in S3 to the dependency cache layer.

[0018] The model version can be automatically managed and semantically expressed, which is convenient for users to understand and manage the model version. The unified storage and strongly consistent management of the model file and the metadata are realized, improving the efficiency and accuracy of model registration, and facilitating the query and use of the model. The identifier of the base OCI image used by the model training task is synchronously recorded to the dependency cache layer, which facilitates the management and tracing of the dependency relationship of the model, helps to quickly locate and solve the dependency problems that may occur in the model running process, and improves the maintainability and stability of the system.

[0019] As a further limitation of the technical solution of the application, in S5, the steps of deploying the model inference service based on the service mesh, performing gray release and elastic scaling include: S51, generating a Kubernetes custom resource according to the target model version in the model registry, the resource declaring the type of the model server, the storage path of the model file, the required computing resources, and the automatic scaling strategy; S52, deploying the inference service resource to the cluster, the sidecar agent of the service mesh automatically injecting into the service Pod, and meanwhile, configuring the corresponding traffic routing rules in the service mesh to define the gray release strategy; S53, according to the traffic routing rules, routing the user request traffic to the newly deployed model version in stages according to a preset proportion, and deciding whether to proceed to the next stage based on real-time monitoring indicators at the end of each stage; S54, continuously collecting the inference service indicators reported through the sidecar agent, and automatically performing the scaling operation when the indicators trigger a preset threshold.

[0020] Through the gray release strategy, the new version model is gradually pushed to the production environment, reducing the impact of the new version model on the production environment, and improving the deployment efficiency and reliability of the model inference service.

[0021] At the end of each stage, based on real-time monitoring indicators, it is decided whether to proceed to the next stage, which can dynamically adjust the traffic distribution according to the actual traffic and model performance, ensuring the performance and stability of the model inference service. Continuous collection of inference service indicators reported through the sidecar agent, and automatic execution of the scaling operation when the indicators trigger a preset threshold, realizes the dynamic scaling of the inference service, which can dynamically adjust the resources according to the actual load, improves the resource utilization and the availability of the inference service.

[0022] As a further limitation of the technical solution of the present application, the step S6 includes: S61, through the data collection agent deployed on the infrastructure, the service mesh sidecar agent and the custom exporter, real-time collection of monitoring indicators in the following dimensions: Infrastructure indicators: GPU utilization, GPU memory occupation; Model service indicators: request throughput, P50 / P99 inference delay, HTTP error rate; Data quality indicators: group stability index, feature distribution and benchmark distribution KL divergence; S62, based on the normalized monitoring indicators, calculating the composite health score at a preset period, the calculation formula being: Wherein is a preset weight; S63, comparing the calculated composite health score with a preset threshold; when the composite health score is lower than the threshold, automatically generating a governance instruction, the governance instruction triggering two simultaneous actions at the same time: Action one: switch all traffic of the current fault model to the last stable version recorded in the versioned model registry through the service mesh API; Action two: generate a structured retraining trigger alarm event containing the fault service identifier, the composite health score at the time of triggering, and the relevant key indicator snapshot, which is immediately captured by the event listening step of S2; S64, record all context information of this governance event, including the trigger indicators, the composite health score, the executed rollback operation, and the triggered retraining event, as an audit log to the database.

[0023] The automatic governance of the model is realized, including switching the traffic of the current fault model to the last stable version and generating a retraining trigger alarm event, which can timely discover and handle model performance problems, improving the reliability and stability of the model. Recording all context information of the governance event as an audit log to the database facilitates the auditing and tracing of the governance process of the model, improving the manageability and security of the system.

[0024] As a further limitation of the technical solution of the present application, the specific process of the resource scheduling step includes: SS3-5-1, receive scheduling requests from model training tasks or inference service deployment, and parse resource demand profiles, which at least include required GPU types, memory sizes, estimated calculation durations, and task priority labels; SS3-5-2, through the cluster monitoring system, real-time acquisition of dynamic resource states of each computing node, including CPU core utilization, used and available memory of each GPU card, and current available bandwidth of inter-node network; SS3-5-3, for each node, based on the real-time resource state, calculate the node score using a dynamic weight factor algorithm, the calculation formula is: wherein, is a configurable weight coefficient, and ; SS3-5-4, compare the node scores of all nodes, select the node with the highest node score, and bind the task to be scheduled to the selected node for execution.

[0025] It can comprehensively consider the resource demand of the task and the resource state of the node, reasonably allocate heterogeneous computing resources, improve the rationality and efficiency of resource scheduling, ensure the efficient execution of model training and inference tasks, and reduce resource waste and task waiting time.

[0026] In a second aspect, the present application provides an AI model full life cycle management system based on cloud native, which is deployed on a Kubernetes container orchestration platform. The system includes a plurality of cooperatively executed microservice modules, specifically including: a model development and environment solidification module for building a model based on a multi-tenant interactive development platform and automatically encapsulating model code, dependent libraries and runtime environment into an immutable OCI image and pushing the image to a container image repository; an event listening and triggering module for listening to code commit events from a version control repository, timing scheduling tasks or retraining triggering alarm events generated by model performance monitoring; an automated workflow execution module for driving an automated CI / CD pipeline through a declarative workflow engine in response to the listened events; wherein the pipeline in response to the code commit events or the retraining triggering alarm events at least includes a task of pulling corresponding images from the container image repository and executing model training, and outputs a trained model file; a versioned registration module for storing model files and metadata in a versioned model registration center and binding model version numbers with source code commit records in the version control repository; an inference service deployment and operation module for deploying model inference services obtained from the registration center based on a service mesh, performing a gray release strategy and performing elastic scaling of the inference services based on real-time collected service quality indicators; a monitoring and autonomous governance module for establishing a multi-dimensional monitoring indicator system for the deployed model inference services, calculating a composite health score of the model based on the monitoring indicators, and automatically triggering model rollback to the last stable version and generating the retraining triggering alarm event when the composite health score is lower than a preset threshold; a resource scheduling module for scheduling heterogeneous computing resources for model training tasks and model inference services based on a dynamic weight factor algorithm, which at least calculates node scores by integrating CPU utilization, available GPU memory and available network bandwidth of the nodes.

[0027] As can be seen from the above technical solutions, the present application has the following advantages: through the multi-tenant interactive development platform, developers can quickly build models, and model code, dependent libraries and runtime environment can be automatically encapsulated into an immutable OCI image and pushed to a container image repository, reducing the time for environment configuration and dependency management, so that developers can focus more on model development itself.

[0028] The model file and metadata are stored in a versioned model registry center and bound with source code submission records in a version control warehouse, so that strong consistency management of model versions and code versions is achieved, and the change history of the model is easy to trace. The model inference service is deployed based on a service mesh, a gray release strategy is executed, and the new version model can be gradually pushed to the production environment, reducing the impact of the new version model on the production environment and improving the flexibility and reliability of model deployment. At the same time, the inference service is elastically scaled based on real-time collected service quality indicators, so that resources can be dynamically adjusted according to actual load to ensure high performance and high availability of the model service. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0030] Figure 1 The flowchart of the method provided by the embodiment of the present application is shown.

[0031] Figure 2 The block diagram of the system provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0032] In order to make the application purposes, features and advantages of the present application more obvious and easy to understand, the technical solutions protected by the present application will be described clearly and completely by using specific embodiments and drawings. Obviously, the following described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0033] Unless otherwise defined, all technical and scientific terms used in the present application have the same meaning as understood by those skilled in the art to which the present application belongs. The terms used in the specification of the present application are only for the purpose of describing the specific embodiments and are not intended to limit the present application.

[0034] As shown in Figure 1 The embodiment of the present application provides an AI model full life cycle management method based on cloud native, which is cooperatively executed by a plurality of microservices deployed on a Kubernetes container orchestration platform, including: S1, building a model based on a multi-tenant interactive development platform, and automatically encapsulating the model code, dependent library and runtime environment into an immutable OCI image and pushing it to a container image warehouse; S2, listen to code commit events from the version control repository, scheduled tasks, or retraining trigger alarm events generated by model performance monitoring; S3, in response to the events, drive the automated CI / CD pipeline through the workflow engine; wherein the pipeline in response to the code commit event or the retraining trigger alarm event includes at least the task of pulling the corresponding image from the container image repository and executing model training, and outputting the trained model file; S4, store the trained model file and metadata in the versioned model registry, and bind the model version number with the source code commit record in the version control repository; The versioned model registry provides a synchronization mechanism based on GitOps, synchronizes the metadata changes of the model version to a dedicated Git repository in the form of a declarative file, and realizes the audit and traceability of model version management.

[0035] S5, deploy the model inference service obtained from the registry based on the service mesh, execute the gray release strategy, and perform elastic scaling of the inference service based on real-time collected service quality indicators; S6, establish a multi-dimensional monitoring indicator system for the deployed model inference service, calculate the composite health score of the model based on the monitoring indicators; when the composite health score is lower than the preset threshold, automatically trigger the model to roll back to the last stable version, and generate the retraining trigger alarm event; Wherein, for the model training task in S3 and the model inference service in S5, the resource scheduling step is performed: based on a dynamic weight factor algorithm to schedule heterogeneous computing resources, the algorithm at least integrates the CPU utilization, available GPU memory and available network bandwidth of the node to calculate the node score.

[0036] In some embodiments, the step of automatically encapsulating the model code, dependent library and runtime environment into an immutable OCI image in S1 and pushing it to the container image repository includes: S11, parse the model code, automatically identify the Python dependent packages and versions required by the model code, and generate a standardized dependency list file; in this step, parsing the model code includes static analysis of Jupyter Notebook cells to distinguish cells containing import statements from cells containing business logic, and to generate the dependency list file accordingly.

[0037] S12, dynamically generate a Dockerfile based on a pre-defined base image template matched with the kernel of the interactive development platform; the Dockerfile contains instructions for copying the dependency list file, model code and runtime configuration into the image; S13, execute the docker build command or use the Kaniko tool to complete the construction of the OCI image in the container, and perform SHA-256 hash operation on the successfully constructed image to generate a unique digital fingerprint; S14, add metadata tags containing model project name, code version identification, base image information and the digital fingerprint to the image, and push the image with added metadata tags to the container image repository. In this step, the metadata tags added to the image also include model card information, which at least includes model type, intended use, basic description of training data set and fairness constraint description.

[0038] In some embodiments, the step of S2 includes: S21, in the event bus or workflow engine, register the corresponding listener for different types of event sources, and configure the trigger condition and parameter; the workflow engine contains a policy configuration template, which defines: Resource constraints: GPU type (A100 / V100), memory threshold (≥24GB); Verification rules: accuracy tolerance (ΔACC≤1%), fairness index (Demographic Parity difference <0.05); Deployment parameters: canary release phase length (T1=5min, T2=30min), maximum Pod replica number (MaxReplicas=50).

[0039] S22, through the listener, parallelly: Listen to the code commit event of the Webhook push of the version control repository; Listen to the timing scheduling event generated by the Kubernetes CronJob; Listen to the retraining trigger alarm event from the monitoring governance center based on the composite health score score; it should be noted that listening to the code commit event of the Webhook push of the version control repository includes: Set the receiving endpoint of the Webhook to the workflow engine; When the push or merge request event to a specific branch occurs in the version control repository, send an HTTP POST request containing the commit details to the receiving endpoint to trigger the listener.

[0040] The retraining trigger alarm event from the monitoring governance center specifically includes: Listen to the alarm information issued by the Prometheus Alertmanager according to the configured health_score<0.7 rule, and encapsulate the alarm information as a standard format retraining trigger event.

[0041] S23, parse and verify the event raw load monitored, and automatically fill in the context information, wherein for the code submission event, at least extract the code repository address, submission hash, branch information; for the retraining trigger alarm event, at least extract the service name of the alarm source, the health score at the time of triggering and the related monitoring indicators; S24, dispatch and trigger the corresponding automatic CI / CD pipeline in the workflow engine as input parameters with the parsed and filled events; wherein the code submission event is routed to the CI / CD pipeline for new model training and deployment; the retraining trigger alarm event is routed to the CI / CD pipeline for model retraining and version change.

[0042] In some embodiments, in S3, in response to the monitored event, the step of driving the automatic CI / CD pipeline by the workflow engine includes: S31, in response to the monitored event, the workflow engine selects a predefined declarative workflow template according to the event type to instantiate, and binds the parameters parsed in the event load as workflow variables; S32, the workflow first performs an environment preparation task, pulls the immutable OCI image associated with the current code submission hash generated by S1 from the container image repository, and creates a task execution environment based on the image; S33, according to the definition of directed acyclic graph, the following at least two stages of tasks are scheduled in sequence or in parallel: Data verification stage: run data quality checking tools in the environment to verify the mode and distribution of the input data set; Model training stage: start a distributed training framework in the environment, call the model code submitted in S1 for training, and output the model file; the start of the distributed training framework includes: Submit a TrainingJob custom resource to the Kubernetes cluster by the workflow engine; The TrainingJob is taken over by the corresponding Kubernetes Operator, which is responsible for creating and managing multiple Pod replicas to execute distributed training, and the intelligent resource scheduler allocates appropriate GPU nodes for these Pods.

[0043] The model verification stage: loading the trained model file in the environment, calculating the performance indicators on the reserved test set, and comparing with the preset threshold; the comparison with the preset threshold includes fairness audit; the fairness audit calculates the difference of the prediction results of the model on different demographic subgroups, and compares with the predefined fairness indicator threshold, if the difference exceeds the threshold, the model verification fails.

[0044] For the pipeline started by the retraining triggered alarm event, the input data of the data verification stage is the latest data snapshot of the production environment when the alarm is triggered.

[0045] In some embodiments, the step of S4 includes: S41, receiving the pipeline output from S3, including the trained model file, the performance evaluation indicators calculated in the verification stage, and the data set fingerprint used for training; The data set fingerprint is obtained by calculating the SHA-256 hash value of the training data set, and is stored together with the model metadata for subsequent data consistency check and drift analysis.

[0046] S42, based on the version control strategy, automatically generating the semantic version number of the model, and automatically incrementing the version number according to the code change type; The version number format is {major}.{minor}.{patch}-{git_commit_hash}, where {major}.{minor}.{patch} is automatically incremented according to the code change type, and {git_commit_hash} is the code commit hash that triggers this pipeline; The automatic incrementing of the version number based on the code change type includes: If the code change is a bug fix or optimization, increment the {patch} version; If the code change is a new feature and is backward compatible, increment the {minor} version and reset the {patch} version to 0; If the code change is a major update and is not backward compatible, increment the {major} version and reset the {minor} and {patch} versions to 0.

[0047] S43, uploading the model file to the object storage layer, while storing the version number, performance evaluation indicators, data set fingerprint, model file storage path and associated code commit record as a piece of metadata in the relational database of the metadata layer, and establishing a bidirectional index between the two; S44, synchronously recording the identifier of the base OCI image used by the model training task in S3 to the dependency cache layer, so as to completely encapsulate the environment, code and data pedigree of the model training.

[0048] In some embodiments, the inference service mesh module adopts a sidecar container architecture, injecting a monitoring agent (OpenTelemetry Collector) in each model service Pod to collect inference metrics in real time and push them to a Prometheus time series database. In S5, the steps of deploying model inference services based on the service mesh, performing gray release, and performing elastic scaling include: S51. According to the target model version in the model registry center, a Kubernetes custom resource is generated, which declares the type of model server, the storage path of model file, the required computing resources, and the automatic scaling strategy; S52, deploy the inference service resource to the cluster, and the sidecar agent of the service mesh is automatically injected into the service Pod; at the same time, configure the corresponding traffic routing rules in the service mesh to define the gray release strategy; S53, according to the traffic routing rules, route user request traffic to the newly deployed model version in stages according to the preset proportion, and the stages include at least an initial small proportion (such as 5%), a medium proportion (such as 50%), and a full amount (100%), and decide whether to proceed to the next stage based on real-time monitoring indicators after the end of each stage; The gray release strategy includes canary release and shadow deployment. Among them, the shadow deployment is configured to send a copy of the production traffic to the new version of the model service, but the response result of the new version is not returned to the user, and is only used for comparison and verification with the output result of the stable version.

[0049] S54, continuously collect inference service metrics reported through the sidecar agent, and when the metrics trigger a preset threshold, automatically perform scaling operations: horizontal scaling is based on the relationship between queries per second and the number of replicas; vertical scaling dynamically adjusts the resource limit of a single Pod based on the CPU and memory usage of the Pod.

[0050] The vertical scaling operation is performed by the Vertical Pod Autoscaler, which analyzes the historical resource usage pattern of the Pod through machine learning algorithms to recommend and apply more accurate CPU and memory request values and limit values.

[0051] S55, when deploying, if there is a quantized version of the model in the model registry center, the quantized version is preferentially deployed, the quantized version is generated by converting the model weight from FP16 precision to INT8 precision, and is used to reduce inference delay and resource consumption.

[0052] In some embodiments, the steps of S6 include: S61, Collect the following monitoring indicators in real time through the data collection agent deployed on the infrastructure, service mesh sidecar agent, and custom exporter: Infrastructure indicators: GPU utilization, GPU memory occupancy; Model service indicators: request throughput, P50 / P99 inference delay, HTTP error rate; Data quality indicators: group stability index, feature distribution KL divergence from baseline distribution; The data quality indicators include: Use the Apache Flink stream processing engine to perform window calculation on the real-time data stream of the input inference service, and calculate the group stability index and feature distribution difference between the current data window and the training baseline data in real time.

[0053] S62, Calculate the composite health score based on the normalized monitoring indicators collected at the preset period, and the calculation formula is: Wherein is the preset weight; S63, Compare the calculated composite health score with the preset threshold value; when the composite health score is lower than the threshold value, automatically generate a governance instruction, which triggers two simultaneous actions: Action one: switch all traffic of the current faulty model to the last stable version recorded in the versioned model registry through the service mesh API; Action two: generate a structured retraining trigger alarm event, which contains the faulty service identifier, the composite health score at the time of triggering, and the related key indicator snapshot, and is immediately captured by the event listening step of S2; In action two, the generated retraining trigger alarm event is prioritized, and the resource scheduling step allocates high priority to the retraining task triggered by this event and allows it to preempt the computing resources of low-priority tasks.

[0054] If the decline amplitude of the composite health score in a single calculation period exceeds the preset sharp decline threshold, skip part of the evaluation period and immediately trigger the governance instruction.

[0055] S64, Record all context information of this governance event, including the triggering indicators, the composite health score, the executed rollback operation, and the triggered retraining event, as an audit log to the database.

[0056] In some embodiments, the specific process of the resource scheduling step includes: SS3-5-1, receive a scheduling request from a model training task or an inference service deployment, and parse a resource requirement profile, the profile including at least required GPU type, memory size, estimated calculation duration, and task priority label; SS3-5-2, through a cluster monitoring system, real-time acquisition of dynamic resource states of each computing node, including CPU core utilization, used and available memory of each GPU card, and current available bandwidth of inter-node network; SS3-5-3, for each node, based on the real-time resource state, using a dynamic weight factor algorithm to calculate a node score, the calculation formula being: wherein is a configurable weight coefficient, and ; The dynamic weight factor algorithm supports task type-aware weight adaptation: For a computation-intensive model training task, a higher beta value is automatically set, emphasizing the availability of GPU memory; For a delay-sensitive model inference task, a higher gamma value is automatically set, emphasizing network bandwidth to ensure low delay.

[0057] The node score also includes a priority weighting factor: For a task marked as high priority, multiply the node scores of all nodes of the task by a weighting factor greater than 1 to ensure that it is preferentially scheduled to the optimal node.

[0058] SS3-5-4, comparing the node scores of all nodes, selecting the node with the highest node score, and binding the task to be scheduled to the selected node for execution.

[0059] SS3-5-5, after calculating the score in SS3-5-3, for a task requiring multi-card parallel training, further evaluating the GPU interconnection topology of the node; preferentially selecting a node with NVLink high-speed interconnection channels between GPUs, and scheduling the task to the GPU group with the most NVLink connections on the node.

[0060] SS3-5-6, when the local cluster has insufficient resources to meet a high-priority task, distributing the scheduling request and node score calculation logic to other Kubernetes clusters within the federation, including edge clusters, through the Cluster API, and selecting a globally optimal node from among them for task scheduling.

[0061] As Figure 2 shown, the embodiment of the application also provides an AI model full life cycle management system based on cloud native, which is deployed on a Kubernetes container orchestration platform, and the system includes a plurality of cooperatively executed microservice modules, specifically including: a model development and environment solidification module, configured to build a model based on a multi-tenant interactive development platform, and automatically encapsulate model code, dependent libraries and a runtime environment into an immutable OCI image, and push the OCI image to a container image repository; an event listening and triggering module, configured to listen to a code commit event from a version control repository, a timing scheduling task, or a retraining triggering alarm event generated by model performance monitoring; an automated workflow execution module, configured to drive an automated CI / CD pipeline through a declarative workflow engine in response to the listened event; wherein the pipeline in response to the code commit event or the retraining triggering alarm event at least includes a task of pulling a corresponding image from the container image repository and executing model training, and outputting a trained model file; a versioned registration module, configured to store the model file and metadata in a versioned model registration center, and bind a model version number with a source code commit record in the version control repository; an inference service deployment and operation module, configured to deploy a model inference service obtained from the registration center based on a service mesh, perform a gray release strategy, and perform elastic scaling of the inference service based on real-time collected service quality indicators; a monitoring and autonomous governance module, configured to establish a multi-dimensional monitoring indicator system for the deployed model inference service, calculate a composite health score of the model based on the monitoring indicators, and when the composite health score is lower than a preset threshold, automatically trigger a model rollback to a last stable version, and generate the retraining triggering alarm event; a resource scheduling module, configured to schedule heterogeneous computing resources for model training tasks and model inference services based on a dynamic weight factor algorithm, which at least calculates a node score by integrating CPU utilization, available GPU memory and available network bandwidth of the node.

[0062] In some embodiments, the model development and environment solidification module is specifically configured to: parse the model code, automatically identify Python dependent packages and versions required by the model code, and generate a standardized dependent list file; dynamically generate a Dockerfile based on a predefined base image template matched with a kernel of the interactive development platform; the Dockerfile includes instructions for copying the dependent list file, model code and runtime configuration into the image; execute a docker build command or use a Kaniko tool to complete the construction of the OCI image in the container, and perform SHA-256 hash operation on the successfully constructed image to generate a unique digital fingerprint; Add a metadata tag containing the model item name, code version identifier, base image information, and the digital fingerprint to the image, and push the image with the added metadata tag to a container image repository.

[0063] In some embodiments, the event listening and triggering module is specifically configured to: In the event bus or workflow engine, register corresponding listeners for different types of event sources, and configure triggering conditions and parameters; Through the listeners in parallel: Listen to code commit events pushed by Webhook of the version control repository; Listen to timing scheduling events generated by Kubernetes CronJob; Listen to retraining trigger alarm events from the monitoring governance center based on the composite health score; Parse and verify the original payload of the listened events, and automatically fill in the context information; Dispatch and trigger the corresponding automated CI / CD pipelines in the declarative workflow engine as input parameters after parsing and filling the events; wherein, route the code commit event to the CI / CD pipeline for new model training and deployment; route the retraining trigger alarm event to the CI / CD pipeline dedicated for model retraining and version change.

[0064] In some embodiments, the automated workflow execution module is specifically configured to: In response to the listened events, the workflow engine selects a predefined declarative workflow template according to the event type, instantiates it, and binds the parameters parsed from the event payload as workflow variables; Control the workflow to first perform an environment preparation task, pull the immutable OCI image associated with the current code commit hash generated by the model development and environment solidification module from the container image repository, and create a task execution environment based on the image; According to the definition of a directed acyclic graph, sequentially or in parallel, schedule the tasks of at least two stages as follows: Data verification stage: run data quality check tools in the environment to verify the mode and distribution of the input data set; Model training stage: start a distributed training framework in the environment to train and output model files; Model validation stage: load the trained model file in the environment, calculate the performance indicators on the reserved test set, and compare them with the preset threshold; If the model validation is passed, the trained model file and evaluation indicators are prepared as output and registered to the versioned model registration center.

[0065] In some embodiments, the versioned registration module is specifically used for: receiving pipeline output from the automated workflow execution module, including a trained model file, performance evaluation indicators, and a data set fingerprint used for training; automatically generating a semantic version number of the model based on a version control strategy, and automatically incrementing the version number according to the code change type; uploading the model file to an object storage layer, while storing the version number, performance evaluation indicators, data set fingerprint, model file storage path, and associated code submission record as a piece of metadata in a relational database of a metadata layer, and establishing a bidirectional index between the two; synchronously recording the identifier of the base OCI image used by the model training task to the dependency cache layer.

[0066] In some embodiments, the inference service deployment and operation module is specifically used for: generating a Kubernetes custom resource according to the target model version in the model registration center, the resource declaring the type of the model server, the storage path of the model file, the required computing resources, and the automatic scaling strategy; deploying the inference service resource to the cluster, and automatically injecting the service mesh sidecar agent into the service Pod; at the same time, configuring the corresponding traffic routing rules in the service mesh to define the gray release strategy; according to the traffic routing rules, routing user request traffic to the newly deployed model version in stages according to a preset proportion; continuously collecting inference service indicators reported through the sidecar agent, and automatically performing scaling operations when the indicators trigger a preset threshold.

[0067] In some embodiments, the monitoring and autonomous governance module is specifically used for: collecting monitoring indicators in real time in the infrastructure, model service, and data quality dimensions through data collection agents deployed on the infrastructure, service mesh sidecar agents, and custom exporters; based on the normalized monitoring indicators collected at a preset period, calculating a composite health score; comparing the calculated composite health score with a preset threshold; when the composite health score is lower than the threshold, automatically generating a governance instruction, which triggers switching all traffic of the current faulty model to the last stable version through the service mesh API, and generates a structured retraining trigger alarm event.

[0068] In some embodiments, the resource scheduling module is specifically used for: receiving a scheduling request and analyzing a resource demand profile; Through the cluster monitoring system, the dynamic resource state of each computing node is acquired in real time; For each node, based on the real-time resource state, a node score is calculated using a dynamic weight factor algorithm; The node scores of all nodes are compared, the node with the highest node score is selected, and the task to be scheduled is bound to the node for execution.

[0069] The above description of disclosed embodiments enables one of ordinary skill in the art to make or use the application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1.A cloud-native AI model full life cycle management method, characterized in that, The method is cooperatively executed by a plurality of microservices deployed on a Kubernetes container orchestration platform, comprising: S1, constructing a model based on a multi-tenant interactive development platform, and automatically encapsulating model code, dependent libraries and runtime environments into an immutable OCI image, and pushing the image to a container image repository; S2, listening to code commit events from a version control repository, scheduled tasks, or retraining trigger alarm events generated by model performance monitoring; S3, in response to the events, driving an automated CI / CD pipeline through a workflow engine; wherein the pipeline in response to the code commit event or the retraining trigger alarm event at least includes a task of pulling the corresponding image from the container image repository and executing model training, and outputting a trained model file; S4, storing the model file and metadata in a versioned model registry, and binding the model version number with the source code commit record in the version control repository; S5, deploying a model inference service obtained from the registry based on a service mesh, performing a gray release strategy, and performing elastic scaling of the inference service based on real-time collected service quality indicators; S6, establishing a multi-dimensional monitoring indicator system for the deployed model inference service, calculating a composite health score of the model based on the monitoring indicators; when the composite health score is lower than a preset threshold, automatically triggering model rollback to the last stable version, and generating the retraining trigger alarm event; Wherein, for the model training task in S3 and the model inference service in S5, a resource scheduling step is performed: scheduling heterogeneous computing resources based on a dynamic weight factor algorithm, which at least integrates CPU utilization, available GPU memory and available network bandwidth of the node to calculate a node score. 2.The cloud-native based AI model full life cycle management method of claim 1, wherein, The step of automatically encapsulating model code, dependent libraries and runtime environments into an immutable OCI image in S1, and pushing the image to a container image repository, comprises: S11, parsing the model code, automatically identifying the Python dependent packages and versions required by the model code, and generating a standardized dependent list file; S12, dynamically generating a Dockerfile based on a predefined base image template matched with the kernel of the interactive development platform; the Dockerfile includes instructions for copying the dependent list file, model code and runtime configuration into the image; S13, executing a docker build command or using a Kaniko tool to complete the construction of the OCI image in the container, and performing SHA-256 hash operation on the successfully constructed image to generate a unique digital fingerprint; S14, adding metadata tags including model project name, code version identifier, base image information and digital fingerprint to the image, and pushing the image with metadata tags to the container image repository. 3.The cloud-native based AI model full life cycle management method of claim 2, wherein, The step of S2 comprises: S21, registering corresponding listeners for different types of event sources in the event bus or workflow engine, and configuring trigger conditions and parameters; S22, through the listener in parallel: listen to the code commit event pushed by the webhook of the version control repository; listening to the scheduled events generated by the Kubernetes CronJob; listening to the retraining trigger alarm event from the monitoring governance center based on the composite health score; S23, analyzing and verifying the event raw payload, and automatically filling in the context information, wherein at least the code repository address, commit hash, and branch information are extracted for the code commit event; at least the service name of the alarm source, the health score at the time of triggering, and the related monitoring indicators are extracted for the retraining trigger alarm event; S24, dispatching and triggering the corresponding automated CI / CD pipeline in the workflow engine as input parameters based on the parsed and filled event; wherein the code commit event is routed to the CI / CD pipeline for new model training and deployment; the retraining trigger alarm event is routed to the CI / CD pipeline dedicated to model retraining and version change. 4.The cloud-native based AI model full life cycle management method of claim 3, wherein, In S22, the code commit event pushed by the webhook of the version control repository includes: setting the webhook to point to the receiving endpoint of the workflow engine; when the push or merge request event to a specific branch occurs in the version control repository, an HTTP POST request containing the commit details is sent to the receiving endpoint to trigger the listening. 5.The cloud-native based AI model full life cycle management method of claim 4, wherein, In S3, in response to the listened event, the steps of driving the automated CI / CD pipeline by the workflow engine include: S31, in response to the listened event, the workflow engine instantiates the predefined declarative workflow template according to the event type, and binds the parameters parsed from the event payload as workflow variables; S32, the workflow first performs an environment preparation task, pulls the immutable OCI image associated with the current code commit hash from the container image repository, and creates a task execution environment based on the image; S33, according to the definition of directed acyclic graph, the tasks of at least two stages are scheduled in sequence or in parallel: data verification stage: running data quality checking tools in the environment to verify the mode and distribution of the input data set; model training stage: starting a distributed training framework in the environment, calling the model code submitted in S1 for training, and outputting the model file; model validation stage: loading the trained model file in the environment, calculating the performance indicators on the reserved test set, and comparing with the preset threshold; S34, if the model validation is passed, the trained model file and evaluation indicators are prepared as output and registered to the versioned model registry center, and the pipeline success status is reported to the monitoring system. 6.The cloud-native based AI model full life cycle management method of claim 5, wherein, The steps of S4 include: S41, receiving the pipeline output from S3, including the trained model file, the performance evaluation indicators calculated in the validation stage, and the data set fingerprint used for training; S42, based on the version control strategy, automatically generating the semantic version number of the model, and automatically incrementing the version number according to the code change type; S43, upload the model file to the object storage layer, and at the same time, store the version number, performance evaluation index, data set fingerprint, model file storage path and associated code submission record as a piece of metadata in the relational database of the metadata layer, and establish a bidirectional index between the two; S44, synchronize the identifier of the base OCI image used by the model training task in S3 to the dependency cache layer. 7.The cloud-native based AI model full life cycle management method of claim 6, wherein, In S5, the steps of deploying the model inference service based on the service mesh, performing gray release and elastic scaling include: S51, generate a Kubernetes custom resource according to the target model version in the model registry center, which declares the type of model server, the storage path of model file, the required computing resources and the automatic scaling strategy; S52, deploy the inference service resource to the cluster, and the sidecar agent of the service mesh is automatically injected into the service Pod; at the same time, configure the corresponding traffic routing rules in the service mesh to define the gray release strategy; S53, according to the traffic routing rules, route user request traffic to the newly deployed model version in stages according to the preset proportion, and decide whether to proceed to the next stage based on real-time monitoring indicators after the end of each stage; S54, continuously collect the inference service indicators reported through the sidecar agent, and automatically perform scaling operation when the indicators trigger the preset threshold. 8.The cloud-native based AI model full life cycle management method of claim 7, wherein, The steps of S6 include: S61, collect the following monitoring indicators in real time through the data collection agent deployed on the infrastructure, service mesh sidecar agent and custom exporter: Infrastructure indicators: GPU utilization, GPU memory occupancy; Model service indicators: request throughput, P50 / P99 inference delay, HTTP error rate; Data quality indicators: group stability index, KL divergence of feature distribution and benchmark distribution; S62, based on the normalized monitoring indicators, calculate the composite health score at a preset period, and the calculation formula is: wherein is a preset weight; S63, compare the calculated composite health score with the preset threshold; when the composite health score is lower than the threshold, automatically generate a governance instruction, which triggers two actions at the same time: Action one: switch all the traffic of the current faulty model to the last stable version recorded in the versioned model registry center through the service mesh API; Action two: generate a structured retraining trigger alarm event, which contains the faulty service identifier, the composite health score at the triggering time and the related key indicator snapshot, and is immediately captured by the step of event listening in S2; S64, record all the context information of this governance event, including the triggering indicators, the composite health score, the executed rollback operation and the triggered retraining event, as an audit log to the database. 9.The cloud-native based AI model full life cycle management method of claim 8, wherein, The specific process of the resource scheduling step includes: SS3-5-1, receive the scheduling request from the model training task or the inference service deployment, and parse the resource demand profile, which at least includes the required GPU type, memory size, estimated computing time and task priority label; SS3-5-2, through the cluster monitoring system, real-time acquisition of dynamic resource state of each computing node, including CPU core utilization rate, used and available video memory of each GPU card, current available bandwidth of inter-node network; SS3-5-3, for each node, based on real-time resource state, using a dynamic weight factor algorithm to calculate node score, the calculation formula is: wherein, are configurable weight coefficients, and ; SS3-5-4, compare the node scores of all nodes, select the node with the highest node score, and bind the task to be scheduled to the selected node for execution. 10.A cloud-native AI model full life cycle management system, characterized in that, The system is deployed on a Kubernetes container orchestration platform, including a plurality of cooperatively executed microservice modules, specifically including: A model development and environment solidification module for building a model based on a multi-tenant interactive development platform and automatically encapsulating model code, dependent libraries and runtime environments into immutable OCI images and pushing them to a container image repository; An event listening and triggering module for listening to code commit events from a version control repository, scheduled tasks, or retraining trigger alarm events generated by model performance monitoring; An automated workflow execution module for responding to the events listened to by driving an automated CI / CD pipeline through a declarative workflow engine; wherein the pipeline responding to the code commit event or the retraining trigger alarm event at least includes a task of pulling the corresponding image from the container image repository and executing model training, and outputs the trained model file; A versioned registration module for storing model files and metadata in a versioned model registry and binding model version numbers with source code commit records in the version control repository; An inference service deployment and operation module for deploying model inference services obtained from the registry based on a service mesh, performing a gray release strategy, and performing elastic scaling of the inference services based on real-time collected service quality indicators; A monitoring and autonomous governance module for establishing a multi-dimensional monitoring indicator system for the deployed model inference services, calculating a composite health score of the model based on the monitoring indicators; when the composite health score is lower than a preset threshold, automatically triggering model rollback to the last stable version, and generating the retraining trigger alarm event; A resource scheduling module for scheduling heterogeneous computing resources for model training tasks and model inference services based on a dynamic weight factor algorithm, which at least calculates node scores by integrating CPU utilization, available GPU memory and available network bandwidth of the node.

Images