Containerization-based AI model full life cycle management system and method

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a containerization-based AI model full life cycle management system and method, and the system comprises a data preprocessing module, an environment construction module, a flow arrangement module and a link control module. The method corresponds to the system. According to the containerization-based AI model full-life-cycle management system and method provided by the invention, the independent container mirror image of each stage is constructed, the multi-modal data is preprocessed, and automatic process arrangement and version and deployment management and control are carried out, so that the environment standardization isolation of each stage is realized to solve dependency conflicts, the multi-modal data is uniformly processed to avoid the format compatibility problem, and the management efficiency is improved. Human errors are reduced through full-process automatic cooperation, versions are traceable, the method adapts to various deployment environments, AI model management pain points are solved in a multi-dimensional mode, and development, iteration and large-scale landing of the AI model are efficiently supported.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence, and particularly relates to a container-based AI model full life cycle management system and method. BACKGROUND

[0002] With the wide application of artificial intelligence technology in image recognition, natural language processing, speech analysis and other fields, the development and application of AI models have shifted from single-stage technology implementation to full life cycle management covering data preparation, model training, inference service, deployment and operation, and archiving. In this process, enterprises and developers have increasingly high demands for AI model development efficiency, environment compatibility, and version traceability. However, the current traditional AI model management method still has significant technical bottlenecks and cannot meet the needs of full-process automation and standardization.

[0003] First, the environment of each processing stage of the AI model full life cycle is complex and has poor compatibility. In actual development, the data labeling stage needs to rely on special labeling tools, the model training stage needs to adapt to specific versions of deep learning frameworks, CUDA and cuDNN libraries, the inference service stage needs to match inference acceleration engines and hardware drivers, and the deployment and operation stage needs to integrate monitoring and resource scheduling components. The tool chains of these stages are often independent of each other, and there are differences in the configuration requirements of the running environment. Even if the existing container technology is used, it is usually necessary to manually write Dockerfile for each stage to build an image, which is not only tedious, but also difficult to ensure the consistency of the underlying dependency libraries of different stages, and may still cause mismatches between the training and deployment environments. Developers need to manually set up and debug the environment of each stage, which not only consumes a lot of time, but also easily causes problems such as the training environment being runnable and the deployment environment being unusable due to inconsistent environments, seriously affecting the model development and landing efficiency.

[0004] Second, the AI model full life cycle lacks unified process collaboration and automation capabilities. In the existing technology, the standardized data after data preprocessing needs to be manually transferred to the labeling tool, the labeled data set needs to be manually imported into the training framework, and the model file generated by training needs to be manually adapted to the hardware and network parameters of the deployment environment. The connection between stages relies on manual operation, and there is a lack of visual process arrangement tools and automated execution mechanisms. This decentralized management mode not only increases the risk of human error, but also leads to low efficiency of full-process execution, making it difficult to quickly respond to business needs for model iteration and deployment, especially in multi-modal data (image, text, audio) processing scenarios, the complexity of process collaboration is further aggravated.

[0005] In summary, the defects of the current AI model management technology in environmental compatibility and process automation have become key problems restricting the efficient development and large-scale landing of AI models, and a solution that can encapsulate the environment through standardized technology, automatically arrange the whole process, and manage the whole life cycle data in a correlated manner is urgently needed. SUMMARY

[0006] The present application aims to provide a container-based AI model whole life cycle management system and method to solve the technical problems raised in the background art.

[0007] To achieve the above-mentioned purpose, the following technical solutions are disclosed in the present application: In a first aspect, the present application discloses a container-based AI model whole life cycle management system, which comprises: A data preprocessing module configured to receive original AI training data of image, text and audio types, perform data cleaning, format standardization and feature preliminary extraction processing on the original AI training data, and output a preprocessed standardized training data set; An environment construction module configured to construct independent container images for each processing stage based on dependency analysis of tool chains for each processing stage of the AI model whole life cycle, encapsulate the tools required for each processing stage, and output a standardized container image file for each processing stage; wherein the processing stages include a data annotation stage, a model training stage, an inference service stage, a deployment and operation stage, and an offline archiving stage; A process arrangement module configured to receive user-configured whole life cycle process rules, associate the standardized container image file with the standardized training data set, and output a workflow execution scheme with stage dependency relationships and parameter configurations; A link control module configured to load the workflow execution scheme, perform version-associated storage on the annotation data set output by the data annotation stage, the trained model file output by the model training stage, the inference result data output by the inference service stage, and the operation log output by the deployment and operation stage, detect the hardware and network parameters of the target deployment environment, adjust the container resource quota and network strategy, output traceable model version records and deployment packages adapted to the target environment based on the results of the version-associated storage and the detected target environment parameters, and include the version hash value of the workflow execution scheme in the version snapshot for management.

[0008] Preferably, the data preprocessing module comprises: A data classification unit configured to classify the received original AI training data by data type, generate a classification label and bind it to the original data, and output the classified type-specific original data; The data cleaning unit is configured to perform noise filtering and outlier removal on the divided type original data, and output cleaned data; wherein the noise filtering includes image deblurring processing, text stopword processing and audio background noise processing. The feature extraction unit is configured to call an open computer vision library, a word vector tool and an audio feature extraction library to perform preliminary feature extraction on the cleaned data, convert the extracted features into fixed-dimensional vectors or tensors, and output preprocessed standardized training data sets.

[0009] Preferably, the environment construction module comprises: The dependency parsing unit is configured to scan tools in each processing stage of the AI model life cycle through static scanning tool scripts and dynamic analysis tool running logs, extract the names and version adaptation relationships of dependent libraries required by tool running, and generate a dependency list; wherein the scanning logic of each stage includes: identifying the necessary dependent library types for the normal running of the tools in each stage, and confirming the version compatibility constraints between dependent libraries; The staged image generation unit is configured to call the build command of the containerization construction tool based on the dependency list to build data labeling container images, model training container images, inference service container images, deployment and operation container images and offline archiving container images respectively, and preset tool startup scripts and data storage paths in each container image, and output standardized container image files for each processing stage; The image verification unit is configured to perform startup testing and dependency conflict detection on each generated container image, generate an image verification report, and if the verification fails, trigger the staged image generation unit to rebuild the container image, and based on the results of the dependency conflict detection, automatically recommend a compatible dependent library version list.

[0010] Preferably, the data labeling container image comprises: The labeling tool unit is configured to provide basic labeling interactive interfaces for image target detection labeling, text entity labeling and audio emotion labeling based on the tools in the data labeling stage; receive user-uploaded custom labeling templates, parse the labeling fields and labeling rules in the custom labeling templates, adjust the controls of the basic labeling interactive interfaces according to the parsed labeling fields and labeling rules, dynamically generate customized labeling interactive interfaces that adapt to user needs, and receive user operations in the customized labeling interactive interfaces to generate an initial labeling data set. The labeling quality verification unit is configured to perform consistency verification and integrity verification on the initial labeling data set, remove erroneous labeling data and supplement missing labels, and output a final labeling data set.

[0011] Preferably, the model training container image comprises: a distributed training unit configured to receive the standardized training dataset and the user-set training parameters based on a tool of the model training stage, schedule a multi-node GPU resource to perform model parallel training, and output a training intermediate model file; a training monitoring unit configured to collect loss value, accuracy, and GPU usage rate indicators in real time during the model parallel training process, generate a training curve and store it in a training log, trigger an early stop mechanism when there is no improvement in the training indicators for N consecutive iteration periods, and output a final trained model file and a training indicator report.

[0012] Preferably, the inference service container image includes: a model optimization unit configured to call a tool of the inference service stage, perform quantization and operator fusion on the trained model file output by the model training stage, and generate an optimized inference model file; an inference execution unit configured to receive an external inference request, load the optimized inference model file to perform inference calculation, output inference result data, and record the inference request volume and successful response rate to an inference service log.

[0013] Preferably, the deployment and operation container image includes: a resource monitoring unit configured to collect CPU usage, GPU usage, memory occupation, and network throughput indicators of a container runtime in real time based on a client of a tool of the deployment and operation stage, generate monitoring data at a preset time interval and push it to the monitoring tool, and output an operation log; an automatic scaling unit configured to receive the monitoring data output by the resource monitoring unit, increase a preset number of container replicas when the inference request volume is continuously below a preset threshold for a preset period of time, and reduce a preset number of container replicas when the inference request volume is continuously below a preset proportion of the preset threshold for a preset period of time.

[0014] Preferably, the process orchestration module includes: a rule configuration unit configured to provide a visual configuration interface, receive user input of stage execution order, inter-stage data transfer rules, and parameter thresholds, and output process configuration parameters; a workflow generation unit configured to associate the process configuration parameters with the standardized container image file and the standardized training dataset, generate a visual workflow with data annotation nodes, model training nodes, inference service nodes, deployment and operation nodes, and offline archiving nodes, and output a workflow execution scheme; wherein each node is labeled with a corresponding container image identifier and data input and output path; A flow verification unit is configured to perform logical verification and data format verification on the workflow execution scheme, and generate a verification report; if the verification fails, the user is prompted to adjust the flow configuration parameters.

[0015] Preferably, the link control module comprises: A version association unit is configured to bind the annotation data set output by the data annotation stage, the trained model file output by the model training stage, the inference result data output by the inference service stage, and the operation and maintenance log output by the deployment operation and maintenance stage with the container image hash value of the corresponding stage, generate an unalterable version snapshot, and output a traceable model version record; An environment adaptation unit is configured to detect the hardware resources of the target deployment environment through a secure shell protocol and the network parameters of the target deployment environment through an application programming interface, and adjust the container configuration according to the detection results, and output a deployment package adapted to the target environment after packaging the adjusted container configuration and the trained model file; An offline management unit is configured to, in the model offline stage, call the tools of the offline archiving stage, perform irreversible deletion on sensitive data, and generate an audit log that meets the data security regulations; at the same time, update the service routing configuration of the application programming interface gateway, and output an offline archiving report; wherein the sensitive data includes the annotation data set and the training log.

[0016] In a second aspect, the present application discloses a container-based AI model full life cycle management method, applied to the container-based AI model full life cycle management system as described above, which comprises: A data preprocessing step: receiving original AI training data of image, text and audio types, performing data cleaning, format standardization and feature preliminary extraction processing on the original AI training data, and outputting a preprocessed standardized training data set; An environment construction step: based on the dependency relationship analysis of the tool chain of each processing stage of the AI model full life cycle, constructing independent container images for each processing stage, and encapsulating the tools required by each processing stage, and outputting a standardized container image file for each processing stage; wherein the processing stages include the data annotation stage, the model training stage, the inference service stage, the deployment operation and maintenance stage, and the offline archiving stage; A flow arrangement step: receiving the full life cycle flow rules configured by the user, associating the standardized container image file with the standardized training data set, and outputting a workflow execution scheme with stage dependency relationship and parameter configuration; The link management step: loading the workflow execution scheme, storing the version association of the labeled data set output by the data labeling stage, the trained model file output by the model training stage, the inference result data output by the inference service stage, and the operation and maintenance log output by the deployment and operation stage, detecting the hardware and network parameters of the target deployment environment, adjusting the container resource quota and network strategy, outputting the traceable model version record and the deployment package adapted to the target environment based on the results of the version association storage and the detected target environment parameters; and including the version hash value of the workflow execution scheme in the version snapshot for management.

[0017] Beneficial effects: the container-based AI model full life cycle management system and method of the present application solves the technical problems of environment dependence conflict and difficulty in migration in different stages by constructing independent and standardized container images for each stage of the full life cycle, achieves the effect of out-of-box use and one-time construction running anywhere, and greatly reduces the environment adaptation cost; secondly, by performing cleaning, format standardization and feature preliminary extraction on image, text and audio multi-modal raw data, unified preprocessing of multi-modal data is realized, and standardized training data set is output to adapt to subsequent tools, avoiding manual secondary conversion caused by incompatible data formats; thirdly, by receiving user-configured process rules, associating standardized container images and data set generation workflow execution scheme, realizing automatic collaboration of the full life cycle process, reducing human operation errors, and greatly improving the efficiency of the whole process; in addition, by binding data, models, logs and other assets with the container image hash value, an unalterable version snapshot is created, realizing end-to-end accurate tracing from data to model to deployment environment; by sensing the target environment and dynamically generating adaptive configuration, the ability of one-time construction and automatic adaptation to multi-environment deployment is realized, completely solving the flexibility problem of model cross-environment deployment and improving deployment flexibility. Therefore, the container-based AI model full life cycle management system and method of the present application solves the existing AI model management pain points from the aspects of environment, data, process, version and deployment, and efficiently supports the development, iteration and large-scale landing of AI models. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0019] Figure 1 The structural block diagram of the container-based AI model full life cycle management system provided by the embodiments of the present application. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] In this document, the term "comprising" is intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] This embodiment provides, in a first aspect, a method such as Figure 1 The containerized AI model lifecycle management system shown is suitable for the full-process management of multimodal AI models such as image recognition, natural language processing, and speech sentiment analysis. Its core lies in achieving environmental isolation and standardization at each stage through containerization technology, combined with process orchestration and link control to achieve full lifecycle automation. The system includes a data preprocessing module, an environment construction module, a process orchestration module, and a link control module.

[0023] Specifically The data preprocessing module is configured to receive raw AI training data in the form of images (such as JPG / PNG images in RGB format), text (such as news corpora in TXT format), and audio (such as speech segments in WAV format). It performs data cleaning (removing redundant data, such as duplicate image samples; eliminating anomalous data, such as images with abnormal sizes or text containing garbled characters; identifying and deleting outliers in numerical features based on the 3-standard-deviation principle), format standardization (converting images to 224×224 pixel tensor format, text to fixed-length word vector matrices, and audio to Mel-spectral feature vectors, uniformly adapting to the input format for subsequent model training), and preliminary feature extraction (extracting contour features from images using edge detection algorithms, extracting keyword features from text using word frequency statistics, and extracting spectral features from audio using short-time Fourier transform). Finally, it outputs a preprocessed, standardized training dataset (stored in HDF5 format, containing data samples, labels, and feature description information).

[0024] An environment construction module is configured to construct independent container images for each processing stage of the AI model life cycle, including the data labeling stage, the model training stage, the inference service stage, the deployment and operation stage, and the offline archiving stage, and encapsulate the tools required for each processing stage (such as LabelImg tool for the data labeling stage, PyTorch framework for the model training stage, TensorRT engine for the inference service stage, etc.), and output standardized container image files for each processing stage (including tool binary files, dependent libraries, and configuration scripts). For example, the container image for the data labeling stage encapsulates the LabelImg tool and its dependent PyQt5 library, and the container image for the model training stage encapsulates the PyTorch framework and its dependent CUDA library, ensuring that the tools in each stage can run independently in an isolated environment.

[0025] A process orchestration module is configured to receive the life cycle process rules configured by the user through the visual interface (including stage execution order, such as "data labeling → model training → inference service → deployment and operation"; inter-stage data transfer logic, such as annotated dataset transferred to the model training stage through a shared storage directory; parameter threshold, such as model training iteration number ≥ 1000 times), associate the standardized container image files (such as data labeling image labelimg:v1.8.0, training image pytorch:1.13.1) with the standardized training dataset, and generate a workflow execution scheme with stage dependency and parameter configuration (including each stage container image ID, data input / output path, execution trigger condition). For example, the model training stage can be configured to automatically trigger after the final annotated dataset is output by the data labeling stage in the workflow.

[0026] A link control module is configured to load the workflow execution scheme, and store the annotated dataset output by the data labeling stage, the trained model file output by the model training stage, the inference result data (including classification label and confidence) output by the inference service stage, and the operation log (including CPU / GPU usage) output by the deployment and operation stage in version association (binding the SHA256 hash value of each stage container image to form a version snapshot) through a distributed file system (such as HDFS); at the same time, detect the hardware parameters (CPU core number, GPU model, memory capacity) and network parameters (bandwidth, delay, port opening state) of the target deployment environment (such as edge server, private cloud cluster, public cloud instance) through SSH protocol, adjust the container resource quota (such as allocating 2 NVIDIA A100 graphics cards for GPU-intensive tasks) and network strategy (such as configuring port mapping rules for inter-container communication) according to the detection results, and finally output traceable model version records (stored in the form of blockchain to ensure non-tamperability) and deployment packages adapted to the target environment (including container configuration files and model files).

[0027] Based on the above, the container-based AI model full life cycle management system of the embodiment, through unified data cleaning, format standardization and feature preliminary extraction, converts scattered image, text, audio raw data into standardized training data set suitable for the full life cycle, completely solves the problem of incompatible format of traditional multi-modal data and the need for manual secondary conversion; by constructing independent container image and encapsulating tools for data labeling, model training and other five stages, the running environment of each stage is isolated and standardized, the environment dependence conflict of the training environment being runnable and the deployment environment being unusable is avoided, and the environment adaptation time cost is greatly reduced; by associating the standardized image and the data set to generate a work flow execution scheme with stage dependence, the full life cycle is shifted from manual connection to automation collaboration, human operation errors are reduced, and the process execution efficiency is improved; through version association storage and deployment environment parameter dynamic adjustment, not only the traceability (binding container image hash value) of key products such as labeled data and model files is realized, but also the model can quickly adapt to different target environments, solving the pain points of version confusion and low deployment flexibility in traditional management, and providing full-process support for AI model from development to landing.

[0028] In the embodiment, the data preprocessing module includes the following units: A data classification unit configured to classify the received raw AI training data by data type (image, text, audio), generate classification tags (such as “image”, “text”, “audio”) through file suffixes (such as.jpg / .png for images,.txt for text,.wav / .mp3 for audio), and bind them with the raw data (stored in key-value pair format), and output the classified type raw data (stored in different directories according to type).

[0029] A data cleaning unit configured to filter noise and remove outliers from the type raw data: where the image deblurring process uses a Gaussian filter algorithm to remove motion blur, the text stop word removal process filters out meaningless words such as “of” and “is” based on the NLTK library, and the audio background noise removal process separates human voice from environmental noise through spectral subtraction; outlier removal filters images smaller than 32x32 pixels, texts shorter than 10 characters, and audios shorter than 1 second, and outputs the cleaned data.

[0030] The feature extraction unit is configured to call OpenCV (Open Source Computer Vision Library), Word2Vec and Librosa to perform preliminary feature extraction on the cleaned data: edge features of an image are extracted by the Canny algorithm of OpenCV, text is converted into a 300-dimensional word vector by a Word2Vec model, and mel-frequency cepstral coefficient (MFCC) features of audio are extracted by Librosa; the extracted features are uniformly converted into fixed-dimensional vectors or tensors (for example, an image feature is a 224x224x3 tensor, and a text feature is a 1x300 vector), and the preprocessed standardized training data set is output.

[0031] Based on the above refinement design of the data preprocessing module, the data classification unit binds labels by type and stores by category, realizes the structured management of the original data, avoids the subsequent call confusion caused by the mixed storage of image, text and audio data, and improves the data retrieval efficiency; the data cleaning unit effectively removes invalid information in the original data through targeted noise filtering (image deblurring, text stop word removal, etc.) and outlier removal, reduces the interference of noise on subsequent labeling and training, and improves the data quality; the feature extraction unit realizes preliminary extraction and fixed dimension conversion of multi-modal features by calling professional libraries, ensures that the output standardized training data set can directly adapt to the labeling and training tools in the subsequent container image, and does not need to adjust the feature format additionally, lays a data foundation for smooth connection of each stage in the whole life cycle.

[0032] In the embodiment, the environment construction module includes the following units: The dependency analysis unit is configured to scan tools in each processing stage of the whole life cycle of the AI model through static scanning tools scripts and dynamic analysis tool logs, extract the names and version adaptation relationships of the dependent libraries required by the tool running, and generate a dependency list; wherein the scanning logic of each stage includes: for each stage of tool, identifying the necessary dependent library types (such as Python library, system toolkit, driver) in its normal running, and confirming the version compatibility constraints between dependent libraries (such as TensorFlow2.10 needs to match CUDA11.2, and cannot adapt to CUDA12.0). Specifically: Data labeling stage: the tools are image labeling special tools (LabelImg) and multi-modal labeling platform (LabelStudio), which need to scan dependent Python libraries such as PyQt5 (≥5.15.0) and requests (2.25.1); Model training stage: The tool is a deep learning training framework (TensorFlow, PyTorch), which needs to scan TensorFlow2.10 compatible CUDA11.2, cuDNN8.1 and NumPy(≥1.21.0); Inference service stage: The tool is a model inference acceleration engine (TensorRT, full name TensorRuntime), and an open neural network exchange runtime (ONNXRuntime), which needs to scan GPU drivers compatible with TensorRT8.5(≥525.60.13), ONNX1.12 protocol compatible with ONNXRuntime1.14.0; Deployment and operation stage: The tool is a monitoring data collection tool (Prometheus) and a monitoring data visualization platform (Grafana), which needs to scan Go1.20 compatible with Prometheus2.45 and Node.js(≥16.0) compatible with Grafana; Offline archiving stage: The tool is a data security destruction tool (srm tool, Shred tool), which needs to scan the Linux system tool package such as coreutils(≥8.32) and other dependent tools.

[0033] Stage-by-stage image generation unit: configured to call the build command of containerization construction tool (Docker) based on the dependency list to build data annotation container image, model training container image, inference service container image, deployment and operation container image and offline archiving container image respectively; and preset tool startup script and data storage path in each container image, finally output standardized container image file of each processing stage.

[0034] Image verification unit: configured to perform startup test (run container and check whether the tool can be normally initialized, such as whether LabelImg can pop up annotation interface) and dependency conflict detection (check whether dynamic link library is missing through ldd command, check Python library compatibility through pipcheck command) on each container image generated, generate image verification report (including test result and error log); if the verification fails (such as dependency library version conflict), trigger the stage-by-stage image generation unit to rebuild the container image (automatically adjust the dependency library version and then execute the build command again).

[0035] Based on the technical design of the above environment construction module, the dependency analysis unit generates accurate dependency list by scanning the dependency library name and version adaptation relationship of each stage tool, avoiding tool running failure caused by dependency missing or version conflict from the source, solving the problem of dependency checking difficulty in traditional environment construction; The image generation unit generates images in stages by pre-setting the startup script and data storage path, so that the container image not only encapsulates the tool, but also has the ability to start and use immediately, and the developer does not need to manually configure the tool running parameters, which greatly shortens the environment ready time; The image verification unit detects the unqualified image through startup test and dependency conflict detection, and triggers the re-building to avoid the unqualified image flowing into the subsequent process and causing the whole life cycle interruption, and improves the reliability and stability of the container environment.

[0036] Firstly, the data labeling container image includes: The labeling tool unit is configured to provide basic labeling interactive interface of image target detection labeling (including canvas, rectangle box drawing tool, category label drop-down box), text entity labeling (including text editing area, entity type check box, position highlighting button) and audio emotion labeling (including waveform display area, play / pause control, emotion label single selection button) based on the tool of the data labeling stage (LabelImg, LabelStudio); and receives the user uploaded custom labeling template, parses the labeling fields (such as "class", "coordinates") and labeling rules (such as "category label only allows to select cat / dog", "coordinate value must be an integer of 0-1000") in the custom labeling template; according to the parsed labeling fields and rules, adjust the controls of the basic labeling interactive interface (such as only display cat / dog options in the category drop-down box, limit the value range of the coordinate input box), dynamically generate customized labeling interactive interface adapted to user needs; receive user operations in the customized labeling interactive interface (such as marking the cat in the image by drawing a rectangle box, selecting the "cat" label), and generate an initial labeling data set (record target coordinates, label and labeling time).

[0037] The labeling quality checking unit is configured to perform consistency checking (randomly select 15% of the labeling data, re-label by two labeling personnel, calculate the IOU value or text entity matching rate of the two labeling, and determine that it is inconsistent if it is less than 80%) and integrity checking (detect samples without corresponding labeling files in the original data directory) on the initial labeling data set; eliminate incorrect labeling data (such as rectangle box coordinates exceeding the image size range) and supplement missing labeling (prompt the user to label the unmarked samples), and finally output the final labeling data set (including labeling files and checking report).

[0038] Based on the above data labeling container image design, the labeling tool unit provides a basic labeling interaction interface and adjusts the control according to the user-defined template to form a customized interface, which can meet the labeling needs of images, texts, and audios in multiple scenarios, and adapt to user's personalized labeling rules (such as specific class labels and coordinate range limits), avoiding cumbersome labeling operations due to interface mismatch and improving labeling efficiency; the labeling quality checking unit effectively eliminates incorrect labeling (such as boundary box out-of-range) and supplements missing labeling through consistency checking (manual review comparison) and integrity checking (detecting un-labeled data), ensuring that the output labeled data set is accurate and complete, avoiding the impact of low-quality labeled data on subsequent model training, and providing data support for model performance improvement.

[0039] Secondly, the model training container image includes: The distributed training unit is configured to receive standardized training data sets and user-set training parameters (such as learning rate 0.001, batch size 32, and iteration number 2000) based on model training stage tools (such as PyTorch and Horovod), and execute model parallel training (split model layers to different GPUs and synchronize gradient updates) through Horovod framework scheduling multi-node GPU resources (such as 4 servers, each with 8 NVIDIA A100 graphics cards), and save a training intermediate model file (containing weight parameters and optimizer state) every 100 iteration cycles.

[0040] The training monitoring unit is configured to collect loss values (such as cross-entropy loss), accuracy (such as Top-1 accuracy), and GPU usage (such as memory occupancy and computing power utilization) indicators in the model parallel training process in real time through TensorBoard, generate training curves (loss decline curve and accuracy rise curve), and store them in training logs (recorded every 5 minutes); when the training indicators (such as accuracy) do not improve (improvement amplitude <0.1%) for a continuous preset number (such as 5) of iteration cycles, the early stopping mechanism is triggered, the training is stopped, and the final post-training model file (containing the optimal weight parameters) and the training indicator report (containing curves and key indicator summary) are output.

[0041] Based on the above model training container image design, the distributed training unit executes parallel training by scheduling multi-node GPU resources, which greatly shortens the model training time compared to single-node training, especially suitable for large-scale multi-modal data set training scenarios, solving the problem of long training time and low efficiency in traditional training; the training monitoring unit collects loss values, accuracy, and other indicators in real time and generates training curves, allowing developers to intuitively understand the training progress; at the same time, when the training indicators do not improve continuously, the early stopping mechanism is triggered, avoiding the waste of GPU resources caused by invalid iterations, and ensuring that the final model file output has the optimal training effect, balancing training efficiency and model quality.

[0042] Thirdly, the inference service container image includes: Model optimization unit: a tool configured to call the inference service stage (TensorRT, ONNXRuntime), quantize the post-training model file output by the model training stage (convert 32-bit floating-point precision to 8-bit integer precision (INT8) or 16-bit floating-point precision (FP16), reduce model volume and calculation amount) and operator fusion (combine continuous operators such as Conv2D, BN, ReLU into a single operator, reduce calculation steps), generate an optimized inference model file.

[0043] Inference execution unit: configured to receive external inference requests (including data to be inferred) through RESTful API, load the optimized inference model file to perform inference calculation (such as forward propagation through TensorRT's C++ API), output inference result data, and record inference request volume (number of requests per hour) and successful response rate (percentage of requests that successfully return results) to inference service log (including timestamp, request ID, result status).

[0044] Based on the design of the above inference service container image, the model optimization unit reduces the model storage volume by quantizing and fusing the post-training model, reduces the inference calculation steps, shortens the inference time consumption, solves the problem of large model inference volume and high delay, and meets the real-time inference scene demand; the inference execution unit records the inference request volume and successful response rate to the inference service log, which facilitates the developer to monitor the running state of the inference service (such as request peak, fault frequency), troubleshoot service exceptions in a timely manner, and ensure the stable operation of the inference service.

[0045] Fourthly, the deployment and operation container image includes: Resource monitoring unit: configured as a client based on the tools (Prometheus, Grafana) of the deployment and operation stage, real-time collection of CPU usage (core occupancy percentage), GPU usage (computing power utilization), memory occupancy (used / total capacity), and network throughput (sent / received byte number per second) indicators through the exporter component of Prometheus; generate monitoring data once per preset time interval (such as 30 seconds) and push to Grafana dashboard (show real-time line chart, set threshold alarm line, such as trigger yellow alarm when CPU usage > 80%), output operation log (including indicator name, value, collection time).

[0046] Automatic scaling unit: configured to receive the monitoring data output by the resource monitoring unit, when the inference request volume is continuously less than the preset threshold (such as 1000 times / minute) for a preset length of time (such as 5 minutes), increase the number of container replicas by a preset number of additions (such as 2-4) through the HPA (HorizontalPodAutoscaler) of Kubernetes; when the inference request volume is continuously less than the preset proportion (such as 50%) of the preset threshold for a preset length of time (such as 5 minutes), reduce the number of container replicas by a preset number of deletions (such as 1-2) to maintain the balance between supply and demand of container resources (to ensure that the average response time is <100 ms).

[0047] Based on the above design of deploying and operating the container image, the resource monitoring unit collects container running indicators such as CPU usage and GPU usage in real time and pushes them to the visualization platform, allowing developers to monitor resource consumption in real time and promptly identify resource overload or idling problems; the automatic scaling unit dynamically adjusts the number of container replicas according to the inference request volume (increases replicas during peak requests and reduces replicas during troughs), which can not only avoid service overload (such as increased inference delay) due to insufficient replicas, but also avoid resource waste (such as idle CPU) due to excessive replicas, achieving a balance between supply and demand of container resources and improving resource utilization efficiency.

[0048] In this embodiment, the process orchestration module includes the following units: Rule configuration unit: configured to provide a web-based visualization configuration interface (based on Vue.js development), receive user input of stage execution order (adjusted by dragging and dropping nodes, such as "data preprocessing → data annotation → model training"), inter-stage data transfer rules (such as transferring the annotated dataset to the training stage through the NFS shared directory), and parameter thresholds (such as the accuracy of model training must be ≥90%, otherwise trigger retraining), and output process configuration parameters.

[0049] Workflow generation unit: configured to associate the process configuration parameters with the standardized container image file and the standardized training dataset, generate a visual workflow (such as a DAG graph showing the dependency relationship between nodes) with data annotation nodes, model training nodes, inference service nodes, deployment and operation nodes, and offline archiving nodes through Apache Airflow, and output a workflow execution scheme (including the image ID, data input and output path, and resource requirements (such as the number of GPUs) of each node); each node is labeled with the corresponding container image identifier (such as the SHA256 hash value of the image) and the data input and output path (such as the input path of the annotation node is / data / standardized, and the output path is / data / annotation).

[0050] The process verification unit is configured to perform logical verification on the workflow execution scheme (check whether there is a circular dependency in the stage sequence, such as reverse dependency of "model training → data labeling"), and data format verification (check whether the data set format is compatible with the container image tool, such as whether the labeling tool supports HDF5 format input), generate a verification report (list the error type and location); if the verification fails (such as there is a circular dependency), the user is prompted to adjust the process configuration parameters through the interface pop-up window (such as "please adjust the stage sequence to eliminate the circular dependency").

[0051] Based on the design of the above process orchestration module, the rule configuration unit receives user configuration through the visual interface, which is more intuitive and easy to operate than traditional code-based configuration, reduces the technical threshold of process configuration, and allows non-professional developers to quickly complete the whole life cycle rule setting; the workflow generation unit generates a visual workflow with node dependency and path annotation by associating images and data sets, realizes the integration of image-data-process, and avoids the connection fault caused by scattered configuration in each stage; the process verification unit avoids process errors in advance through logical verification (such as detecting stage circular dependency) and data format verification (such as detecting data set and image compatibility), reduces the whole life cycle rework caused by process problems, and improves the correctness and efficiency of process orchestration.

[0052] In the present embodiment, the link control module includes the following units: The version association unit is configured to bind the labeled data set output by the data labeling stage, the trained model file output by the model training stage, the inference result data output by the inference service stage, and the operation and maintenance log output by the deployment and operation and maintenance stage with the container image hash value (such as the SHA256 value of the labeling image) of the corresponding stage through a hash algorithm (SHA256), to generate an unalterable version snapshot (stored in a distributed database such as MongoDB), and output a traceable model version record (including data hash, image version, generation time of each stage, supporting version number query).

[0053] An environment adaptation unit is configured to log in to a target deployment environment (such as an edge device NVIDIA Jetson AGX) through a secure shell protocol (SSH), execute commands (such as nvidia-smi and free-m) to detect hardware resources (GPU model and memory capacity), detect network parameters (in-cluster node communication delay and port opening list) of the target deployment environment through an application programming interface (API, such as a Kubernetes API), adjust container configuration (such as assigning 1 GPU to an edge device and 4 GPUs to a cloud server) according to the detection results, select an optimal template from a preconfigured container configuration template library or generate an adapted container orchestration file (such as a Kubernetes YAML) in real time, and pack the configuration and an optimized and version-bound post-training model file to output a deployment package that is adapted to the target environment and ready for plug-and-play deployment.

[0054] An offline management unit is configured to, in a model offline stage, call tools (srm tools and Shred tools) in an offline archiving container image, perform irreversible deletion (file contents are overwritten multiple times and then deleted through an srm-rf command to ensure that the data cannot be recovered) on sensitive data (annotated data sets and training logs), and generate an audit log that meets data security regulations; meanwhile, the service routing configuration is updated (URL mapping of the offline model is removed) through a management interface of an API gateway, and an offline archiving report (containing a list of deleted data, routing update records, and operation time) is output.

[0055] Based on the design of the link management module, the version association unit generates a version snapshot by binding key data at each stage to a container image hash value, forming a unique association relationship among data, images, and versions. When a model has performance problems or deployment failures, the root cause (such as annotation data errors or incompatible image versions) can be quickly traced and located, solving the problem of difficult traceability in traditional version management. The environment adaptation unit detects target environment parameters and adjusts container configuration, allowing the model to adapt to different environments such as edge devices, private clouds, and public clouds without manual modification, improving deployment flexibility. The offline management unit performs irreversible deletion of sensitive data and updates service routing, which not only prevents sensitive information such as annotated data sets and training logs from being leaked, but also ensures that the offline model completely exits the service, meeting data security and compliance requirements.

[0056] The embodiment provides, in a second aspect, a container-based AI model full-life-cycle management method, which is applied to the container-based AI model full-life-cycle management system as described above, and the method comprises the following steps: A data preprocessing step: receiving original AI training data of image, text, and audio types, performing data cleaning, format standardization, and feature preliminary extraction processing on the original AI training data, and outputting a preprocessed standardized training data set; The environment construction step includes: based on the dependency analysis of the tool chain of each processing stage of the AI model full life cycle, constructing an independent container image for each processing stage, and encapsulating the tools required by each processing stage, and outputting a standardized container image file for each processing stage; wherein the processing stage includes a data annotation stage, a model training stage, an inference service stage, a deployment and operation stage, and an offline archiving stage. The process arrangement step includes: receiving a user-configured full life cycle process rule, associating the standardized container image file with the standardized training data set, and outputting a workflow execution scheme with stage dependencies and parameter configurations. The link management step includes: loading the workflow execution scheme, version-associating and storing the annotation data set output by the data annotation stage, the trained model file output by the model training stage, the inference result data output by the inference service stage, and the operation log output by the deployment and operation stage, detecting the hardware and network parameters of the target deployment environment, adjusting the container resource quota and network strategy, based on the version-associating and storing result and the detected target environment parameters, outputting a traceable model version record and a deployment package adapted to the target environment; and including the version hash value of the workflow execution scheme in the version snapshot for management.

[0057] It should be noted that the container-based AI model full life cycle management method of the present embodiment corresponds to the container-based AI model full life cycle management system described above, and therefore, the parts not described in detail in the container-based AI model full life cycle management method of the present embodiment (including but not limited to specific implementation means, technical effects, etc.) can be correspondingly referred to the related description in the container-based AI model full life cycle management system described above, which will not be repeated here.

[0058] In the embodiments provided by the present application, it should be understood that the embodiments described herein can be realized by hardware, software, firmware, middleware, codes or any proper combination thereof. For hardware implementation, the processor can be realized in one or more of the following components: an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor, a controller, a microcontroller, a microprocessor, other electronic units designed to perform the functions described herein, or a combination thereof. For software implementation, the procedures described herein can be implemented with a computer program that directs relevant hardware to complete the procedures. When implemented, the above program can be stored in a computer readable storage medium or transmitted as one or more instructions or codes on a computer readable storage medium. The computer readable storage medium includes computer storage media and communication media including any medium that facilitates the transfer of a computer program from one place to another. The storage medium can be any available medium that can be accessed by a computer. The computer readable storage medium can include but not limited to RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage medium or other magnetic storage devices, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer.

[0059] Finally, it should be noted that the above only describes the preferred embodiments of the present application and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified or some technical features can be replaced by equivalent ones, and any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A containerized AI model lifecycle management system, characterized in that, The system includes: The data preprocessing module is configured to receive raw AI training data in the form of images, text and audio, perform data cleaning, format standardization and preliminary feature extraction on the raw AI training data, and output the preprocessed standardized training dataset. The environment building module is configured to: based on the dependency analysis of the toolchain for each processing stage of the entire AI model lifecycle, build independent container images for each processing stage, encapsulate the tools required for each processing stage, and output standardized container image files for each processing stage; wherein the processing stages include the data annotation stage, model training stage, inference service stage, deployment and maintenance stage, and offline archiving stage. The process orchestration module is configured to: receive user-configured full lifecycle process rules, associate the standardized container image file with the standardized training dataset, and output a workflow execution scheme with stage dependencies and parameter configurations; The link management module is configured to: load the workflow execution plan, perform version-related storage on the labeled dataset output in the data labeling stage, the trained model file output in the model training stage, the inference result data output in the inference service stage, and the operation and maintenance log output in the deployment and maintenance stage; simultaneously detect the hardware and network parameters of the target deployment environment, adjust container resource quotas and network policies; based on the results of the version-related storage and the detected target environment parameters, output a traceable model version record and a deployment package adapted to the target environment; and include the version hash value of the workflow execution plan in the version snapshot for management.

2. The containerized AI model lifecycle management system according to claim 1, characterized in that, The data preprocessing module includes: The data classification unit is configured to: classify the received raw AI training data according to data type, generate classification labels and bind them to the raw data, and output the classified raw data. The data cleaning unit is configured to: perform noise filtering and outlier removal on the categorized raw data, and output cleaned data; wherein the noise filtering includes image deblurring, text stop word removal, and audio background noise removal. The feature extraction unit is configured to: call open computer vision libraries, word vector tools, and audio feature extraction libraries to perform preliminary feature extraction on the cleaned data, convert the extracted features into vectors or tensors of fixed dimensions, and output the preprocessed standardized training dataset.

3. The containerized AI model lifecycle management system according to claim 1, characterized in that, The environment construction module includes: The dependency resolution unit is configured to: scan the tools at each processing stage of the entire lifecycle of the AI ​​model through static scanning tool scripts and dynamic analysis tool operation logs, extract the names and version compatibility relationships of the dependency libraries required for tool operation, and generate a dependency list; wherein, the scanning logic of each stage includes: for the tools at each stage, identifying the types of dependency libraries necessary for their normal operation, and confirming the version compatibility constraints between dependency libraries. The phased image generation unit is configured to: based on the dependency list, call the build command of the containerized build tool to build data annotation container image, model training container image, inference service container image, deployment and maintenance container image and offline archiving container image respectively, and preset the tool startup script and data storage path in each container image, and output the standardized container image file for each processing stage; The image verification unit is configured to perform startup tests and dependency conflict detection on each generated container image, generate an image verification report, and if the verification fails, trigger the phased image generation unit to rebuild the container image. Based on the result of the dependency conflict detection, it can automatically recommend a list of compatible dependency library versions.

4. The containerized AI model lifecycle management system according to claim 3, characterized in that, The data annotation container image includes: The annotation tool unit is configured to: provide a basic annotation interactive interface for image object detection annotation, text entity annotation, and audio sentiment annotation based on the tools in the data annotation stage; receive user-uploaded custom annotation templates, parse the annotation fields and rules in the custom annotation templates, adjust the controls of the basic annotation interactive interface according to the parsed annotation fields and rules, dynamically generate a customized annotation interactive interface adapted to user needs; and receive user operations in the customized annotation interactive interface to generate an initial annotation dataset. The annotation quality verification unit is configured to: perform consistency and integrity verification on the initial annotation dataset, remove erroneous annotation data and supplement missing annotations, and output the final annotation dataset.

5. The containerized AI model lifecycle management system according to claim 3, characterized in that, The model training container image includes: The distributed training unit is configured as: a tool based on the model training phase, which receives a standardized training dataset and user-defined training parameters, schedules multi-node GPU resources to perform parallel model training, and outputs intermediate training model files; The training monitoring unit is configured to: collect loss values, accuracy, and GPU utilization metrics in real time during the parallel training process of the model, generate training curves and store them in the training log; trigger an early stopping mechanism when the training metrics do not improve for N consecutive iterations, and output the final post-trained model file and training metric report.

6. The containerized AI model lifecycle management system according to claim 3, characterized in that, The inference service container image includes: The model optimization unit is configured to: invoke the tools in the inference service phase to quantize and fuse the trained model file output in the model training phase, and generate an optimized inference model file; The inference execution unit is configured to: receive external inference requests, load the optimized inference model file, perform inference calculations, output inference result data, and record the number of inference requests and the success rate in the inference service log.

7. The containerized AI model lifecycle management system according to claim 3, characterized in that, The deployed and maintained container image includes: The resource monitoring unit is configured to: collect CPU utilization, GPU utilization, memory usage and network throughput metrics of containers in real time on the client side of the tools during the deployment and maintenance phase, generate monitoring data once at preset time intervals and push it to the monitoring tools, and output maintenance logs. The automatic scaling unit is configured to: receive monitoring data output by the resource monitoring unit; when the inference request volume is continuously not less than a preset threshold for a preset duration, increase the number of container replicas by a preset number; when the inference request volume is continuously not less than a preset proportion of the preset threshold for a preset duration, decrease the number of container replicas by a preset number.

8. The containerized AI model lifecycle management system according to claim 1, characterized in that, The process orchestration module includes: The rule configuration unit is configured to: provide a visual configuration interface, receive user input on the stage execution order, inter-stage data transfer rules and parameter thresholds, and output process configuration parameters; The workflow generation unit is configured to associate the process configuration parameters with the standardized container image file and the standardized training dataset to generate a visual workflow with data annotation nodes, model training nodes, inference service nodes, deployment and maintenance nodes, and offline archiving nodes, and output the workflow execution plan; wherein each node is labeled with the corresponding container image identifier and data input / output path; The process verification unit is configured to: perform logical verification and data format verification on the workflow execution plan, and generate a verification report; if the verification fails, the user is prompted to adjust the process configuration parameters.

9. The containerized AI model lifecycle management system according to claim 1, characterized in that, The link management module includes: The version association unit is configured to bind the labeled dataset output in the data annotation phase, the trained model file output in the model training phase, the inference result data output in the inference service phase, and the operation and maintenance logs output in the deployment and operation and maintenance phase with the container image hash value of the corresponding phase, generate an immutable version snapshot, and output a traceable model version record. The environment adaptation unit is configured to: detect the hardware resources of the target deployment environment through the secure shell protocol and the network parameters of the target deployment environment through the application programming interface, adjust the container configuration according to the detection results, and package the adjusted container configuration and the trained model file to output a deployment package adapted to the target environment. The offline management unit is configured to: during the model offline phase, invoke the tools in the offline archiving phase to perform irreversible deletion of sensitive data and generate audit logs that comply with data security regulations; at the same time, update the service routing configuration of the application programming interface gateway and output an offline archiving report; wherein the sensitive data includes labeled datasets and training logs.

10. A containerized AI model lifecycle management method, applied to the containerized AI model lifecycle management system as described in any one of claims 1-9, characterized in that, The method includes: Data preprocessing steps: Receive raw AI training data in image, text and audio formats, perform data cleaning, format standardization and preliminary feature extraction on the raw AI training data, and output the preprocessed standardized training dataset; Environment setup steps: Based on the dependency analysis of the toolchain for each processing stage of the AI ​​model's entire lifecycle, independent container images for each processing stage are built, and the tools required for each processing stage are encapsulated, outputting standardized container image files for each processing stage; wherein the processing stages include the data annotation stage, model training stage, inference service stage, deployment and maintenance stage, and offline archiving stage. Process orchestration steps: Receive the user-configured full lifecycle process rules, associate the standardized container image file with the standardized training dataset, and output a workflow execution plan with stage dependencies and parameter configurations; Link management steps: Load the workflow execution plan, perform version association storage on the labeled dataset output in the data annotation stage, the trained model file output in the model training stage, the inference result data output in the inference service stage, and the operation and maintenance log output in the deployment and maintenance stage, and simultaneously detect the hardware and network parameters of the target deployment environment, adjust container resource quotas and network policies, and based on the results of the version association storage and the detected target environment parameters, output a traceable model version record and a deployment package adapted to the target environment; and include the version hash value of the workflow execution plan in the version snapshot for management.

Images