Artificial intelligence whole-process automatic learning system based on assembly line

Abstract

The invention provides an artificial intelligence whole-process automatic learning system based on an assembly line. Comprising an assembly line arrangement unit used for performing metadata association, state synchronization and task scheduling on each function unit in the same assembly line; the data management unit is used for acquiring the training data from the heterogeneous data source and performing format standardization, preprocessing and versioning storage on the training data; the training management unit is used for executing multiple rounds of model training and parameter tuning on the computing resource cluster to generate a temporary model file; the model evaluation and management unit is used for extracting a corresponding temporary model according to the identifier of the temporary model file, and performing evaluation index calculation and index comparison on the temporary model; and the release management unit is used for constructing a service package and pushing the service package to a preset DevOps mirror image warehouse to trigger a deployment process. According to the method, data version traceability, training process standardization, continuous retention of evaluation indexes, transparent model release and easy rollback can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a pipeline-based artificial intelligence full-process automatic learning system. BACKGROUND

[0002] With the large-scale landing of artificial intelligence algorithms in the fields of computer vision, natural language processing, search recommendation, prediction analysis, etc., algorithm research and development has become a key support for enterprise digital transformation. Modern algorithms have fast iteration speed and large model size, which puts forward higher requirements for data quality, computing resources and engineering delivery process; therefore, building an efficient, traceable and easy-to-collaborate development environment has become an industry consensus.

[0003] In traditional practice, each business line usually stores training data on local servers or scattered third-party platforms, lacks centralized data lakes or data warehouses, and thus it is difficult to unify data versions and scattered permission management, which not only increases security risks but also hinders cross-team data reuse. Algorithm engineers often need to master multiple sets of notebooks, distributed training frameworks and tuning tools, and the training processes of models in different fields are self-governed and cannot form standardized pipelines; once key personnel leave, data preprocessing scripts or training parameters are difficult to reproduce, and the research and development chain is easily interrupted.

[0004] Code repositories, inference images and online scripts are also hosted on different systems, lacking unified version management and approval mechanisms; model evaluation often relies on temporary scripts, and indicators are recorded manually, making it difficult to retrieve or quantitatively compare historical best results. The model release process lacks deep integration with the enterprise DevOps system, and the configuration is not transparent, rollback is difficult, and maintenance costs are high. Overall, the existing decentralized tool chain cannot meet the comprehensive requirements of enterprise-level artificial intelligence projects for data security, process standardization and continuous delivery. SUMMARY

[0005] Therefore, the present application provides a pipeline-based artificial intelligence full-process automatic learning system to solve the problems of untraceable data version, lack of standardized training process, difficult to retain evaluation indicators, and non-transparent and difficult-to-rollback model release configuration in the prior art.

[0006] The embodiment of the application provides a pipeline-based artificial intelligence full-process automatic learning system, comprising: a pipeline arrangement unit, configured to associate metadata, synchronize states and schedule tasks of each functional unit in the same pipeline; a data management unit, configured to obtain training data from a heterogeneous data source, perform format standardization, preprocessing and versioned storage on the training data, and output a data version identifier corresponding to the training data to the pipeline arrangement unit; a training management unit, configured to perform multiple rounds of model training and parameter optimization on a computing resource cluster according to the data version identifier and a preset model configuration, generate a temporary model file, and synchronize an identifier of the temporary model file to the pipeline arrangement unit; a model evaluation and management unit, configured to extract a corresponding temporary model according to the identifier of the temporary model file, perform evaluation index calculation and index comparison on the temporary model, register the temporary model meeting a preset index threshold as a target model, and generate a target model version identifier; and a release management unit, configured to call a corresponding base image and environment variable to build a service package based on the target model version identifier, and push the service package to a preset DevOps image repository to trigger a deployment process.

[0007] The above at least one technical solution adopted by the embodiment of the application can achieve the following beneficial effects:

[0008] Through the pipeline arrangement unit, metadata of each functional unit in the same pipeline is associated, states are synchronized, and tasks are scheduled; the data management unit is configured to obtain training data from a heterogeneous data source, perform format standardization, preprocessing and versioned storage on the training data, and output a data version identifier corresponding to the training data to the pipeline arrangement unit; the training management unit is configured to perform multiple rounds of model training and parameter optimization on a computing resource cluster according to the data version identifier and a preset model configuration, generate a temporary model file, and synchronize an identifier of the temporary model file to the pipeline arrangement unit; the model evaluation and management unit is configured to extract a corresponding temporary model according to the identifier of the temporary model file, perform evaluation index calculation and index comparison on the temporary model, register the temporary model meeting a preset index threshold as a target model, and generate a target model version identifier; and the release management unit is configured to call a corresponding base image and environment variable to build a service package based on the target model version identifier, and push the service package to a preset DevOps image repository to trigger a deployment process. The application can realize data version traceability, training process standardization, evaluation index continuous retention, model release transparency and easy rollback. BRIEF DESCRIPTION OF DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0010] Figure 1 is the overall architecture schematic diagram of the artificial intelligence full-process automatic learning system based on pipeline involved in the actual scene of the embodiments of the present application;

[0011] Figure 2 is the overall structure schematic diagram of the artificial intelligence full-process automatic learning system based on pipeline provided by the embodiments of the present application. DETAILED DESCRIPTION

[0012] In the following description, specific details such as specific system structures, techniques, etc. are presented for the purpose of explanation, not for the purpose of limitation, so that the embodiments of the present application can be thoroughly understood. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0013] At present, algorithm development is widely used in various fields with the rapid development of artificial intelligence. For algorithm personnel and data analysis personnel, efficient development tools and scientific process management are crucial.

[0014] In the prior art, data of different algorithm scenes such as machine learning, NLP, search and recommendation, vision, etc. are usually stored in local or scattered platforms; algorithm development in different fields depends on different tools and platforms, training process lacks standardization, code and service management is scattered, and there is no unified model publishing process and evaluation result storage mechanism. Therefore, the prior art still has the following defects:

[0015] Scattered data management: knowledge inheritance is easy to interrupt, data set resources cannot be fully utilized, there are data security risks, and it is inconvenient to trace and manage.

[0016] Scattered development tools and platforms: algorithm and engineering team need to learn and use multiple systems, which increases development complexity and learning cost.

[0017] Non-standard training process: key personnel leaving easily leads to process interruption.

[0018] Lack of unified management: code and service management is scattered, knowledge inheritance is easy to interrupt, and service lacks continuous support.

[0019] Evaluation and release issues: lack of evaluation result retention, difficult to trace historical model effect;

[0020] Model release process is not unified, online process is not transparent.

[0021] Therefore, a one-stop development platform for algorithm personnel is needed to solve the above problems, which can provide a series of processes of data management, training management, model evaluation and release management based on pipeline automation, help users quickly create and release models, and manage the whole cycle of AI workflow.

[0022] The overall architecture of the pipeline-based artificial intelligence full-process automatic learning system provided by the present application in actual scenarios will be described below in conjunction with the accompanying drawings. Figure 1 The overall architecture of the pipeline-based artificial intelligence full-process automatic learning system provided by the present application in actual scenarios will be described below in conjunction with the accompanying drawings. Figure 1 As shown in FIG. 1, the overall architecture of the system can specifically include the following contents:

[0023] Data management module: provide various AI scene data for model training, covering data acquisition, verification, version management and storage functions. Through centralized management, solve the problems caused by data dispersion.

[0024] Training management module: create a pipeline as a workspace, and push the pipeline information to the devops platform to store the pipeline information. Adjust the data set and model parameters based on the pipeline, provide multi-round training task mode, and improve the training efficiency and model accuracy.

[0025] Model evaluation and management module: automatically record and evaluate the evaluation indexes of the model under different configurations, realize the standardized comparison of the model effect, and quickly determine the optimal model configuration.

[0026] Release management module: integrated with the devops system, supports one-key deployment of the model to the cloud, and unified management of online resources and versions.

[0027] Algorithm management module: in addition to the automatic learning process, provide management functions of evaluation indexes, code, training parameters, basic image, inference configuration, and can allocate training resource cluster / machine.

[0028] The system realizes the concept of "one-stop", that is, each link of AI development, including data management, model training, model evaluation, model release, and service management, can be completed on the AI capability platform. This makes AI development simpler and more convenient. To meet the needs of AI developers with different experiences, the application provides a convenient and easy-to-use use process. For example, business developers can use the automatic learning process to quickly release online services without worrying about models or codes; AI beginners can use pre-built algorithms to build online services without worrying about model development.

[0029] Finally, the system supports application to various AI algorithm scenarios such as image classification, object detection, text classification, search recommendation, prediction analysis, and face recognition, greatly broadening the possibility of automatic learning.

[0030] The system breaks through the limitations of algorithm development and management systems of computer vision platforms, search recommendation platforms, and machine learning platforms, and integrates them into a unified system platform. Referring to Huawei Cloud's ModelArts and Alibaba Cloud's artificial intelligence platform PAI, combined with the specific business status and needs of Longhu AI, the system provides customized management of the full life cycle of algorithm services according to the dimensions of the group's products and systems. It supports one-key deployment in a cloud environment and provides detailed user permission control to ensure system and data security.

[0031] The pipeline-based artificial intelligence full-process automatic learning system provided by the embodiments of the application will be described below in conjunction with the accompanying drawings and specific embodiments. Figure 2 is a schematic diagram of the overall structure of the pipeline-based artificial intelligence full-process automatic learning system provided by the embodiments of the application, as Figure 2 shown, the pipeline-based artificial intelligence full-process automatic learning system can specifically include the following contents:

[0032] The pipeline arrangement unit 201 is configured to associate metadata, synchronize states, and schedule tasks for each functional unit within the same pipeline.

[0033] The data management unit 202 is configured to obtain training data from a heterogeneous data source, perform format standardization, preprocessing, and versioned storage on the training data, and output a data version identifier corresponding to the training data to the pipeline arrangement unit.

[0034] The training management unit 203 is configured to perform multiple rounds of model training and parameter tuning on a computing resource cluster according to the data version identifier and a preset model configuration, generate a temporary model file, and synchronize the identifier of the temporary model file to the pipeline arrangement unit.

[0035] The model evaluation and management unit 204 is configured to extract a corresponding temporary model according to the identification of the temporary model file, perform evaluation index calculation and index comparison on the temporary model, register the temporary model satisfying a preset index threshold as a target model, and generate a target model version identification;

[0036] The release management unit 205 is configured to call a corresponding base image and environment variable construction service package based on the target model version identification, and push the service package to a preset DevOps image warehouse to trigger a deployment process.

[0037] In some embodiments, the pipeline arrangement unit is specifically configured to:

[0038] generate a pipeline task identification corresponding to the target algorithm scene, and register task metadata;

[0039] map the data version identification, the model configuration parameter and the computing resource information into a node dependency relationship based on the task metadata;

[0040] sequentially schedule the data management unit, the training management unit, the model evaluation and management unit and the release management unit according to the node dependency relationship, and automatically trigger the next node when the identification output by the previous node is reached;

[0041] continuously synchronize the node state to a preset metadata table during the node running, and perform a pause, retry or rollback operation according to a preset rule when an abnormal state is detected.

[0042] Specifically, the pipeline arrangement unit is deployed as an independent containerized service, which mainly includes a task identification generation module, a metadata persistence module, a dependency relationship analysis module, a scheduling execution module and a state monitoring module. The task identification generation module generates a globally unique pipeline task identification by calling a random number generator combined with the current timestamp when receiving a creation instruction, and immediately writes it into the metadata persistence module. The metadata persistence module is based on a relational database, uses the task identification as the primary key, and persistently stores fields such as task name, algorithm scene, creator, creation time, current state, etc. The dependency relationship analysis module parses the data version identification, model configuration parameter and computing resource information into a directed acyclic graph structure according to a preset node template, and assigns a unique node identification to each node. The scheduling execution module works based on an event-driven mechanism, and automatically triggers the execution of a node when it is detected that all the preceding nodes of the node are completed and the output identification has been written into the database. For the training node, the scheduling execution module calls the REST interface of the container orchestration platform, submits a job description file to the GPU or CPU resource pool and attaches the node identification. The state monitoring module periodically pulls the running log and container state of each node, writes the node state back to the metadata table, and sends a state event to the message queue for real-time display on the front-end visualization page.

[0043] Further, when the data management unit completes data preprocessing, it generates a data snapshot named with a SHA256 hash value in the object storage, and returns the hash value as a data version identifier to the pipeline orchestration unit through an HTTP POST request. The orchestration unit finds the corresponding node record in the database, updates the node state to SUCCESS, and writes the output identifier field. The dependency relationship analysis module has recorded in advance that the training node depends on the data node after the pipeline is created. Therefore, the state monitoring module immediately sends a trigger event to the scheduling execution module after detecting that the data node is successful. The scheduling execution module then issues a start instruction to the training management unit, which includes the data version identifier and the model configuration parameters. The training node generates a temporary model file in the distributed training cluster, and uploads the temporary model file identifier through a callback interface after training is completed.

[0044] Further, the state monitoring module has an exception detection sub-thread. If the node heartbeat update is not received within the preset timeout time, the node state is set to ERROR, and the error reason field is written. When the pipeline orchestration unit detects that a node state is ERROR, it first determines whether the node is allowed to retry according to the node type. If the node is allowed to retry, the scheduling execution module resubmits the job according to the exponential backoff strategy. If the number of retries has reached the upper limit, the entire pipeline state is set to FAILED, and the upper-level business system is notified. For the prequel nodes that have been successfully completed, the output identifier has been persisted, and can be used as a rollback baseline to restart the pipeline after manual confirmation, avoiding repeated calculation.

[0045] In the image classification scenario, the data version identifier registered by the data management unit is IMGCLS20250523001, the model configuration parameter points to the ResNet50 baseline model, and the computing resource requirement is 4 GPUs per node. After receiving the creation instruction, the pipeline orchestration unit generates the pipeline identifier PIPE230523001 and registers the metadata. The dependency relationship analysis module automatically generates a dependency relationship graph including the data node, the training node, the evaluation node, and the release node. The data node triggers the training node after completion, and the training node passes the temporary model file identifier MODEL230523A to the evaluation node after completion. The evaluation node generates the target model version identifier VER230523A after the evaluation indicator reaches the preset threshold, and outputs it to the release node. The release node builds a service package and successfully pushes it to the DevOps image repository, and the pipeline state is finally updated to SUCCESS.

[0046] In the natural language processing task, the training node needs to use the multi-machine multi-card strategy. When constructing the dependency relationship, the pipeline arrangement unit marks the training node as needing cross-node resources. The scheduling execution module submits a job description file with nodeSelector and hostNetwork parameters to the container orchestration platform to ensure that the training job is allocated to a GPU node that supports RDMA network. After the release node completes the service package construction, it first deploys two instances for gray release. The state monitoring module continuously collects the call delay. When the gray instance runs stably for 1 hour and the delay index is lower than the threshold, the release node is expanded to the full instance, realizing smooth switching.

[0047] Through the above implementation, the pipeline arrangement unit realizes automatic task arrangement, node scheduling and exception handling for multi-algorithm scenarios. The whole process data and model products are passed through the unique identifier throughout each unit, ensuring closed-loop management from artificial intelligence model development to deployment.

[0048] In some embodiments, the data management unit is specifically configured to:

[0049] Obtain training data from a heterogeneous data source and perform format standardization and field mapping based on a preset field template;

[0050] Perform preprocessing on the standardized training data, write the preprocessed training data to object storage and a database, and generate a data version identifier based on a hash fingerprint and a timestamp;

[0051] Register the data version identifier in a metadata table and provide a data pulling interface to the training management unit;

[0052] Perform integrity checking and distribution statistics on the stored training data, and write the checking and statistical results to the metadata table for subsequent process calling.

[0053] Specifically, the present embodiment gives a complete implementation of the data management unit. The data management unit is deployed as an independent micro-service cluster, which is internally divided into a data source access module, a field mapping module, a preprocessing pipeline module, a version generation and storage module, a quality checking module, and a metadata service module. Each module is decoupled through a message queue.

[0054] In actual operation, the business personnel first selects a target algorithm scene in the data management page and uploads or registers a data source address. The data source access module calls the corresponding connector according to the configuration file, such as an object storage read-write interface, a relational database driver or a file transfer protocol client, pulls the original data file from the heterogeneous source and writes it into the cluster temporary cache. After the pulling is completed, the field mapping module maps the original field to an intermediate format containing a data identifier, a feature field, a label field and a timestamp field according to the preset field template for the algorithm scene. For example, in the image classification scene, the field template maps the original file path to the feature field, maps the directory name to the label field, and appends a unique file fingerprint as the data identifier.

[0055] Further, after the field mapping is completed, the preprocessing pipeline module starts a multi-stage pipeline job. The first stage performs hash deduplication on the mapped file, and removes duplicate records by comparing the SHA256 value; the second stage detects missing values and format errors, and if any abnormality is found, immediately writes an error log and marks the record as invalid; the third stage performs desensitization processing according to the algorithm scene configuration, for example, regular replacement is performed on the text field containing personal information; the fourth stage randomly splits the data according to the training, validation and test ratio, and labels each record with a data shard label. The preprocessed data file is written into the object storage bucket aisys tem-dataset, and the structured description information is written into the database table dataset_item, which contains the file path, label, split label and verification fingerprint.

[0056] Further, the version generation and storage module receives the preprocessing completion event, traverses all file paths in the current batch, concatenates the SHA256 fingerprint in lexicographic order and calculates the overall fingerprint again, then concatenates the UTC timestamp to generate the data version identifier IMGCLS20250523T103045Z_6AF3E21. The generated version identifier is written into the database table dataset_version together with the fingerprint digest, the number of files and the object storage prefix, and is sent to the metadata service module through the message bus.

[0057] Further, the quality checking module asynchronously consumes the message, randomly samples a part of the records to recalculate the fingerprint, checks the total amount of samples and the label distribution, calculates statistical indicators such as pixel mean, variance or text length distribution, and writes the checking result and statistical information into the table dataset_quality. When any indicator exceeds the preset threshold, the module writes an abnormal flag to the metadata service module, and the subsequent pipeline scheduling unit can determine whether to terminate the process or enter the manual review branch when reading the metadata.

[0058] Further, the metadata service module exposes a REST interface / datasets / {versionId} / manifest to the outside. The training management unit only needs to carry the data version identifier to call the above interface when creating a training task, and can obtain a manifest file containing the object storage file path, access token and split label. The metadata service module also pushes the data version identifier and data quality status to the pipeline orchestration unit, ensuring that subsequent nodes can perform training, evaluation and release operations based on consistent data sources.

[0059] Through the above implementation, the data management unit completes the full-link operation from heterogeneous data source acquisition, field standardization, preprocessing, version generation, quality checking to metadata release, providing a unified, traceable and quality controllable input data basis for the training management unit.

[0060] In some embodiments, the training management unit is specifically used for:

[0061] According to the data version identifier, the corresponding data set is pulled, and the model structure and hyperparameters are loaded according to the preset model configuration;

[0062] Starting a distributed training job on a computing resource cluster, performing multiple rounds of iteration and updating of model weights, and dynamically adjusting hyperparameters during training;

[0063] Generating a temporary model file snapshot at the end of each round of training, and recording the identifier of the temporary model file to the metadata table;

[0064] After completing the preset training rounds, the latest temporary model file identifier is synchronized to the pipeline orchestration unit for subsequent model evaluation and management unit calling.

[0065] Specifically, the internal structure, operation steps and key technical points of the training management unit are described in detail to explain how it realizes the standardized and traceable management of model training under the pipeline framework.

[0066] The training management unit is encapsulated as a horizontally scalable micro-service cluster, and the core components include a task access module, a data pulling module, a model loading module, a job scheduling module, a hyperparameter optimization module, a snapshot generation module and a metadata synchronization module. The modules interact with each other in an event bus manner, ensuring decoupling in high-concurrency scenarios.

[0067] When the pipeline orchestration unit pushes the data version identifier IMGCLS20250523T103045Z_6AF3E21, the task access module first parses the model configuration parameters and resource specifications passed along and generates a unique training task identifier TRAIN230523A internally. The data pulling module then calls the manifest interface exposed by the data management unit to pull the training, validation, and test shards in batches. To ensure data consistency, the data pulling process recalculates the SHA256 fingerprint of the pulled file locally on the computing node and compares it with the manifest. After verification, it can proceed to the next stage.

[0068] The model loading module instantiates the ResNet50 baseline structure according to the preset model configuration and reads the hyperparameter file passed with the task, which contains items such as learning rate, batch size, optimizer type, and regularization weight. If an item in the hyperparameter file is set to AUTO, it is adjusted dynamically by the hyperparameter tuning module during training. For example, the hyperparameter tuning module can automatically reduce the learning rate at the end of the 3rd iteration based on the validation set accuracy and write the adjustment record to the tuning log.

[0069] The job scheduling module submits the training job to the cluster scheduler according to the resource specification description, which specifies the number of nodes, the number of GPU cards, the network interconnection requirements, and the data mounting path. When using the full data parallel strategy, the job scheduling module starts a gradient synchronizer on each training node to update the model weights in the full AllReduce manner. For large-scale models, the hybrid precision and gradient accumulation flags can be enabled in the description, and the scheduler will pull up the GPU nodes that support Tensor core instructions accordingly. During training, the logs and indicators of each node are collected in real time by the log agent and output to the pipeline console. Users can view the training progress and GPU utilization on the pipeline-based training process page.

[0070] At the end of each training round, the snapshot generation module serializes the current weight file and writes it to the object storage bucket aisystem-model, with the file naming format SNAPSHOT_EPOCHxx_TASKTRAIN230523A. Then it generates the snapshot identifier SNAP_2305231502_xx, which is written to the table model_snapshot along with the training task identifier, data version identifier, current hyperparameter settings, and timestamp. If periodic evaluation is enabled, the snapshot generation module passes the snapshot identifier to the evaluation trigger, which starts a lightweight inference job in the background to quickly evaluate the validation set and return the core indicators for display on the monitoring panel.

[0071] When the number of iterations reaches the preset value or the early stop condition is triggered, the metadata synchronization module writes the latest snapshot identifier SNAP_2305231502_final into the table model_task_state, and sends the identifier and the training task state SUCCESS to the pipeline scheduling unit through the message queue. The pipeline scheduling unit triggers the model evaluation and management unit to perform comprehensive evaluation accordingly. The key metadata in the entire process, including the data version identifier, model configuration parameters, resource specifications, hyperparameter tuning records, and snapshot identifier, are persisted in a foreign key association manner to ensure that the training environment can be accurately reproduced for subsequent audits or experimental reproduction.

[0072] If the job scheduling module detects node downtime or GPU overflow during training, the state callback interface will immediately set the node state to FAILED. After receiving the failure event, the metadata synchronization module updates the task state to ERROR and writes the failure reason into the table model_task_state. According to the pre-prepared retry strategy, the pipeline scheduling unit allows the training management unit to automatically restart the job up to two times; if it still cannot be recovered, it terminates the pipeline and notifies the operation and maintenance personnel. After the training job ends, whether it is successful or not, the job scheduling module will trigger a resource recycling task to unload the mounted volume and release the GPU nodes, avoiding resource leakage.

[0073] Through the above implementation, the training management unit can complete the complete closed loop from data set pulling, model instantiation, distributed training, hyperparameter tuning to snapshot generation and metadata synchronization under a unified pipeline framework, ensuring the efficiency and traceability of the model development process, and providing consistent and reliable model products for subsequent evaluation, release, and other links.

[0074] In some embodiments, the model evaluation and management unit is specifically configured to:

[0075] extract the corresponding temporary model according to the temporary model file identifier, and call a preset evaluation data set to run an inference process to generate evaluation indicators;

[0076] compare the evaluation indicators with a preset threshold or historical benchmark value to determine whether the temporary model meets the registration condition;

[0077] When the registration condition is met, the temporary model is written into the model registration library, a target model version identifier is generated, and the evaluation indicators and the target model version identifier are stored in association in the model registration library;

[0078] synchronize the target model version identifier to the pipeline scheduling unit for subsequent calling by the release management unit.

[0079] Specifically, the model evaluation and management unit is composed of a model extraction module, an inference execution module, an index calculation module, an index comparison module, a model registration module, and a version synchronization module. Each module communicates information through an internal message bus, and externally communicates with the training management unit and the pipeline arrangement unit through a REST interface. The overall implementation process is as follows:

[0080] After completing the specified round of training, the training management unit pushes the temporary model file identifier SNAP_2305231502_final to the model extraction module.

[0081] The model extraction module calls the object storage API to pull the corresponding weight file, and loads the model structure and weight locally.

[0082] The inference execution module calls the preset evaluation dataset IMGCLS_VAL202505 (stored in the object storage bucket aisystem-eval), performs forward inference on each sample, and writes the output to a ring buffer.

[0083] The index calculation module reads the buffer content in real time, calculates the Accuracy, Precision, Recall, F1, and average inference latency according to the scene configuration, and packs the results into a JSON object.

[0084] The index comparison module compares the core indicators in the JSON object with the threshold table THRESHOLD_IMGCLS and the historical best indicator record BEST_IMGCLS. If Accuracy ≥ 0.92 and F1 is not lower than 98% of the historical best value, a registration pass signal is returned.

[0085] After receiving the pass signal, the model registration module writes the model weight file, dependency configuration, and evaluation index to the model registration library, and generates a unique target model version identifier VER_IMGCLS20250523_01.

[0086] The version synchronization module writes VER_IMGCLS20250523_01 and the evaluation index to the table model_version, and synchronizes the target model version identifier to the pipeline arrangement unit through the event bus, for the release management unit to pull.

[0087] Further, the index calculation module adopts a streaming processing method, and can calculate part of the indexes in real time after the forward inference of the inference execution module is completed, thereby improving the overall efficiency. For the image classification scene, the module updates the confusion matrix and calculates the stage accuracy and F1 after every one hundred images are inferred, and the stage results are pushed to the model evaluation result display page through WebSocket for real-time observation by algorithm engineers. The index comparison module maintains a threshold table THRESHOLD IMGCLS, and the threshold table defines the minimum qualified value of each index; at the same time, the historical best index and the corresponding version identifier are stored in the table BEST IMGCLS. When the current model is better than the threshold on the core index and is not lower than 98% of the historical best, it is considered to meet the registration condition.

[0088] Further, the model registration module adopts the following rules when generating the target model version identifier: splicing the scene abbreviation, the registration date, and the four-bit incremental number to ensure that multiple registrations in the same date do not conflict. The target model version identifier, the weight file path, the dependent image identifier, the evaluation index JSON, and the creation time are written into the table model_registry. Subsequently, the object storage lifecycle management strategy is used to automatically set a thirty-day cold data archiving strategy for VER IMGCLS20250523_01, thereby reducing the storage cost.

[0089] Further, the version synchronization module sends a POST request to the pipeline arrangement unit, and the payload contains the target model version identifier, the weight file hash, the evaluation index summary, and the registration time. After receiving the request, the pipeline arrangement unit writes the information into its global metadata table, and triggers the release management unit to start the service package construction process. If the algorithm researcher needs to compare the historical model, he / she can input any target model version identifier on the image (model) management page for retrieval, and the system returns the evaluation index, the weight file path, and the dependent image information according to the model_registry table, thereby realizing fast backtracking and reproducing the experimental environment.

[0090] Further, if the inference time is abnormal (the single batch time consumption exceeds twice the upper threshold) or the index calculation module detects Nan during the inference execution process, the inference execution module will abort the current evaluation and write an error log. After receiving the error event, the index comparison module writes the evaluation state as FAILED, and notifies the pipeline arrangement unit. According to the retry strategy, the pipeline arrangement unit allows the model evaluation and management unit to restart the evaluation task once; if it still fails, the temporary model is marked as invalid, and the training management unit is requested to provide the next round of training results.

[0091] Through the above embodiments, the model evaluation and management unit realizes rapid evaluation, accurate comparison and reliable version registration of temporary models, provides standardized and traceable target model version identifiers for the release management unit, and thus guarantees the controllable quality and consistent process of models before going online.

[0092] In some embodiments, the release management unit is specifically used for:

[0093] extracting a target model and its dependent configurations based on the target model version identifier;

[0094] calling a base image and environment variables corresponding to the target model to construct a service package containing an inference script and a model weight file;

[0095] writing a version number, a check value and deployment metadata for the service package, and generating an image tag corresponding to the target model version identifier;

[0096] pushing the service package to a preset DevOps image repository, and triggering a continuous delivery pipeline through an interface to deploy to a computing resource cluster;

[0097] registering a mapping relationship between the service package and the target model version identifier in the image repository for subsequent rollback or upgrade calls.

[0098] Specifically, when the pipeline orchestration unit pushes the target model version identifier VER_IMGCLS20250523_01, the model retrieval module of the release management unit obtains the model weight path oss: / / aisystem-model / VER_IMGCLS20250523_01 / model.pt, the dependent image identifier BASE_PYTORCH1.14_CUDA11.8, and the inference script path git: / / algo-repo / imgcls / infer.py by querying the table model_registry. The retrieval results are handed over to the image construction module together with the environment variable list env.yaml.

[0099] Further, the image construction module pulls the corresponding OCI compatible base image according to the base image identifier, and copies the model weight file, the inference script and the environment variable list in the construction context. The construction process calculates the SHA256 fingerprint of the model weight file and writes it into the list manifest.json. The image construction module generates the service package IMAGE_TAR_VER_IMGCLS20250523_01 after completing the layer caching locally, and outputs the container image tag ai / imgcls:VER_IMGCLS20250523_01. The overall image digest sha256:DEF0A1B2C3 is calculated and written into the OCI tag io.ai.digest as soon as the construction is completed.

[0100] In some examples, the tag generation module writes the following metadata for the service package:

[0101] io.ai.version = VER IMGCLS20250523_01

[0102] io.ai.base = BASE PYTORCH1.14_CUDA11.8

[0103] io.ai.created = 2025-05-23T15:36:45Z

[0104] io.ai.digest = sha256:DEF0A1B2C3

[0105] At the same time, the module inserts a record into the image management database image_meta, with fields including model version identifier, image tag, image digest, creation time, and status INIT.

[0106] Further, the repository pushing module calls the OCI Registry API of the internal network to push ai / imgcls:VER IMGCLS20250523_01 to devops.registry.internal:5000. After the pushing is successful, the continuous delivery triggering module submits a JSON payload to the enterprise DevOps platform interface POST / pipelines / deploy, with the image tag, target cluster inference-prod, and deployment template ID HELM IMGCLS STD specified in the payload. The DevOps pipeline then executes four stages of image security scanning, image pulling, Helm rendering, and Kubernetes deployment. The deployment template is preset with a replica number of 2 and an available readiness probe, and the probe script calls the / model / health endpoint to verify the availability of the inference service.

[0107] Further, when the DevOps pipeline returns a status of SUCCEEDED, the mapping registration module writes the image tag, deployment environment, and deployment time into the table image_model_map, and updates the record status to ACTIVE, and the status of the old version VER IMGCLS20250507_02 is changed from ACTIVE to STANDBY. If the subsequent monitoring system detects that the average response delay of the new version exceeds the threshold, the pipeline orchestration unit can refer to the mapping table to roll back the image tag ai / imgcls:VER IMGCLS20250507_02 to the current production version, and the release management unit triggers a rapid rollback process through the same continuous delivery interface to achieve seamless switching.

[0108] Further, in the image pushing or continuous delivery process, if a network timeout or security scan failure occurs, the repository pushing module or the continuous delivery triggering module sets the image_meta record state to FAILED and writes the failure reason field. After receiving the FAILED event, the pipeline orchestration unit automatically re-triggers the release process once according to the retry strategy; if the release process still fails, the pipeline is terminated and an alarm information is generated to notify the operation and maintenance person in charge.

[0109] Through the above implementation, the release management unit completes the closed-loop operation of the target model, including dependency pulling, service package building, metadata injection, image pushing, continuous delivery triggering, and mapping table registration, thereby providing a standardized, traceable, and easy-to-rollback delivery path for model online.

[0110] In some embodiments, the system further comprises:

[0111] An algorithm management unit is configured to maintain an index relationship among the base image, model code, evaluation script, and resource configuration file, and provide the index relationship to the training management unit, model evaluation and management unit, and release management unit.

[0112] Specifically, the algorithm management unit is deployed as an independent microservice, and the core modules include an algorithm registration module, an image index module, a code repository mapping module, an evaluation script repository mapping module, a resource template repository, a version control module, and an external interface module. The modules share the same meta database algorithm_meta, and implement state notification through an event message bus.

[0113] When a researcher selects “New Algorithm” on the algorithm management page, the algorithm registration module pops up a form, requiring the researcher to fill in the algorithm name, applicable scenario, base image identifier, main code repository address, default evaluation script identifier, and resource configuration template. After submission, the algorithm registration module generates a unique algorithm identifier ALG_NLP_BERTCLS for the algorithm, and records the algorithm identifier, version number, image identifier, code repository URL, evaluation script URL, resource template ID, creator, creation time, and state INIT in the algorithm_meta table. At this time, the image index module, code repository mapping module, and evaluation script mapping module each write mapping records to their sub-tables, realizing multi-dimensional indexing.

[0114] The image index module saves the mapping between the algorithm identifier and the base image identifier in the table algo_image_map, and periodically checks the image digest to prevent the image from being tampered with.

[0115] The code repository mapping module records the main branch commit hash for ALG_NLP_BERTCLS. Subsequently, when the code repository triggers a new commit event, the version control module will listen to the Webhook and automatically generate a new algorithm version ALG_NLP_BERTCLS_v002.

[0116] The evaluation script mapping module maintains a table algo_eval_map, which records the evaluation script URL and its verification fingerprint. If the script is updated, it will be synchronized with the version control module to generate a new script version and associate it with the algorithm version.

[0117] The resource template repository saves YAML format resource description files, including CPU, GPU, memory, disk, and network bandwidth quotas. The template ID selected during algorithm registration is associated with the algorithm identifier in a one-to-many relationship in the table algo_resource_map. The algorithm management unit checks the requester's permission level when training or evaluation is called. Only users with access to the template can successfully schedule the corresponding computing resources.

[0118] In the training phase, the training management unit queries the base image identifier and resource template ID from the algorithm management unit through the REST interface / algo / {algorithm identifier} / metadata when creating a task. The algorithm management unit returns the image identifier BASE_PYTORCH2.1_CUDA12.1 and the resource template ID GPU_X4_48G based on the algorithm_meta table, and includes the code repository commit hash in the return body.

[0119] In the evaluation phase, the model evaluation and management unit obtains the evaluation script URL and verification fingerprint through the interface / algo / {algorithm identifier} / eval for offline evaluation by pulling the script.

[0120] In the release phase, the release management unit uses the interface / algo / {algorithm identifier} / image to query the image digest when building the service package to prevent the use of outdated base images. At the same time, it obtains the main code repository address through / algo / {algorithm identifier} / code to facilitate automatic generation of code reference information in the inference image.

[0121] In some examples, the version control module generates a Version Histo ry entry for each algorithm version change event, and adds a version number field version in the algorithm meta table, with an initial value of v001; when any of the image upgrade, code change or evaluation script update occurs, the version is automatically incremented and the trigger reason is recorded. Users can expand the "history version" list on the algorithm management page, select any version ALG_NLP_BERTCLS_v002, and view the corresponding image summary, commit hash, script fingerprint and commit description to achieve quick backtracking.

[0122] Further, the algorithm management unit is provided with a timing verification task, which pulls the image summary, code repository HEAD hash and script fingerprint of all registered algorithms every 24 hours, and compares them with the archive fingerprint. If a difference is found, an alarm event ALG_DIFF_DETECTED is sent to the pipeline arrangement unit, and the algorithm state is marked as INCONSISTENT in the algorithm meta table. The pipeline arrangement unit accordingly prevents the use of algorithm versions with consistency risks from entering subsequent training or release links.

[0123] Through the above implementation, the algorithm management unit realizes accurate indexing and version linkage between the base image, model code, evaluation script and resource template, provides unified, reliable and traceable algorithm metadata for training, evaluation and release links, and further enhances the complete life cycle management capability of the system in enterprise-level artificial intelligence development.

[0124] In some embodiments, the algorithm management unit is specifically used for:

[0125] Establishing an algorithm metadata directory, writing algorithm identification, applicable scenarios, creator information and version number into an index table;

[0126] Recording the mapping relationship between the base image identification, model code repository address, evaluation script identification and resource configuration template for each algorithm, and maintaining the mapping relationship in the index table;

[0127] When the training management unit initiates an algorithm call request, return the corresponding base image identification and resource configuration template;

[0128] When the model evaluation and management unit initiates an algorithm call request, return the corresponding evaluation script identification;

[0129] When the release management unit initiates an algorithm call request, return the corresponding base image identification and model code repository address;

[0130] When a new version of the base image, model code, or evaluation script is detected, a version snapshot of the algorithm is generated and the index table is updated.

[0131] Specifically, the embodiment describes the implementation of the algorithm management unit in the enterprise-level integrated artificial intelligence automatic learning system. This unit is responsible for the registration, mapping, querying, and version snapshot generation of algorithm assets, providing unified metadata support for the training, evaluation, and release stages.

[0132] The algorithm management unit first establishes the algorithm metadata directory through the "algorithm registration" function. After the researcher fills in the algorithm name, applicable scenarios, creator, algorithm description, and other items on the algorithm management page, the system generates a unique algorithm identifier ALG_20250523_BERTCLS for the algorithm and writes the fields: algorithm identifier, applicable scenarios, creator, creation time, and initial version number v1 in the index table algo_index. The index table takes the algorithm identifier as the primary key, ensuring that any subsequent versions can be traced based on the same primary key.

[0133] Further, after registration is complete, the algorithm management unit automatically enters the mapping relationship configuration wizard, guiding the user to bind resources item by item. The specific implementation process is as follows:

[0134] Base image identifier: Select BASE_PYTORCH2.1_CUDA12.1 in the image repository dropdown list, and the image fingerprint value is also written into the table algo_image_map.

[0135] Model code repository address: Fill in git: / / repo.ai / bertcls.git in the input box; the system subscribes to the Push event of the repository through Webhook, writes the repository URL and the current HEAD commit hash into the table algo_code_map.

[0136] Evaluation script identifier: Select EVAL_BERTCLS_F1 from the evaluation script repository and write it into the table algo_eval_map.

[0137] Resource configuration template: Select the GPU_X4_48G template, and the template ID is associated with the algorithm identifier and stored in the table algo_resource_map.

[0138] Further, the algorithm management unit exposes REST interfaces to provide metadata on demand for each business unit:

[0139] The training management unit calls GET / algo / ALG_20250523_BERTCLS / train when creating a training task, and the algorithm management unit returns the base image identifier BASE PYTORCH2.1_CUDA12.1 and the resource template GPU_X4_48G.

[0140] The model evaluation and management unit calls GET / algo / ALG_20250523_BERTCLS / eval before performing offline evaluation, and the system returns the evaluation script identifier EVAL_BERTCLS_F1 and the script fingerprint.

[0141] The release management unit calls GET / algo / ALG_20250523_BERTCLS / deploy when building an inference service package, and the system returns the base image identifier and the model code repository address git: / / repo.ai / bertcls.git for the image building stage to pull the code.

[0142] Further, the algorithm management unit is configured with an event listener that listens to version change events of the image repository, code repository, and script repository, for example:

[0143] When the base image BASE PYTORCH2.1_CUDA12.1 generates a new Digest in the image repository, the image repository pushes an event to the algorithm management unit. The version control module generates a new snapshot ALG_20250523_BERTCLS_v2 for the algorithm, records the change type IMG_UPDATE, and updates the version number to v2 in the algo_index table.

[0144] When the code repository has a Push event, the algorithm management unit records the latest commit hash and determines whether it crosses the preset importance threshold. If it is a trunk version update, a new version ALG_20250523_BERTCLS_v3 is generated and the change reason CODE_UPDATE is written.

[0145] When the evaluation script repository is updated, a script snapshot is generated and the algorithm version is incremented.

[0146] The version snapshot table algo_version records the snapshot ID, parent version, change type, and generation time, realizing a complete version tree. Developers can select any version in the "Version History" panel of the algorithm management page and view its image Digest, code commit hash, and script fingerprint, realizing quick rollback.

[0147] To ensure that the algorithm metadata is consistent with the actual resources, the algorithm management unit is provided with a daily timing verification task. The task traverses all algorithm versions in the index table, re-pulls the corresponding image Digest, code commit hash and script fingerprint, and compares them with the archive fingerprint. If there is inconsistency, the system will identify the "fingerprint drift" in red on the algorithm details page, and send an alarm ALG_INCONSISTENT to the pipeline arrangement unit to prevent the use of the affected algorithm version from entering the new training or release process.

[0148] Through the above method of the embodiment, the algorithm management unit not only maps the multi-dimensional resources of the basic image, model code, evaluation script and resource template in the database for accurate indexing, but also maintains high synchronization between resource versions and algorithm versions through an event-driven snapshot mechanism, providing authoritative and consistent metadata for each stage of training, evaluation and release, and ensuring the traceability and stability of enterprise-level artificial intelligence pipelines.

[0149] In some embodiments, the system further comprises:

[0150] A service call log unit is configured to collect call data of the deployed service instance, write the call data to a log storage, and feed back a log identifier to the pipeline arrangement unit.

[0151] Specifically, the following embodiments will illustrate the implementation of the service call log unit in collecting, centrally storing and feeding back logs of the inference service after model deployment, to ensure the traceability of the online running state and the ability to quickly locate faults.

[0152] The service call log unit is composed of a sidecar collection module, a log aggregation module, a log storage module, an identifier generation module and an interface gateway module, and all modules are connected through an internal message queue. The sidecar collection module is injected into the inference container in a shared network namespace manner, without the need to modify the business image to complete log collection.

[0153] Further, when the inference service instance starts, the sidecar collection module parses the traffic in the same Pod, and records the request method, interface path, call timestamp, response code, response duration and GPU memory occupation for each HTTP or gRPC call. To reduce the impact on business delay, the collection module encapsulates a single log into a binary message with a length of less than 1KB, and sends it to the log aggregation module through a local ring buffer asynchronously.

[0154] Further, the log aggregation module is deployed as a cluster DaemonSet, establishing a high-throughput pipeline with the node local file system. When the sidecar collection module pushes a batch of messages, the aggregation module first calculates the CRC32 fingerprint of each log within a one-second time window, and de-duplicates the logs with the same fingerprint to prevent redundancy caused by retry requests. The de-duplicated log stream is written to the message queue KafkaTopicInferenceLog for consumption by the downstream storage module.

[0155] Further, the log storage module is composed of a real-time storage submodule and a periodic archiving submodule. The real-time storage submodule consumes KafkaTopicInferenceLog, writes the logs to the columnar database ClickHouse cluster, and creates partitions by date and service instance dimension for fast query. The periodic archiving submodule compresses the cold logs of the previous day at zero o'clock every day and uploads the compressed files to the object storage bucket aisystem-log-archive, setting a lifecycle policy to automatically delete the files after 180 days.

[0156] Further, the log storage module is composed of a real-time storage submodule and a periodic archiving submodule. The real-time storage submodule consumes KafkaTopicInferenceLog, writes the logs to the columnar database ClickHouse cluster, and creates partitions by date and service instance dimension for fast query. The periodic archiving submodule compresses the cold logs of the previous day at zero o'clock every day and uploads the compressed files to the object storage bucket aisystem-log-archive, setting a lifecycle policy to automatically delete the files after 180 days.

[0157] Further, the interface gateway module provides a REST query interface / logs / {log identifier} externally, and the pipeline orchestration unit or operation and maintenance personnel can pull all call details or aggregated indicators within the window according to the log identifier at any time. If the log storage module detects that the error rate or latency indicator exceeds the threshold in real-time calculation, it will immediately call the interface gateway module to push the event ALERT_INFERENCE_LATENCY_HIGH to the alarm center and update the abnormal flag field in the log_manifest table at the same time.

[0158] Further, to prevent log loss caused by network disconnection, the sidecar collection module configures a 128MB sequential write buffer on the local disk. When the network is restored, the unsent logs are retried for pushing in the original order. The log aggregation module detects Kafka availability through the preStop hook before node restart, and if Kafka is unreachable, it writes the unsent logs to the node persistent volume safely, and automatically supplements the transmission after the node is back online, ensuring data integrity.

[0159] Through the above embodiments, the service call log unit realizes closed-loop management from collection, aggregation, deduplication, storage, identification generation to exception alarm, provides accurate online running data for the pipeline arrangement unit, and ensures visual monitoring and rapid rollback capability after model deployment.

[0160] In some embodiments, the service call log unit is specifically used for:

[0161] Structurally recording the request timestamp, request parameter fingerprint, response duration, and resource occupation information of the service instance;

[0162] Aggregating the structured records according to a preset time window, generating a call statistical entry, and writing it to a log storage;

[0163] Generating a log identification for each call statistical entry and synchronizing the log identification to the pipeline arrangement unit;

[0164] When receiving a query instruction from the pipeline arrangement unit, providing a retrieval interface based on the service instance identification or the log identification.

[0165] Specifically, the embodiments gradually explain the implementation of the service call log unit in log collection, aggregation, identification generation, and query interaction after the model goes online, to meet the needs of enterprise-level inference services for performance monitoring and traceable operation and maintenance.

[0166] The service call log unit is deployed as a distributed microservice, composed of a request collection module, a time window aggregation module, a log storage module, an identification generation module, and a query gateway module, and interconnected through internal event flow channels. The request collection module is injected in the sidecar mode with the inference container, ensuring that log collection can be completed without modifying the business image.

[0167] Further, the request collection module intercepts inbound and outbound calls of the inference container, and immediately generates a structured record after the call returns, with fields including: service instance identification; request timestamp (millisecond-level UTC); request parameter fingerprint (SHA256 value calculated for the request message); response duration (milliseconds); GPU memory occupation, CPU usage, and memory occupation ratio. Each record is serialized by JSON and written to a local buffer queue, and is sent to the time window aggregation module in an asynchronous manner.

[0168] Further, the time window aggregation module aggregates the structured records in fixed one-minute sliding windows. When the window is closed, the module calculates the average response time, P99 response time, success rate, and resource peak of the records of the same service instance, and generates a call statistics entry. The core fields of the statistics entry include: service instance identifier, window start and end time, aggregation indicators, and original record count. To reduce storage overhead, the original records are transferred to cold storage according to the life cycle strategy after the successful generation of the statistics entry.

[0169] Further, the identification generation module generates a unique log identification for each call statistics entry, with the rule: LOG_ <date> <hour> <last four digits of instance> <window serial number>, such as LOG_202505231630_BC12_W12. After generation, the identification and its summary information are written to the table log_index, and a POST request is sent to the pipeline scheduling unit through the REST interface, with the fields including: pipeline task identifier, service instance identifier, log identification, and window start and end time. The pipeline scheduling unit associates the log identification to the current deployment version accordingly, realizing the coherent tracing of the running state and the development state.

[0170] Further, the log storage module uses the columnar database ClickHouse to quickly write the call statistics entry in the table log_stats with (date, service instance) as the partition key; and establishes a foreign key mapping between the log_index table and ClickHouse. The query gateway module provides the GET / logs!instanceId= or GET / logs / {log identification} interfaces, for example:

[0171] If the service instance identifier and time range are specified, the gateway retrieves the corresponding partition in the log_stats table and returns the aggregation indicator sequence;

[0172] If the log identification is specified, the gateway first locates the window information in the log_index table, and then returns the single call statistics entry and its summary indicators.

[0173] The query result can be pulled by the pipeline scheduling unit and displayed in real time on the operation and maintenance panel.

[0174] In some examples, if the time window aggregation module detects that the network is unavailable when writing to ClickHouse, the call statistics entry is temporarily stored in the local disk queue; the system sets the maximum retry duration to 15 minutes, and if it still fails, an ERROR_LOG_WRITE event is sent to the pipeline scheduling unit, and the failure reason is recorded in the log_index table, ensuring that the fault is visible.

[0175] Through the above implementation, the service call log unit realizes fine-grained collection, windowed aggregation and log identification management of request-level indicators, provides unified, low-latency running state monitoring data and query interfaces for the pipeline arrangement unit, and meets the performance insight and abnormal source tracing requirements of large-scale inference services after going online.

[0176] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A pipeline-based artificial intelligence full-process automatic learning system, characterized in that, The pipeline arrangement unit is configured to: generate a pipeline task identifier corresponding to a target algorithm scenario, and register task metadata; map the data version identifier, model configuration parameters, and computing resource information into a node dependency relationship based on the task metadata; sequentially schedule the data management unit, the training management unit, the model evaluation and management unit, and the release management unit according to the node dependency relationship, and automatically trigger a next node when an identifier output by a previous node is received; continuously synchronize node states to a preset metadata table during node operation, and perform a pause, retry, or rollback operation according to a preset rule when an abnormal state is detected. The data management unit is configured to:

2. The system of claim 1, wherein, obtain training data from a heterogeneous data source, and perform format standardization and field mapping based on a preset field template; perform preprocessing on the standardized training data, write the preprocessed training data into object storage and a database, and generate a data version identifier based on a hash fingerprint and a timestamp; register the data version identifier in a metadata table, and provide a data pulling interface to the training management unit; perform integrity checking and distribution statistics on the stored training data, and write the checking and statistical results into the metadata table for subsequent process calling. The training management unit is configured to:

3. The system of claim 1, wherein, pull a corresponding data set according to the data version identifier, and load a model structure and hyperparameters according to a preset model configuration; start a distributed training job on a computing resource cluster, perform multi-round iterative updating on model weights, and dynamically adjust hyperparameters during training; generate a temporary model file snapshot at the end of each round of training, and record an identifier of the temporary model file to a metadata table; after a preset number of training rounds are completed, synchronize the identifier of the latest temporary model file to the pipeline arrangement unit for subsequent calling by the model evaluation and management unit. The model evaluation and management unit is configured to:

4. The system of claim 1, wherein, extract a corresponding temporary model according to the identifier of the temporary model file, perform evaluation index calculation and index comparison on the temporary model, register the temporary model meeting a preset index threshold as a target model, and generate a target model version identifier, call a corresponding base image and environment variable construction service package based on the target model version identifier, and push the service package to a preset DevOps image repository to trigger a deployment process. The release management unit is configured to:

5. The system of claim 1, wherein, call a corresponding base image and environment variable construction service package based on the target model version identifier, and push the service package to a preset DevOps image repository to trigger a deployment process. According to the temporary model file identifier, a corresponding temporary model is extracted, and a preset evaluation data set is called to run an inference process to generate an evaluation index; The evaluation index is compared with a preset threshold value or a historical benchmark value to determine whether the temporary model meets a registration condition; When the registration condition is met, the temporary model is written into a model registration library, a target model version identifier is generated, and the evaluation index and the target model version identifier are stored in association in the model registration library; The target model version identifier is synchronized to the pipeline arrangement unit for subsequent calling by the release management unit.

6. The system of claim 1, wherein, The release management unit is specifically configured to: extract a target model and its dependent configuration based on the target model version identifier; call a base image and an environment variable corresponding to the target model to construct a service package containing an inference script and a model weight file; write a version number, a check value, and deployment metadata for the service package, and generate an image tag corresponding to the target model version identifier; push the service package to a preset DevOps image repository, and trigger a continuous delivery pipeline through an interface to be deployed to a computing resource cluster; register a mapping relationship between the service package and the target model version identifier in the image repository for subsequent rollback or upgrade calling.

7. The system of claim 1, wherein, The system further includes: an algorithm management unit configured to maintain an index relationship among the base image, the model code, the evaluation script, and the resource configuration file, and provide the index relationship to the training management unit, the model evaluation and management unit, and the release management unit.

8. The system of claim 7, wherein, The algorithm management unit is specifically configured to: establish an algorithm metadata directory, and write an algorithm identifier, an applicable scenario, creator information, and a version number into an index table; record a mapping relationship among a base image identifier, a model code repository address, an evaluation script identifier, and a resource configuration template for each algorithm, and maintain the mapping relationship in the index table; return the corresponding base image identifier and resource configuration template when the training management unit initiates an algorithm calling request; return the corresponding evaluation script identifier when the model evaluation and management unit initiates an algorithm calling request; return the corresponding base image identifier and model code repository address when the release management unit initiates an algorithm calling request; generate a version snapshot for the algorithm and update the index table when a new version of the base image, the model code, or the evaluation script is detected.

9. The system of claim 1, wherein, The system further includes: a service calling log unit configured to collect calling data of a deployed service instance, write the calling data into a log storage, and feed back a log identifier to the pipeline arrangement unit.

10. The system of claim 9, wherein, The service calling log unit is specifically configured to: structurally record a request timestamp, a request parameter fingerprint, a response duration, and resource occupation information of the service instance; aggregate the structured records in a preset time window, generate a calling statistical entry, and write the calling statistical entry into the log storage; generate a log identifier for each calling statistical entry, and synchronize the log identifier to the pipeline arrangement unit; provide a retrieval interface based on a service instance identifier or a log identifier when a query instruction of the pipeline arrangement unit is received.

Images