Model training and deployment method and system based on Jenkins and version control, terminal and medium

Abstract

The invention belongs to the technical field of artificial intelligence engineering, and particularly discloses a model training and deploying method and system based on Jenkins and version control, a terminal and a medium, and the method comprises the steps that a submission event of a GitLab code warehouse is monitored through Jenkins, difference data information including a training script, a data set version and model configuration is extracted in combination with DVC metadata, and the submission event of the GitLab code warehouse is obtained; and a training task is dynamically generated, and computing resources are automatically allocated to execute a training process. Through the process, full-process automation and closed-loop management of model training, verification, deployment and monitoring are achieved, development efficiency, deployment safety and model traceability are improved, and the method is suitable for intelligent model development scenes with high-frequency iteration and sensitive performance.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of artificial intelligence engineering, and particularly relates to a model training and deployment method, system, terminal and medium based on Jenkins and version control. BACKGROUND

[0002] In existing artificial intelligence applications, especially in high-demand scenarios such as medical auxiliary diagnosis and intelligent recommendation, the development and deployment frequency of machine learning models is constantly increasing, and the model version update cycle is becoming shorter. Traditional model development processes usually rely on manual operations and dispersed tool combinations, such as using Git to manage code and using DVC or other tools to manage data, but lack a unified automated scheduling and verification system, resulting in multiple key problems in the model development and deployment process.

[0003] Firstly, there is a lack of strong binding mechanism between code and data versions, and model training often relies on manual confirmation of configuration and path consistency, which can easily lead to non-reproducible training environments or version chaos. Secondly, existing processes usually use "commit differences" or "time intervals" in code repositories as training task triggering basis, lack intelligent analysis of the actual impact range of changes, and even small modifications to unrelated modules (such as log recording, interface annotation, etc.) can trigger the entire model training process, resulting in serious waste of computing resources and reduction of system throughput. Thirdly, model verification methods generally rely on a single accuracy evaluation, ignoring the requirements for response delay, system stability, and adversarial robustness in the actual deployment process. In addition, the model deployment process is mostly manual, lacking version traceability and automatic rollback mechanisms, resulting in a long time-consuming and high-risk process for locating and repairing faults in production environments. SUMMARY

[0004] The present application provides a model training and deployment method, system, terminal and medium based on Jenkins and version control, which solves the problem of lack of strong binding mechanism between code and data versions in the prior art, and the problem that model training often relies on manual confirmation of configuration and path consistency, which can easily lead to non-reproducible training environments or version chaos. It also solves the problem that even small modifications to unrelated modules can trigger the entire model training process, resulting in serious waste of computing resources and reduction of system throughput.

[0005] The technical solutions adopted by the present application are as follows: In a first aspect, the present application provides a model training and deployment method based on Jenkins and version control, which comprises the following steps: Step S1, through Jenkins, listen to and obtain the update submission submitted by the user to the GitLab code repository; When it is detected that there is a submission difference, calculate the change impact score; When the change impact score exceeds the preset threshold, extract difference data information including training scripts, dataset versions, and model configurations; Otherwise, record the version identifier of this submission, the list of changed files, and the impact analysis results, and skip the model training process, only update the model version log and experiment record status, and deploy to the pre-release environment or production environment; Step S2, Jenkins generates training tasks according to the difference data information in step S1 and dynamically allocates computing resources to execute the training tasks; Load the dataset and model version corresponding to the difference data information; Execute the training script, generate a model weight file with a unique version identifier, and record the configuration file during the training process to MLflow or equivalent tools; During the training process, if resource usage anomalies are detected, the task is aborted and the intermediate model state is saved; Step S3, perform test verification operations on the trained model, and the verification test includes functional testing, performance testing, and security testing; Step S4, based on the model weight and configuration file that pass the test verification, build a Docker image with a unique version identifier, and deploy it to the pre-release environment or production environment, while updating the model service registration information; Step S5, perform online verification in the production environment, compare the index differences of the new and old models in the actual business scenario; If service anomalies are detected, trigger the rollback process to restore to the model image of the previous stable version.

[0006] Further, in step S1, the change impact score is calculated according to the following scoring function:

[0007] Where S is the change impact score; is the model sensitivity factor, representing the prior influence of the ith type of change on the model performance; is the base weight of the ith type of change; is a function mapping of the change amount; is a dependency propagation correction factor, used to describe the spread range of code changes in the dependency graph; In , is the number of semantic change points, which are extracted by static analysis tools and classified and counted by a set of rule engines, thereby obtaining ; The process of extracting differential data information includes parsing the joint change record of GitLab commit logs and DVC metadata, and locating the specific change content of training scripts, dataset versions and model parameter configurations by comparing the hash value of Git commit with the version index tracked by DVC.

[0008] Further, the generation of the training task in step S2 includes dynamically constructing task execution parameters according to the differential data information, and the parameters include at least one computing resource allocation strategy, wherein the computing resource scheduling satisfies the following constraint condition:

[0009] Wherein, S is the size of the dataset, C is the model complexity coefficient, R is the current available computing power coefficient, is the minimum required GPU number, and the training task is allocated resources according to the calculation result.

[0010] Further, the process of executing the training script in step S2 includes using an adaptive learning rate adjustment algorithm to dynamically adjust the learning rate according to the loss function change rate on the current validation set in each training round, and the algorithm includes the following formula:

[0011] Wherein, is the learning rate of the current round, is the adjustment coefficient, and are the current and previous round validation losses respectively, is a constant to prevent division by zero.

[0012] Further, the performance test in step S3 includes predicting the inference performance bottleneck according to the loss change rate in the model training process, and using the following regression estimation model to predict the concurrent delay:

[0013] Wherein, is the 99th percentile response delay predicted, is the loss change rate, is the learning rate of the final training round, is the regression coefficient, and the predicted value is used to determine whether to skip the performance measurement stage or directly trigger the resource optimization suggestion.

[0014] Further, the security test in step S3 includes performing a fast gradient sign method attack on the model, and introducing a performance sensitivity weighted perturbation factor, and the perturbation generation formula is as follows:

[0015] Wherein, is an original sample, is an adversarial sample after adding perturbation, is a base perturbation amplitude coefficient, is a performance sensitivity amplification factor, is a model loss function, is a model parameter, is a target label, is an input sample the gradient of the loss function.

[0016] Further, the version identification Tag generated by constructing the Docker image in step S4 encodes the following information:

[0017] wherein, is the hash value of the current Git commit, is the version number of the dataset / model weight recorded by the current DVC, is the residual accuracy of the model in the adversarial sample attack test; whether to enter the production environment deployment in step S4 is determined by the following comprehensive score:

[0018] wherein, is the deployment comprehensive score, , , is a weighting coefficient, satisfying , is the prediction accuracy of the model on the validation set or test set, is the maximum delay threshold set by the system.

[0019] In a second aspect, the present application provides a model training and deployment system based on Jenkins and version control, comprising: a listening module for listening to and obtaining an update submission submitted by a user to a GitLab code repository through Jenkins, and extracting difference data information including a training script, a dataset version and a model configuration when detecting a submission difference; a task generation module for generating a training task according to the difference data information and dynamically allocating computing resources to execute the training task; a training execution module for loading the dataset and model version corresponding to the difference data information, executing the training script, generating a model weight file with a unique version identification, and recording the configuration file in the training process to MLflow or an equivalent tool, and if an abnormal resource usage is detected during the training process, the task is aborted and the intermediate model state is saved; A test verification module is configured to perform test verification operations on the trained model, including function tests, performance tests and security tests; A deployment module is configured to construct a Docker image with a unique version identifier based on the model weight and configuration file verified by the test, and deploy it to a pre-release environment or a production environment, while updating the model service registration information; An online verification module is configured to perform online verification in the production environment, compare the index differences between the new and old models in the actual business scenario, and trigger a rollback process to restore to the model image of the previous stable version when service anomalies are detected.

[0020] In a third aspect, the application provides a terminal, comprising: A memory is configured to store a Jenkins and version control based model training and deployment program; A processor is configured to perform the steps of the Jenkins and version control based model training and deployment method as described in the first aspect when executing the Jenkins and version control based model training and deployment system.

[0021] In a fourth aspect, the application provides a computer readable storage medium, which stores computer instructions, and when the computer reads the computer instructions in the storage medium, the computer executes the Jenkins and version control based model training and deployment method as described in the first aspect.

[0022] The above technical solutions can be seen that the advantages of the present application are: (1) The present application realizes unified change perception mechanism for three types of core resources, i.e. code, data and configuration, by Jenkins listening to GitLab submission events and combining DVC metadata analysis, improves the accuracy of version tracking and the real-time performance of training triggering, effectively reduces the frequency of manual operation intervention, and enhances the degree of process automation; The present application introduces a model change impact analysis mechanism based on preset rules, which can automatically evaluate the potential impact of code submission on model training results, realize on-demand triggering of training tasks and fine-grained scheduling of resources, and thus significantly improve the resource utilization and engineering controllability of the training process.

[0023] (2) The scheduling constraint formula between data size, model complexity and system available computing power is introduced, combined with dynamic GPU allocation strategy, to realize on-demand allocation and optimized utilization of computing resources, significantly improve training efficiency and reduce resource waste, and has superior scalability in high concurrency and multi-task scenarios.

[0024] (3) An adaptive learning rate adjustment algorithm based on the rate of change of training loss is adopted, and a delay prediction model associated with the change of loss and learning rate is constructed, which not only improves the convergence efficiency of the model training process, but also provides data support for performance bottleneck prediction, supporting skipping the actual stress test link in some scenarios and improving test efficiency.

[0025] (4) The predicted delay index is embedded in the process of generating adversarial samples, and by introducing a delay sensitivity disturbance factor, the safety vulnerability of the model under high load conditions is effectively simulated, the comprehensive evaluation ability of the model robustness is enhanced, and the depth of risk control before deployment is improved.

[0026] (5) A unified Docker image version identifier is constructed by integrating Git commit hash, DVC data version, performance delay and security robustness, so that each image Tag has complete traceability capability, which is conducive to deployment rollback, index comparison and version evolution management, and improves the transparency and controllability of the whole life cycle of the model. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed to be used in the description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of these drawings.

[0028] Figure 1 The flowchart of the model training and deployment method based on Jenkins and version control provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0029] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0030] Please refer to Figure 1 The present application provides a model training and deployment method based on Jenkins and version control, which comprises the following steps: Step S1, through Jenkins, the user's update submission to the GitLab code repository is listened to and obtained; When it is detected that there is a submission difference, the change impact score is calculated; When the change impact score exceeds the preset threshold, the difference data information including the training script, the data set version and the model configuration is extracted; Otherwise, record the version identifier of this submission, the list of changed files and the impact analysis results, and skip the model training process, only update the model version log and experiment record status, and deploy to the pre-release environment or production environment; In step S1, the change impact score is calculated according to the following scoring function:

[0031] Where S is the change impact score; is the model sensitivity factor, representing the prior influence degree of the i-th type of change on the model performance; is the base weight of the i-th type of change; is the function mapping of the change amount; is the dependency propagation correction factor, used to describe the spread range of code changes in the dependency graph; In , is the number of semantic change points, which are extracted by static analysis tools and classified and counted by a set of rule engines, thereby obtaining ; The process of extracting difference data information includes parsing the joint change record of GitLab submission log and DVC metadata, and locating the specific change content of training script, dataset version and model parameter configuration by comparing the hash value of Git submission and the version index tracked by DVC.

[0032] Step S1 in the application is used to realize automatic detection and difference content extraction of model training related changes, to drive the subsequent automatic training process. This step listens to the submission event in the GitLab code repository through the Jenkins continuous integration tool, and starts the change recognition process after listening to the user submission update. In order to ensure the accuracy and integrity of the change content, the system not only detects the submission difference at the code level, but also combines the DVC data version control mechanism to comprehensively extract change information in aspects such as training script, dataset version and model configuration file.

[0033] In practical applications, GitLab is mainly responsible for managing code versions, including model training scripts and parameter configuration files, while DVC is used to manage large volume files such as datasets and model weights closely related to model training. After the combination of the two, all resources relied on by a model version can be completely covered. After receiving the update trigger event from GitLab, Jenkins first identifies the Git hash value corresponding to the current commit and parses the file differences contained in this commit. Then, the system further compares the version index in the DVC metadata to obtain the specific version of the dataset or model file referenced by this commit. By comparing the Git commit record and the DVC version index, the system can accurately identify whether there are key differences such as updates to training script content, changes to dataset path or content, and adjustments to model parameter configuration.

[0034] For example, when the algorithm structure in the training script is modified or the numerical values of hyperparameters such as learning rate and training rounds in the configuration file are changed, the system can locate these changes by comparing the differences between the previous and subsequent versions. Similarly, when the version of the data file tracked by DVC is switched, even if the code itself has not been significantly modified, the system can determine that it is an effective difference that affects the model training result. After the above difference information is extracted and summarized, it will be used as a direct basis for subsequent training task generation and scheduling, ensuring the pertinence, correctness and automation level of the training process, effectively supporting the rapid response and consistency control of model iteration.

[0035] Step S2, Jenkins generates a training task according to the difference data information in step S1 and dynamically allocates computing resources to execute the training task; Load the dataset and model version corresponding to the difference data information; Execute the training script, generate a model weight file with a unique version identifier, and record the configuration file during the training process in MLflow or an equivalent tool; During the training process, if an abnormal resource usage is detected, the task is aborted and the intermediate model state is saved; The generation of the training task in step S2 includes dynamically constructing task execution parameters according to the difference data information, the parameters including at least one computing resource allocation strategy, wherein the computing resource scheduling satisfies the following constraint condition:

[0036] Where S is the dataset size, C is the model complexity coefficient, R is the current available computing power coefficient, is the minimum number of GPUs required, and the training task is allocated resources according to this calculation result.

[0037] The process of executing the training script in step S2 includes using an adaptive learning rate adjustment algorithm to dynamically adjust the learning rate according to the loss function change rate on the current validation set in each training round, and the algorithm includes the following formula:

[0038] wherein, is the learning rate of the current round, is the adjustment coefficient, and are the current and previous round validation losses, respectively, is a constant to prevent division by zero.

[0039] Step S2 in the present application is used to realize the automatic generation of training tasks, resource scheduling, training execution and dynamic optimization in the training process. Based on the difference data information extracted in step S1, the training process is triggered by the Jenkins automation engine, and adaptive resource allocation is performed in combination with the task size and system resource state. The generation of the training task is not only a simple execution call of the model script, but also dynamically constructs execution parameters including model size, data capacity and system resource capacity according to the characteristics of the current task, so as to optimize the scheduling and execution efficiency of the training task.

[0040] In the training task generation stage, the system first parses the key indicators in the difference data information, including the data set size, the model structure complexity and the related configuration parameters, combines the available computing power resources in the current system, adopts a preset resource scheduling strategy, and calculates the minimum resource unit required for training. In one embodiment, the system constructs a calculation model according to the data volume, the model complexity and the available resource ratio, to estimate the number of GPUs required for training, so as to realize accurate allocation of resources and avoid resource waste or performance bottlenecks.

[0041] After resource scheduling is completed, the system loads the data set and model version corresponding to the difference data information, and calls the training script to start the training process. In the training process, the system introduces an adaptive learning rate adjustment mechanism to dynamically adjust the learning rate according to the loss change trend of the validation set after each training round. The adjustment mechanism calculates the difference between the current loss value and the previous loss value, judges the convergence rate, and accordingly appropriately increases or decreases the learning rate, so as to speed up the convergence rate or suppress the oscillation. In the algorithm design, the adjustment coefficient and the zero division stability factor are introduced to ensure the smoothness and numerical safety of the learning rate change process. This mechanism can significantly improve the training efficiency and effectively reduce the performance fluctuations caused by improper learning rate.

[0042] In addition, the system monitors resource usage in real time during the training process, especially key indicators such as memory occupancy and computing load. When an abnormal resource usage is found, for example, the memory exceeds the set threshold or the GPU is unresponsive for a long time, the system will immediately suspend the current training task and store the completed intermediate state model to ensure that part of the results is preserved and provide basic support for subsequent task recovery or optimization. Through the above mechanism, the present application can realize dynamic scheduling, intelligent control and risk tolerance of the training process, and provide robust and efficient support for high-frequency and high-complexity model iteration.

[0043] Step S3, performing test verification operation on the trained model, the verification test including function test, performance test and security test; The performance test in step S3 includes predicting the inference performance bottleneck according to the loss change rate in the model training process, and using the following regression estimation model to predict the concurrent delay:

[0044] Wherein, is the 99th percentile response delay of the prediction, is the loss change rate, is the learning rate of the final training round, is the regression coefficient, and the predicted value is used to determine whether to skip the performance measurement stage or directly trigger resource optimization suggestion.

[0045] The security test in step S3 includes performing fast gradient sign method attack on the model, and introducing performance sensitivity weighted perturbation factor, and the perturbation generation formula is as follows:

[0046] Wherein, is the original sample, is the adversarial sample after adding perturbation, is the basic perturbation amplitude coefficient, is the performance sensitivity amplification factor, is the model loss function, is the model parameter, is the target label, is the input sample the gradient of the loss function.

[0047] The step S3 in the application is used for performing systematic test verification on the trained model, ensuring that the model meets the preset standards in multiple dimensions such as function, performance and security before being put into deployment. Among them, the function test is used to verify whether the output format of the model under standard input conforms to the specification, the logic is consistent, and the basic matching relationship with the expected label is established. As a basic inspection link, the function test is mainly used to exclude structural errors or serious deviations that may occur in the training process.

[0048] In terms of performance testing, the application does not rely only on traditional stress testing tools to directly apply load, but introduces a prediction mechanism based on training process data. Specifically, after the model training is completed, the system extracts the loss function change of the model at each training round, and combines the learning rate of the final round to construct a regression model for predicting the response delay of the model under high concurrency conditions, especially focusing on the 99th percentile response time. The prediction model estimates the inference stability of the model in the convergence state through the relationship between the loss reduction speed and the learning rate, and then infers the possible performance bottleneck of the model in the concurrent request scenario. If the prediction result shows that the model still has stability under high load, the system can choose to skip part of the performance measurement steps, thereby shortening the online period; if the prediction result approaches or exceeds the delay threshold set by the system, the system will trigger resource optimization suggestions, including adjusting the deployment structure, rescheduling computing resources and other strategies, to ensure stable operation of subsequent services.

[0049] In terms of security testing, the application introduces an adversarial sample attack mechanism to test the robustness of the model. Specifically, the fast gradient sign method is used to simulate the attack path and add perturbations to the input data to evaluate the response stability and anti-interference ability of the model when facing unexpected inputs. To enhance the practical significance of the test, the prediction delay result in the performance test is introduced into the perturbation calculation process, and a performance sensitivity amplification factor is set to make the perturbation received by the model more aggressive under heavy inference bottleneck conditions, thereby testing its stability under extreme operating conditions. The perturbation design considers factors such as input gradient, loss function, model parameters and labels, and constructs a composite test environment through the weighted relationship between perturbation amplitude and inference load. If the accuracy of the model significantly decreases under this test, it is considered that the robustness risk is too high and is not allowed to go online.

[0050] In one specific embodiment, the system automatically reads the validation set loss value in each training process and calculates the change rate after the model training is completed, while recording the final learning rate during the training process. Based on the decline speed of the training loss and the stability of the learning rate, the system applies a set of regression coefficients to predict whether the response time of the model under the scenario of 100 concurrent user accesses meets the business requirements. Once the prediction result exceeds the preset threshold, the performance alarm is triggered, the system suspends automatic deployment, and prompts the need to perform resource reallocation or model retraining.

[0051] Subsequently, the system applies the aforementioned prediction delay result as part of the perturbation adjustment factor during the security test phase. The perturbation generation strategy is based on the original sample, combined with the gradient information of the current model to the input and the partial derivative direction of the loss function, and applies a perturbation term with an amplification coefficient. The perturbation strength is adaptively adjusted according to the degree of performance risk, thereby establishing a linkage test system between performance and security. This method not only improves the test pertinence, but also significantly improves the robustness screening ability of the model in the actual deployment environment, ensuring that only high-quality model versions can be put into production operation.

[0052] Step S4, based on the model weight and configuration file verified by the test, build a Docker image with a unique version identifier, and deploy it to the pre-release environment or production environment, while updating the model service registration information; The version identifier Tag generated by building the Docker image in step S4 encodes the following information:

[0053] Among them, is the hash value of the current Git commit, is the version number of the dataset / model weight recorded by the current DVC, is the residual accuracy of the model in the adversarial sample attack test; Whether to enter the production environment deployment in step S4 is determined by the following comprehensive score:

[0054] Among them, is the deployment comprehensive score, , , is the weighting coefficient, satisfying , is the prediction accuracy of the model on the validation set or test set, is the maximum delay threshold set by the system.

[0055] The step S4 in the application is used for completing model deployment preparation and online decision, and the core includes the construction of the Docker image and the generation of the version identification thereof, and a deployment score judgment mechanism based on multiple indexes. After the model is verified by testing, the system packs the model weight file output by training, the corresponding training script, parameter configuration and other resources to generate a Docker image, which is used for subsequent consistent deployment in different environments. To ensure that each image has traceability and performance identification capability, the application proposes a generation mechanism of a structured version identification Tag. The version identification not only marks the code and data versions relied on by the image, but also integrates the key performance indicators of the model, so that the deployment image itself has identifiable and comparable ability.

[0056] Specifically, the version identification Tag is composed of three core fields: one is a Git commit hash value, used to identify the code version on which the current image is based; the second is a DVC data index, used to identify the specific version of the used dataset and model weight file; and the third is the residual accuracy of the model in the security test phase, i.e. the prediction accuracy level that the model can still maintain under adversarial sample attack. The three constitute a unified version code, which is convenient for version comparison, problem backtracking and risk assessment in the image repository. For example, when deploying, if it is found that the security performance of a certain version is better than that of other versions, the system can preferentially select and implement differential control. At the same time, when an online problem occurs, the specific dependent version can be traced back through the version identification, and the problem source can be quickly located.

[0057] In order to further improve the decision-making scientificity of the deployment link, the application introduces a comprehensive deployment score mechanism for screening among multiple model versions that pass the test. The score mechanism considers key dimensions such as the accuracy, inference performance and security robustness of the model, and forms a unified numerical score through weighted combination as the basis for judging whether to enter the production environment for deployment. Three weighting factors are introduced in the scoring model, respectively corresponding to the accuracy index, the delay performance index and the security robustness index, and the sum of the weighting coefficients is one, reflecting the deployment priority set by the system designer according to the specific business scenario.

[0058] Among them, the accuracy term is directly taken from the prediction accuracy result of the model on the validation set or test set; the performance term is calculated by a delay prediction model to calculate the 99th percentile inference delay, and compared with the maximum acceptable delay threshold set by the system to form a normalized score; and the security term uses the residual accuracy of the model under adversarial sample attack as the robustness score. Through the weighted combination of the three indexes, the system can quantify the deployment value of different model versions, and set a score threshold, only the model with a score higher than the threshold is allowed to enter the production environment. For the model with a score close to the critical value, the system can automatically recommend delay deployment, strengthen monitoring or manual review, etc. to avoid the online of high-risk versions.

[0059] In one specific embodiment, after the system completes the model function test, performance prediction and security assessment, it automatically calls the version identification construction module to generate an image Tag containing the code version, data version and security robustness field. Subsequently, the system collects the accuracy, prediction delay value and residual accuracy of the adversarial attack of the model version, substitutes it into the scoring formula for comprehensive calculation, and compares the result with the deployment threshold. If the score is significantly higher than the set standard, the system automatically completes the image release, service registration and online action; if the score is insufficient, the online process is suspended and optimization suggestions are output to ensure that the online model has reliability, stability and security. This mechanism constitutes a quality closed loop from the training product to the actual deployment, effectively supporting intelligent management and high-quality operation and maintenance of the model throughout its life cycle.

[0060] Step S5, performing online verification in the production environment, comparing the index difference between the new and old models in the actual business scenario; If service anomalies are monitored, the rollback process is triggered to restore to the model image of the previous stable version.

[0061] Step S5 in the present application is used to realize the running state verification and abnormal response mechanism of the deployed model in the production environment, ensuring that the new version model can continuously maintain stable performance and reliable output after being officially put online. After the model is deployed to the production system, in order to further confirm its adaptation effect in the real business scenario, the system will start the online verification process, compare the new model with the old model currently running online in real time, and evaluate its key business index performance under actual input data.

[0062] This verification process is usually carried out in the way of canary release or A / B testing, that is, part of the business traffic is directed to the new model, and the rest remains unchanged. Thus, the output performance of the two models under the same business conditions is obtained within the same time window. The system focuses on collecting multiple dimension data including prediction accuracy, interface response time, request error rate, business conversion index, etc., and determines whether the performance difference between the new and old models is significant and whether it meets the online expectation through statistical analysis. If the new model performs stably or better than the old model in the main indicators, the system will gradually increase its traffic proportion and eventually replace it completely; if the new model has performance degradation, increased delay or rising abnormal response rate, etc., the system will determine that the new version has deployment risks and is not suitable for continuous operation.

[0063] To ensure the stability of online services, the application sets up an abnormality monitoring and rollback mechanism. When the core indicators (such as delay and error rate) exceed the preset threshold in the online verification process for a plurality of monitoring periods in succession, or a dramatic performance fluctuation occurs within a short period of time, the system will automatically trigger the model version rollback process. The rollback process quickly locates the last stable version based on the image version management mechanism, and replaces the original service instance with the mirror image of the stable version, while completing service registration, traffic switching and state synchronization, to ensure business continuity is not affected.

[0064] In one specific embodiment, after the model is deployed, the system first directs no more than ten percent of real-time request traffic to the new model for preliminary verification without affecting the main business. The system continuously monitors the response time, accuracy, interface success rate and other key indicators of the version model in the background, and generates an analysis report every set period. If it is found that the response time of the new model interface increases by more than fifty percent compared to the old model, or the error rate is continuously higher than the set baseline, the system will immediately suspend the service of the model, call the last mirror version to replace the existing instance, and synchronously adjust the configuration file and registration information to complete the full-process automated rollback. Through this mechanism, the application not only enables fast model online, but also has high controllability and fault-tolerant operation guarantee capability, significantly improving the safety and business continuity of model deployment.

[0065] In some embodiments, the present application provides a model training and deployment system based on Jenkins and version control, which comprises: A listening module for listening to and obtaining an update submission submitted by a user to a GitLab code repository through Jenkins, and extracting difference data information including training scripts, dataset versions and model configurations when detecting submission differences; A task generation module for generating a training task according to the difference data information and dynamically allocating computing resources to execute the training task; A training execution module for loading the dataset and model version corresponding to the difference data information, executing the training script, generating a model weight file with a unique version identifier, and recording the configuration file in the training process to MLflow or equivalent tools, and if an abnormal resource usage is detected during the training process, the task is aborted and the intermediate model state is saved; A test verification module for performing a test verification operation on the trained model, the test including a function test, a performance test and a security test; A deployment module for constructing a Docker image with a unique version identifier based on the model weight and configuration file verified by the test, and deploying it to a pre-release environment or a production environment, while updating the model service registration information; The online verification module is used to perform online verification in the production environment, compare the differences in metrics between the old and new models in actual business scenarios, and trigger a rollback process when service anomalies are detected to restore the model image to the previous stable version.

[0066] The system first includes a monitoring module, which uses a Jenkins webhook mechanism to listen for updates submitted by users to the GitLab code repository. When the monitoring module detects a new commit event, it automatically triggers a change extraction process to identify changes to key resources, including training scripts, dataset versions, and model configurations. This module can comprehensively analyze Git commit records and DVC metadata to accurately determine whether there are any valid changes affecting model training results.

[0067] After the monitoring module completes the differential data extraction, the task generation module receives the differential data information and constructs a training task accordingly. The training task generation process combines information such as the current model complexity, dataset size, and system idle resources, automatically matching the appropriate type and quantity of computing resources. Particularly in GPU resource allocation, it can dynamically allocate the resources required for concurrent training according to scheduling rules, ensuring that the task is completed efficiently within the given resource constraints. After task generation, the parameters required for task execution are passed to downstream modules, automatically starting the training process.

[0068] The training execution module is responsible for actually calling the training script, loading the specified version of the dataset and model parameters, and executing model training. During training, this module continuously records model configuration, training metrics, and runtime environment information, and writes this data to a unified model management platform such as MLflow, ensuring the traceability and repeatability of all training versions. Simultaneously, this module has resource monitoring capabilities, capable of detecting the usage status of GPU or CPU resources in real time. When it detects situations such as memory usage exceeding a threshold, uneven computing power utilization, or abnormal interruption of the training process, the system will proactively terminate the training task and save the intermediate model state of the current training epoch for easy recovery and continuation of training later.

[0069] After training, the testing and validation module automatically takes over the output results and performs comprehensive multi-dimensional testing on the model. This module includes at least functional testing to verify the model's output format and basic accuracy; performance testing to predict response latency or system bottlenecks in high-concurrency scenarios; and security testing to verify the model's robustness against adversarial examples or anomalous inputs. By comprehensively evaluating the results of these three types of tests, the system assesses whether the model is ready for deployment.

[0070] The deployment module starts operation after the test passes, and is responsible for packaging the training product and related configurations into a standard Docker image. When building the image, the deployment module automatically generates a unique version identifier, including code commit hash, data version number, and security robustness information, to ensure the image's identifiability and traceability. The module deploys the built image to the pre-release or production environment, updates the model service registration information, and completes the replacement and release of the model service version.

[0071] To ensure the effectiveness of the model in real business scenarios after deployment, the system also includes an online verification module. This module guides part of the request traffic to access the new model instance and compares the core business indicator performance of the new and old models in actual operation, such as prediction accuracy, response time, and error rate. When the system detects performance degradation or response abnormalities in the new model, it immediately triggers an automatic rollback mechanism to replace the current service image with the last verified stable version, ensuring the continuous availability and service quality of the business system. Through the close cooperation of the above modules, the present application realizes the full-process automated management of the model from submission to training, from verification to deployment, and from monitoring to rollback, with high scalability, reliability, and engineering practical value.

[0072] In some embodiments, the present application provides a terminal, comprising: a memory for storing a Jenkins and version control based model training and deployment program; a processor for executing the Jenkins and version control based model training and deployment system to implement the steps of the Jenkins and version control based model training and deployment method.

[0073] In some embodiments, the present application provides a computer readable storage medium, which stores computer instructions, and when a computer reads the computer instructions in the storage medium, the computer executes the Jenkins and version control based model training and deployment method.

[0074] The above is merely an example and explanation of the present application, and those skilled in the art can make various modifications or supplements or use similar ways to replace the described specific embodiments, as long as they do not deviate from the concept of the present application or exceed the scope defined by the present application, and all should be within the protection scope of the present application.

Claims

1. A model training and deployment method based on Jenkins and version control, characterized in that, Includes the following steps: Step S1: Monitor and retrieve user-submitted update commits to the GitLab code repository using Jenkins; When a difference in submissions is detected, a change impact score is calculated. When the impact score of the change exceeds the preset threshold, extract the difference data information, including training script, dataset version and model configuration. Otherwise, record the version identifier, list of changed files and impact analysis results of this submission, skip the model training process, only update the model version log and experimental record status, and deploy to the pre-release environment or production environment; Step S2: Jenkins generates training tasks based on the difference data information in step S1 and dynamically allocates computing resources to execute the training tasks; Load the dataset and model version corresponding to the difference data information; Execute the training script, generate a model weight file with a unique version identifier, and record the configuration file during the training process to MLflow or an equivalent tool; If abnormal resource usage is detected during training, the task is stopped and the intermediate model state is saved. Step S3: Perform testing and verification operations on the trained model. The verification tests include functional testing, performance testing, and security testing. Step S4: Based on the model weights and configuration files that have passed the test and verification, build a Docker image with a unique version identifier and deploy it to the pre-release environment or production environment, while updating the model service registration information; Step S5: Perform online verification in the production environment to compare the differences in metrics between the old and new models in actual business scenarios; If a service anomaly is detected, a rollback process is triggered to restore the model image to a previous stable version.

2. The model training and deployment method based on Jenkins and version control according to claim 1, characterized in that, In step S1, the impact score of the change is calculated according to the following scoring function: Where S represents the impact score of the change; is the model sensitivity factor, representing the prior impact of the i-th type of change on model performance; The basic weight for the i-th type of change; For function mapping of changes; This is a dependency propagation correction factor used to describe the scope of code changes in the dependency graph; exist middle, The number of semantic change points is calculated by extracting them using static analysis tools and then classifying and counting them using a predefined rule engine. ; The process of extracting discrepancy data information includes parsing the joint change record of GitLab commit logs and DVC metadata, and locating the specific changes in training scripts, dataset versions and model parameter configurations by comparing the hash value of Git commits with the version index tracked by DVC.

3. The model training and deployment method based on Jenkins and version control according to claim 1, characterized in that, Step S2 involves generating the training task by dynamically constructing task execution parameters based on the differential data information. These parameters include at least one computational resource allocation strategy, wherein the computational resource scheduling satisfies the following constraints: Where S is the dataset size, C is the model complexity coefficient, and R is the currently available computing power coefficient. To minimize the number of GPUs required, the training task allocates resources based on the calculation results.

4. The model training and deployment method based on Jenkins and version control according to claim 3, characterized in that, The process of executing the training script in step S2 includes using an adaptive learning rate adjustment algorithm to dynamically adjust the learning rate in each training epoch based on the rate of change of the loss function on the current validation set. The algorithm includes the following formula: in, The learning rate for the current round. For adjustment coefficients, and These represent the current and previous rounds of verification losses, respectively. To prevent division by zero of constants.

5. The model training and deployment method based on Jenkins and version control according to claim 4, characterized in that, The performance testing in step S3 includes predicting inference performance bottlenecks based on the rate of loss change during model training, and using the following regression estimation model to predict concurrent latency: in, The predicted 99th percentile response delay, For the rate of change of loss, The learning rate for the final training round. The predicted value is used as a regression coefficient to determine whether to skip the performance testing phase or directly trigger resource optimization suggestions.

6. The model training and deployment method based on Jenkins and version control according to claim 5, characterized in that, The security test in step S3 includes performing a fast gradient sign attack on the model and introducing a performance-sensitive weighted perturbation factor, the perturbation generation formula of which is as follows: in, For the original sample, For adversarial examples with added perturbations. Based on the basic disturbance amplitude coefficient, This is the performance sensitivity amplification factor. The model loss function, For model parameters, For target tags, For input samples The gradient of the loss function.

7. The model training and deployment method based on Jenkins and version control according to claim 6, characterized in that, The version identifier tag generated during step S4 when building the Docker image contains the following information: in, The hash value of the current Git commit. This is the version number of the dataset / model weights in the current DVC record. The residual accuracy of the model in adversarial sample attack tests; Whether to proceed with production deployment in step S4 is determined by the following comprehensive scoring: in, To deploy a comprehensive scoring system, , , For the weighting coefficients, satisfying , The prediction accuracy of the model on the validation or test set. The maximum latency threshold set for the system.

8. A model training and deployment system based on Jenkins and version control, characterized in that, The system includes: The monitoring module is used to monitor and obtain user updates to the GitLab code repository through Jenkins, and when a difference in the commit is detected, it extracts the difference data information, including training scripts, dataset versions and model configurations. The task generation module is used to generate training tasks based on the difference data information and dynamically allocate computing resources to execute the training tasks. The training execution module is used to load the dataset and model version corresponding to the differential data information, execute the training script, generate a model weight file with a unique version identifier, and record the configuration file during the training process to MLflow or an equivalent tool. If abnormal resource usage is detected during the training process, the task is stopped and the intermediate model state is saved. The testing and verification module is used to perform testing and verification operations on the trained model. The tests include functional testing, performance testing, and security testing. The deployment module is used to build a Docker image with a unique version identifier based on the model weights and configuration files that have passed the test and verification, and deploy it to the pre-release environment or production environment, while updating the model service registration information. The online verification module is used to perform online verification in the production environment, compare the differences in metrics between the old and new models in actual business scenarios, and trigger a rollback process when service anomalies are detected to restore the model image to the previous stable version.

9. A terminal, characterized in that, include: Storage is used to store model training and deployment programs based on Jenkins and version control; A processor is used to implement the steps of the model training and deployment method based on Jenkins and version control as described in claim 1 when executing the Jenkins and version control-based model training and deployment system.

10. A computer-readable storage medium, characterized in that, The storage medium stores computer instructions. When the computer reads the computer instructions from the storage medium, the computer executes the model training and deployment method based on Jenkins and version control as described in claim 1.

Images