Large model training system

Abstract

The embodiment of the invention relates to a large model training system. The system comprises a model design module, a data acquisition module, a model training module, a model monitoring module, a basic model library, a large model library, a model database and a monitoring database, wherein the model design module is used for large model assembly and data set initialization; the data acquisition module is used for updating the model database; the model training module is used for model training; and the model monitoring module is used for deploying large model application examples, monitoring example models, regularly extracting model data according to a monitoring data list, forwarding extracted data packets to the data acquisition module, regularly evaluating model performance according to the monitoring data list to obtain corresponding evaluation results, and feeding back the evaluation results to users. According to the invention, the model training efficiency can be improved, and the real-time monitoring capability of the model performance can be improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a large model training system. Background Art

[0002] With the maturity and development of artificial intelligence (AI) technology and unmanned / autonomous driving technology, a large number of machine learning models and deep learning models have been applied to the field of unmanned / autonomous driving technology. The field of unmanned / autonomous driving has certain customization requirements for AI models and high requirements for their predictive stability, which means: 1) In some scenarios, it is necessary to assemble and customize large models with specific functions based on conventional models; 2) The customized model needs to be fully trained before it is put into use for the first time; 3) After the customized model is put into use, its performance can be tracked and monitored, and when the model performance decreases, the relevant staff can be informed in time to start a new round of model training as soon as possible. At present, no system solution that can meet the above three requirements at the same time has been found on the market, and this is also the technical problem that the present invention needs to solve. Summary of the invention

[0003] The purpose of the present invention is to provide a large model training system in view of the defects of the prior art, which system includes: a model design module, a data acquisition module, a model training module, a model monitoring module, a basic model library, a large model library, a model database and a monitoring database; wherein the model design module is used to perform corresponding large model assembly and model data set initialization according to the basic model library, the large model library and the model database; the data acquisition module is used to perform data update processing on the model database according to the data packet input by the user or the model monitoring module; the model training module is used to perform corresponding model training processing according to the training configuration, the model database and the large model library input by the user; the model monitoring module is used to perform corresponding large model application instance deployment according to the deployment configuration and the large model library input by the user, and create a corresponding monitoring data list for the model instance in the monitoring database, and perform data acquisition on the input / output data of the model instance and the corresponding real observation data and update the corresponding monitoring data list based on the acquisition results, and regularly perform model data extraction according to the monitoring data list to obtain the corresponding data packet and send it to the data acquisition module, and regularly perform model performance evaluation according to the monitoring data list to obtain the corresponding evaluation results and feedback to the user. The present invention can assemble and customize a large model with specific functions based on conventional models, can fully train the customized model before it is put into use for the first time, can update the model data set used for training based on user input data and monitored model application data, can track and monitor the performance of the customized model in real time after it is put into use, and promptly inform relevant staff when the model performance is abnormal so as to start a new round of model training as soon as possible; through the system solution provided by the present invention, the customization complexity of the model can be reduced, the efficiency of model training can be improved, the real-time nature of training data can be improved, and the real-time tracking of model performance can be improved.

[0004] To achieve the above-mentioned purpose, an embodiment of the present invention provides a large model training system, the system comprising: a model design module, a data acquisition module, a model training module, a model monitoring module, a basic model library, a large model library, a model database and a monitoring database;

[0005] The model design module is connected to the basic model library, the large model library and the model database respectively; the data acquisition module is connected to the model database; the model training module is connected to the model database and the large model library respectively; the model monitoring module is connected to the data acquisition module, the large model library and the monitoring database respectively;

[0006] The model design module is used to assemble the corresponding large model and initialize the model data set according to the basic model library, the large model library and the model database;

[0007] The data acquisition module is used to receive a first data packet input by a user or the model monitoring module; and perform data update processing on the model database according to the first data packet;

[0008] The model training module is used to receive a first training configuration input by a user; and perform corresponding model training processing according to the first training configuration, the model database and the large model library;

[0009] The model monitoring module is used to receive a first deployment configuration input by a user; and perform corresponding large model application instance deployment processing according to the first deployment configuration and the large model library to obtain a corresponding first model instance; and create a corresponding first monitoring data list for the first model instance in the monitoring database; and perform data collection on the input / output data of the first model instance and the corresponding real observation data and update the corresponding first monitoring data list based on the collection results; and periodically perform model data extraction according to the first monitoring data list to obtain the corresponding first data packet and send it to the data collection module; and periodically perform model performance evaluation according to the first monitoring data list to obtain the corresponding first evaluation result and feedback it to the user;

[0010] The basic model library includes a plurality of first basic models; each of the first basic models is a type of machine learning model or a deep learning model;

[0011] The large model library includes a plurality of first large models; each of the first large models corresponds to a first large model identifier;

[0012] The model database is used to store multiple first model data sets; the first model data sets correspond to the first large model one by one; each of the first model data sets corresponds to a first data set identifier; the first model data set includes multiple first data records; the first data record includes a first model input tensor and a first model label tensor;

[0013] The monitoring database includes a plurality of the first monitoring data lists; the first monitoring data lists include a plurality of first monitoring data records; the first monitoring data records include a second model input tensor, a first model output tensor and a first real observation tensor.

[0014] Preferably, the model design module is specifically used for assembling the corresponding large model and initializing the model data set according to the basic model library, the large model library and the model database.

[0015] Providing a user with a visual model assembly interface having a draggable toolbar, a draggable design window, and a model submission button; the draggable toolbar includes a plurality of draggable components, and the draggable components correspond one to one with the first basic model of the basic model library;

[0016] When the user drags any of the draggable components into the drag-and-drop design window, a corresponding basic model implementation is created for the first basic model corresponding to the currently dragged draggable component; when the user selects any of the draggable components from the drag-and-drop design window, a corresponding component property configuration toolbar is provided for the user to configure the basic model implementation corresponding to the currently selected draggable component; and in the drag-and-drop design window, a corresponding component connection tool is provided for each of the draggable components to create a connection relationship between the current component and other components;

[0017] When the user clicks the model submission button, a new model is constructed based on the configuration status and connection relationship of all the basic model implementation examples and recorded as the corresponding current construction model; and based on the preset key layer definition rules, a batch normalization layer is added to the output end of each key layer of the current construction model; the current construction model with the batch normalization layer added is added to the large model library as a new first large model; and a unique first large model identifier is assigned to this new first large model; and an empty data set is initialized on the model database for this new first large model as the corresponding first model data set; and a unique first data set identifier is assigned to the first model data set initialized this time.

[0018] Preferably, the first data packet consists of a second data set identifier and a first collected data list; the first collected data list includes a plurality of first collected data records; the first collected data record includes first input data and first label data;

[0019] The first training configuration includes a second large model identifier, a first training round threshold, a first split ratio, a first batch size, a first optimizer, a first initial learning rate, a first learning rate decay factor, a first loss function, a first regularization term type, a first model validation function, a first model evaluation function set, and a first evaluation parameter set;

[0020] The first training round threshold is the maximum training round;

[0021] The first split ratio is the ratio of the total number of records in the training set, the validation set, and the test set;

[0022] The first batch size is the number of samples in a single batch;

[0023] The first optimizer includes an SGD optimizer, an ADAM optimizer and an RMSprop optimizer;

[0024] The first initial learning rate is an optimizer learning rate used by the optimizer in the first round;

[0025] The first learning rate attenuation factor is the attenuation factor of the optimizer learning rate in the previous and next two rounds;

[0026] The first regularization term type includes an L1 regularization term and an L2 regularization term;

[0027] The first model verification function is an accuracy calculation function;

[0028] The first model evaluation function set consists of four types of evaluation functions, namely, accuracy calculation function, precision calculation function, recall calculation function and F1 score calculation function;

[0029] The first evaluation parameter set consists of four types of evaluation value ranges, namely, accuracy range, precision range, recall range and F1 score range;

[0030] The first deployment configuration includes a third large model identifier, a first running host address, a first host software resource configuration parameter, and a first host hardware resource configuration parameter.

[0031] Preferably, the data acquisition module is specifically used to extract the corresponding second data set identifier and the first acquired data list from the first data packet as the corresponding current data set identifier and current data list when the model database is updated according to the first data packet; and perform data cleaning, data standardization and data denoising on the current data list; and use the first model data set whose first data set identifier in the model database matches the current data set identifier as the corresponding current model data set; and use each of the first acquired data records in the current data list as the corresponding current record, and perform corresponding data tensor conversion on the first input data and the first label data of the current record to obtain the corresponding first model input tensor and the first model label tensor to form a corresponding first data record and add it to the current model data set.

[0032] Preferably, the model training module is specifically used for performing corresponding model training processing according to the first training configuration, the model database and the large model library.

[0033] Step 501, extracting the corresponding second large model identifier, the first training round threshold, the first segmentation ratio, the first batch size, the first optimizer, the first initial learning rate, the first learning rate decay factor, the first loss function, the first regularization term type, the first model validation function, the first model evaluation function set and the first evaluation parameter set from the first training configuration;

[0034] Step 502, identifying the type of the first regularization term; if the first regularization term type is an L1 regularization term, the first loss function and an L1 regularization loss term are added to form the corresponding first model loss function; if the first regularization term type is an L2 regularization term, the first loss function and an L2 regularization loss term are added to form the corresponding first model loss function;

[0035] Specifically:

[0036] When the first regularization term type is L1 regularization term:

[0037] When the first regularization term type is L2 regularization term:

[0038] Among them, L M is the loss function of the first model; L A is the first loss function; P is the set of prediction-label pairs; {θ k} is the model parameter set of the current large model, θ k is the kth model parameter, 1≤k≤K, K is the total number of model parameters of the current large model; λ is the preset regularization loss term coefficient;

[0039] Step 503: take the first large model whose first large model identifier matches the second large model identifier in the large model library as the corresponding current large model; and take the first model data set corresponding to the current large model in the model database as the corresponding current model data set; and randomly divide the current model data set into three sub-data sets based on the first division ratio, namely, a first training set, a first validation set and a first test set; and initialize the first round counter to 0; and take the first initial learning rate as the corresponding current learning rate;

[0040] The first training set, the first validation set, and the first test set are all composed of a plurality of the first data records, and the ratio of the total number of records in the first training set, the first validation set, and the first test set satisfies the first segmentation ratio;

[0041] Step 504: divide the first training set into batches based on the first batch size to obtain a plurality of corresponding first batch data sets; record the total number of records in the first batch data sets as the corresponding current iteration threshold; and use the first of the first batch data sets as the corresponding current batch data set; and initialize the first iteration counter to 0;

[0042] wherein the first batch of data sets consists of the first data records of the first batch size;

[0043] Step 505: record the first model input tensor and the first model label tensor of each of the first data records of the current batch data set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding first prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding first prediction-label pair set from all the obtained first prediction-label pairs; and set the learning rate parameter of the first optimizer based on the current learning rate; and bring the first prediction-label pair set into the first model loss function L M , and based on the first optimizer moving towards making the first model loss function L M Perform a round of model parameter modulation on the current large model in the direction of reaching the minimum value;

[0044] Step 506, adding 1 to the first iteration counter; then identifying whether the first iteration counter is less than the current iteration threshold; if so, taking the next first batch data set as the new current batch data set and returning to step 505; if not, going to step 507;

[0045] Step 507: record the first model input tensor and the first model label tensor of each of the first data records of the first validation set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding third prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding third prediction-label pair set from all the obtained third prediction-label pairs; and bring the third prediction-label pair set into the first model loss function L MCalculate the loss value to obtain the corresponding first-round loss value; and calculate the corresponding accuracy rate according to the third prediction-label pair set and the first model verification function to obtain the corresponding first-round accuracy rate; and use the model parameters of the current large model at the current moment as the corresponding first-round model parameters; and form a corresponding first-round record with the first-round loss value, the first-round accuracy rate and the first-round model parameters obtained this time;

[0046] Step 508: add 1 to the first round counter; then use the current learning rate as the corresponding old learning rate; and reset the current learning rate according to the first learning rate attenuation factor, the first round counter and the old learning rate. now =a old ×d ep , a now is the current learning rate after reset, a old is the old learning rate, d is the first learning rate decay factor, and ep is the first round counter; and whether the first round counter is less than the first training round threshold is identified; if so, the first first batch data set is used as the new current batch data set, and the first iteration counter is reset to 0, and the process returns to step 505; if not, the process goes to step 509;

[0047] Step 509, normalize the first-round loss values ​​and the first-round accuracy of all the first-round records obtained respectively; and calculate the corresponding weighted sum coefficient = w1×normalized accuracy+w2×normalized loss value based on the normalized loss value and normalized accuracy corresponding to each first-round record, the normalized accuracy>0, the normalized loss value>0, w1 and w2 are two preset weighting coefficients and w1>0>w2; and use the first-round model parameters of the first-round record with the largest weighted sum coefficient as the corresponding current optimal model parameters; and reset the model parameters of the current large model based on the current optimal model parameters;

[0048] Step 510, record the first model input tensor and the first model label tensor of each of the first data records of the first test set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding fourth prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding fourth prediction-label pair set from all the obtained fourth prediction-label pairs; and perform corresponding accuracy, precision, recall and F1 score calculations according to the fourth prediction-label pair set and the accuracy calculation function, the precision calculation function, the recall calculation function and the F1 score calculation function of the first model evaluation function set to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0049] Step 511, and identify whether the first accuracy rate, the first precision, the first recall rate and the first F1 score obtained all meet the corresponding accuracy rate range, the precision range, the recall rate range and the F1 score range in the first evaluation parameter set; if the first accuracy rate does not meet the corresponding accuracy rate range, or the first precision does not meet the corresponding precision range, or the first recall rate does not meet the corresponding recall rate range, or the first F1 score does not meet the corresponding F1 score range, then reduce the first round counter by 1, and use the first first batch data set as the new current batch data set, and reset the first iteration counter to 0, and return to step 505; if the first accuracy rate, the first precision, the first recall rate and the first F1 score all meet the corresponding accuracy rate range, the precision range, the recall rate range and the F1 score range, then stop model training and solidify the model parameters of the current large model.

[0050] Preferably, the model monitoring module is specifically used to extract the corresponding third large model identifier, the first running host address, the first host software resource configuration parameters and the first host hardware resource configuration parameters from the first deployment configuration when the corresponding large model application instance deployment processing is performed according to the first deployment configuration and the large model library to obtain the corresponding first model instance; and use the first large model that matches the first large model identifier and the third large model identifier in the large model library as the corresponding current large model; and use the running host corresponding to the first running host address as the corresponding current host; and perform corresponding software and hardware resource configuration processing on the large model running environment of the current host based on the first host software resource configuration parameters and the first host hardware resource configuration parameters; and after the software and hardware resources are successfully configured, create a model instance of the current large model on the large model running environment of the current host and record it as the corresponding first model instance.

[0051] Preferably, the model monitoring module is specifically used to collect data on the model input tensor and the corresponding model output tensor of the first model instance at any time t when collecting data on the input / output data of the first model instance and the corresponding real observation data and updating the corresponding first monitoring data list based on the collection result, and record it as the corresponding input tensor x t and the output tensor y t ; and through the preset observation query interface, the input tensor x at any time t t The corresponding real observation data tensor is queried to obtain the corresponding real tensor z t ; and the input tensor x at each time t t , the output tensor y t and the real tensor z t The corresponding second model input tensor, the first model output tensor and the first real observation tensor form a corresponding first monitoring data record which is added to the corresponding first monitoring data list.

[0052] Preferably, the model monitoring module is specifically used to extract the first monitoring data records added in the first monitoring data list within the most recent first time period to form a corresponding first record set when the model data is extracted regularly according to the first monitoring data list to obtain the corresponding first data packet and send it to the data acquisition module; and use the first large model corresponding to the first monitoring data list as the corresponding current large model; and use the first data set identifier corresponding to the current large model as the corresponding second data set identifier; and use the second model input tensor and the first real observation tensor of each of the first monitoring data records in the first record set as the corresponding first input data and the first label data to form a corresponding first acquisition data record; and use all the first acquisition data records obtained this time to form a corresponding first acquisition data list; and use the second data set identifier and the first acquisition data list obtained this time to form a corresponding first data packet and send it to the data acquisition module.

[0053] Preferably, the model monitoring module is specifically used for, when the model performance evaluation is performed regularly according to the first monitoring data list to obtain the corresponding first evaluation result and to provide feedback to the user,

[0054] At every preset second time duration, the current time is taken as the corresponding third time, and the time obtained by subtracting the second time duration from the third time is recorded as the corresponding second time, and the time obtained by subtracting the second time duration from the second time is recorded as the corresponding first time; and the time period from the first time to the second time is recorded as the corresponding first time period; and the time period from the second time to the third time is recorded as the corresponding second time period;

[0055] and extracting the first monitoring data record added in the first time period from the first monitoring data list to form a corresponding second record set; and extracting the first monitoring data record added in the second time period from the first monitoring data list to form a corresponding third record set;

[0056] A corresponding fifth prediction-label pair is formed by the first model output tensor and the first real observation tensor of each of the first monitoring data records in the second record set, and a corresponding fifth prediction-label pair set is formed by all the obtained fifth prediction-label pairs; a corresponding sixth prediction-label pair is formed by the first model output tensor and the first real observation tensor of each of the first monitoring data records in the third record set, and a corresponding sixth prediction-label pair set is formed by all the obtained sixth prediction-label pairs;

[0057] And according to the fifth and sixth prediction-label pair sets, the corresponding prediction error standard deviation is calculated to obtain the corresponding first and second standard deviations; and according to the first and second standard deviations, the corresponding first standard deviation change rate is calculated; and whether the first standard deviation change rate satisfies a preset first change rate range is identified; if satisfied, the corresponding first recognition state is set to no abnormality; if not satisfied, the corresponding first recognition state is set to abnormality; wherein the first recognition state includes no abnormality and abnormality;

[0058] When the first model instance corresponding to the first monitoring data list is a classification model, the corresponding KL divergence calculation is performed according to the fifth and sixth prediction-label pair sets to obtain the corresponding first and second KL divergences; and the corresponding first divergence change rate is calculated according to the first and second KL divergences; and whether the first divergence change rate satisfies a preset second change rate range is identified; if so, the corresponding second recognition state is set to no abnormality; if not, the corresponding second recognition state is set to abnormality; wherein the second recognition state includes no abnormality and abnormality;

[0059] When the first model instance corresponding to the first monitoring data list is a regression calculation model, the corresponding average confidence interval width is calculated according to the fifth and sixth prediction-label pair sets based on the Monte Carlo method to obtain the corresponding first and second average confidence interval widths; and the corresponding first interval width change rate is calculated according to the first and second average confidence interval widths; and whether the first interval width change rate satisfies a preset third change rate range is identified; if so, the corresponding third identification state is set to no abnormality; if not, the corresponding third identification state is set to abnormality; wherein the third identification state includes no abnormality and abnormality;

[0060] And when the first model instance corresponding to the first monitoring data list is a classification model, the first evaluation result corresponding to the third time, the first recognition state and the second recognition state is fed back to the user; and when the first model instance corresponding to the first monitoring data list is a regression calculation model, the first evaluation result corresponding to the third time, the first recognition state and the third recognition state is fed back to the user.

[0061] The embodiment of the present invention provides a large model training system. As can be seen from the above content, the system includes: a model design module, a data acquisition module, a model training module, a model monitoring module, a basic model library, a large model library, a model database and a monitoring database; wherein the model design module is used to perform corresponding large model assembly and model data set initialization according to the basic model library, the large model library and the model database; the data acquisition module is used to perform data update processing on the model database according to the data packet input by the user or the model monitoring module; the model training module is used to perform corresponding model training processing according to the training configuration, the model database and the large model library input by the user; the model monitoring module is used to perform corresponding large model application instance deployment according to the deployment configuration and the large model library input by the user, and create a corresponding monitoring data list for the model instance in the monitoring database, and perform data acquisition on the input / output data of the model instance and the corresponding real observation data and update the corresponding monitoring data list based on the acquisition results, and regularly perform model data extraction according to the monitoring data list to obtain the corresponding data packet and send it to the data acquisition module, and regularly perform model performance evaluation according to the monitoring data list to obtain the corresponding evaluation results and feedback to the user. The embodiments of the present invention can assemble and customize a large model with specific functions based on conventional models, can fully train the customized model before it is put into use for the first time, can update the model data set used for training based on user input data and monitored model application data, can track and monitor the performance of the customized model in real time after it is put into use, and promptly inform relevant staff when the model performance is abnormal so as to start a new round of model training as soon as possible; the system solution provided by the embodiments of the present invention not only reduces the complexity of model customization, improves the real-time nature of training data and model training efficiency, but also improves the real-time tracking of model performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 A module structure diagram of a large model training system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] like Figure 1As shown in the module structure diagram of a large model training system provided by an embodiment of the present invention, the large model training system provided by an embodiment of the present invention mainly includes: model design module 1, data acquisition module 2, model training module 3, model monitoring module 4, basic model library 5, large model library 6, model database 7 and monitoring database 8.

[0065] The connection relationship of each system component in the system is as follows: the model design module 1 is connected to the basic model library 5, the large model library 6 and the model database 7 respectively; the data acquisition module 2 is connected to the model database 7; the model training module 3 is connected to the model database 7 and the large model library 6 respectively; the model monitoring module 4 is connected to the data acquisition module 2, the large model library 6 and the monitoring database 8 respectively. The component functions of each system component in the system are as follows.

[0066] (I) Model design module 1:

[0067] The model design module 1 of the embodiment of the present invention is used to assemble the corresponding large model and initialize the model data set according to the basic model library 5, the large model library 6 and the model database 7.

[0068] In a specific implementation of the embodiment of the present invention, the model design module 1 is specifically used to assemble the corresponding large model and initialize the model data set according to the basic model library 5, the large model library 6 and the model database 7:

[0069] Step A1, providing a user with a visual model assembly interface with a draggable toolbar, a drag-and-drop design window, and a model submission button;

[0070] The draggable toolbar includes a plurality of draggable components, and the draggable components correspond one-to-one to the first basic model of the basic model library 5;

[0071] Step A2, when the user drags any draggable component into the drag-and-drop design window, a corresponding basic model embodiment is created for the first basic model corresponding to the currently draggable component; and when the user selects any draggable component from the drag-and-drop design window, a corresponding component property configuration toolbar is provided for the user to configure the basic model embodiment corresponding to the currently selected draggable component; and in the drag-and-drop design window, a corresponding component connection tool is provided for each draggable component to create a connection relationship between the current component and other components;

[0072] Step A3, when the user clicks the model submission button, a new model is constructed based on the configuration status and connection relationship of all basic model implementation examples and recorded as the corresponding current construction model; and based on the preset key layer definition rules, a batch normalization layer is added to the output end of each key layer of the current construction model; the current construction model with the batch normalization layer added is added to the large model library 6 as a new first large model; and a unique first large model identifier is assigned to this new first large model; and an empty data set is initialized for this new first large model on the model database 7 as the corresponding first model data set; and a unique first data set identifier is assigned to the first model data set initialized this time.

[0073] Here, the key layer definition rules of the embodiment of the present invention can be customized according to specific application requirements. In general, convolutional layers, fully connected layers, etc. are used as key layers.

[0074] (II) Data acquisition module 2:

[0075] The data acquisition module 2 of the embodiment of the present invention is used to receive a first data packet input by a user or a model monitoring module 4; and perform data update processing on the model database 7 according to the first data packet.

[0076] The first data packet is composed of a second data set identifier and a first collected data list; the first collected data list includes a plurality of first collected data records; and the first collected data record includes first input data and first label data.

[0077] Here, the method for updating the model training data set in the embodiment of the present invention can not only be updated based on user input data but also based on the real-time monitoring data of the model monitoring module 4. This can not only expand the data range and enhance the generalization of the data, but also improve the real-time performance of the data.

[0078] In another specific implementation of the embodiment of the present invention, the data acquisition module 2 is specifically used to extract the corresponding second data set identifier and the first collected data list from the first data packet as the corresponding current data set identifier and current data list when the model database 7 is updated according to the first data packet; and perform data cleaning, data standardization and data denoising on the current data list; and use the first model data set whose first data set identifier in the model database 7 matches the current data set identifier as the corresponding current model data set; and use each first collected data record in the current data list as the corresponding current record, and perform corresponding data tensor conversion on the first input data and the first label data of the current record to obtain the corresponding first model input tensor and the first model label tensor to form a corresponding first data record and add it to the current model data set.

[0079] Here, when the embodiment of the present invention performs data cleaning processing on the current data list, it uses a noise removal method to eliminate irrelevant or erroneous information in the data, and fills in the missing items in the data by deletion, induction, interpolation and other methods, and processes the outliers in the data by removal, pruning and replacement. When the embodiment of the present invention performs data standardization processing on the current data list, it scales the data to a uniform range and eliminates the influence of different dimensions by normalization, and converts the data into a distribution with a mean of 0 and a standard deviation of 1 by standardization processing to balance the weights of different features. When the embodiment of the present invention performs data denoising processing on the current data list, it performs smoothing filtering processing on the time series data based on a filter algorithm (such as a Kalman filter algorithm, an enhanced Kalman filter algorithm, etc.), and performs mutation noise elimination processing on the data by averaging, sliding average or weighted average.

[0080] (III) Model training module 3:

[0081] The model training module 3 of the embodiment of the present invention is used to receive a first training configuration input by a user; and perform corresponding model training processing according to the first training configuration, the model database 7 and the large model library 6.

[0082] Here, the first training configuration includes a second large model identifier, a first training round threshold, a first split ratio, a first batch size, a first optimizer, a first initial learning rate, a first learning rate decay factor, a first loss function, a first regularization term type, a first model validation function, a first model evaluation function set, and a first evaluation parameter set; wherein:

[0083] The first training round threshold is the maximum training round;

[0084] The first split ratio is the ratio of the total number of records in the training set, validation set, and test set, for example, 8:1:1;

[0085] The first batch size is the number of samples in a single batch;

[0086] The first optimizer includes an SGD optimizer, an ADAM optimizer, and an RMSprop optimizer;

[0087] The first initial learning rate is the optimizer learning rate used by the optimizer in the first round;

[0088] The first learning rate attenuation factor is the attenuation factor of the optimizer learning rate in the previous and next two rounds;

[0089] The first loss function is a conventional loss function specified by the user, such as L1 loss function, L2 loss function, cross entropy loss function, etc.;

[0090] The first regularization term type includes L1 regularization term and L2 regularization term;

[0091] The first model validation function is an accuracy calculation function;

[0092] The first model evaluation function set consists of four types of evaluation functions, namely accuracy calculation function, precision calculation function, recall calculation function and F1 score calculation function;

[0093] The first evaluation parameter set consists of four types of evaluation value ranges, namely accuracy range, precision range, recall range and F1 score range.

[0094] In another specific implementation of the embodiment of the present invention, the model training module 3 is specifically used to perform corresponding model training processing according to the first training configuration, the model database 7 and the large model library 6:

[0095] Step B1, extracting from the first training configuration the corresponding second largest model identifier, the first training round threshold, the first segmentation ratio, the first batch size, the first optimizer, the first initial learning rate, the first learning rate decay factor, the first loss function, the first regularization term type, the first model validation function, the first model evaluation function set and the first evaluation parameter set;

[0096] Step B2, identifying the type of the first regularization term; if the first regularization term type is an L1 regularization term, the first model loss function is formed by adding the first loss function and an L1 regularization loss term; if the first regularization term type is an L2 regularization term, the first model loss function is formed by adding the first loss function and an L2 regularization loss term;

[0097] Specifically:

[0098] When the first regularization term type is L1 regularization term:

[0099]

[0100] When the first regularization term type is L2 regularization term:

[0101]

[0102] Among them, L M is the loss function of the first model; L A is the first loss function; P is the set of prediction-label pairs; {θ k} is the model parameter set of the current large model, θ k is the kth model parameter, 1≤k≤K, K is the total number of model parameters of the current large model; λ is the preset regularization loss term coefficient;

[0103] Here, the embodiment of the present invention adds a regularized loss term (L1 regularized loss term, L2 regularized loss term) on the basis of the first loss function specified by the user to construct the first model loss function used in the training process; the purpose of doing so is to restrict the model parameters through the regularized loss term, thereby reducing the complexity of training and preventing overfitting;

[0104] Step B3, taking the first large model whose first large model identifier matches the second large model identifier in the large model library 6 as the corresponding current large model; and taking the first model data set corresponding to the current large model in the model database 7 as the corresponding current model data set; and randomly dividing the current model data set into three sub-data sets based on the first division ratio, namely, the first training set, the first verification set and the first test set; and initializing the first round counter to 0; and taking the first initial learning rate as the corresponding current learning rate;

[0105] The first training set, the first validation set, and the first test set are all composed of a plurality of first data records, and the ratio of the total number of records of the first training set, the first validation set, and the first test set satisfies a first segmentation ratio;

[0106] Step B4, batch-dividing the first training set based on the first batch size to obtain multiple corresponding first batch data sets; and recording the total number of records in the first batch data set as the corresponding current iteration threshold; and taking the first first batch data set as the corresponding current batch data set; and initializing the first iteration counter to 0;

[0107] The first batch of data sets consists of the first data records of the first batch size;

[0108] Here, the batch partitioning mode of the first training set in the embodiment of the present invention can be customized based on application requirements; a common partitioning mode is a sliding slice partitioning mode, that is, the first training set is subjected to sliding slices with the first batch size as the slice length and a preset sliding step length, and each training set slice obtained thereby is a first batch data set;

[0109] Step B5, record the first model input tensor and the first model label tensor of each first data record of the current batch data set as the corresponding current input tensor and current label tensor, and input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding first prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding first prediction-label pair set from all the obtained first prediction-label pairs; and set the learning rate parameter of the first optimizer based on the current learning rate; and bring the first prediction-label pair set into the first model loss function L M , and based on the first optimizer towards making the first model loss function LM A round of model parameter modulation is performed on the current large model in the direction of reaching the minimum value;

[0110] Here, the parameter modulation method of each optimizer is different. The parameter modulation methods of the commonly used SGD optimizer, ADAM optimizer, and RMSprop optimizer are all public technical solutions, which can be obtained by consulting public literature and will not be further described here;

[0111] Step B6, add 1 to the first iteration counter; then identify whether the first iteration counter is less than the current iteration threshold; if so, take the next first batch data set as the new current batch data set and return to step B5; if not, go to step B7;

[0112] Step B7, record the first model input tensor and the first model label tensor of each first data record of the first validation set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding third prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding third prediction-label pair set from all the obtained third prediction-label pairs; and bring the third prediction-label pair set into the first model loss function L M Calculate the loss value to obtain the corresponding first-round loss value; and calculate the corresponding accuracy rate according to the third prediction-label pair set and the first model verification function to obtain the corresponding first-round accuracy rate; and use the model parameters of the current large model at the current moment as the corresponding first-round model parameters; and form a corresponding first-round record with the first-round loss value, the first-round accuracy rate and the first-round model parameters obtained this time;

[0113] Here, when the embodiment of the present invention performs the corresponding accuracy calculation according to the third prediction-label pair set and the first model verification function, a corresponding confusion matrix A1 is first constructed based on the third prediction-label pair set, and each matrix unit of the confusion matrix A1 corresponds to a third prediction-label pair; then the prediction purpose of the current large model is judged;

[0114] If the current large model is used for numerical prediction, the Monte Carlo method is used to calculate the corresponding credible interval according to the predicted value and label value of each matrix unit to obtain a credible prediction value range, and identify whether the predicted value of each matrix unit meets the corresponding credible prediction value range. If it meets the range, the true / false state corresponding to the current matrix unit is set to true; if it does not meet the range, the true / false state corresponding to the current matrix unit is set to false. In this way, a confusion matrix B1 with true / false matrix unit values ​​can be reconstructed. Based on the confusion matrix B1, the calculation factors required for the conventional accuracy calculation function can be counted to obtain the corresponding first-round accuracy.

[0115] If the current large model is used for classification prediction, then the predicted value and label value of each matrix unit of the confusion matrix A1 is a pair of prediction type and label type. It is directly identified whether the predicted value of each matrix unit matches the corresponding label value. If it matches, the true / false state corresponding to the current matrix unit is set to true; if it does not match, the true / false state corresponding to the current matrix unit is set to false. In this way, a confusion matrix C1 with true / false matrix unit values ​​can be reconstructed. Based on the confusion matrix C1, the calculation factors required by the conventional accuracy calculation function can be counted to obtain the corresponding first-round accuracy.

[0116] Step B8, add 1 to the first round counter; then use the current learning rate as the corresponding old learning rate; and reset the current learning rate according to the first learning rate decay factor, the first round counter and the old learning rate; and identify whether the first round counter is less than the first training round threshold; if so, use the first first batch data set as the new current batch data set, reset the first iteration counter to 0, and return to step B5; if not, go to step B9;

[0117] Here, the calculation formula for resetting the learning rate is: now =a old ×d ep ,

[0118] Among them, a now is the current learning rate after reset, a old is the old learning rate, d is the first learning rate decay factor, and ep is the first round counter;

[0119] Step B9, normalizing the first-round loss values ​​and first-round accuracy rates of all first-round records respectively; and calculating the corresponding weighted sum coefficient based on the normalized loss values ​​and normalized accuracy rates corresponding to each first-round record; and taking the first-round model parameters of the first-round record with the largest weighted sum coefficient as the corresponding current optimal model parameters; and resetting the model parameters of the current large model based on the current optimal model parameters;

[0120] Here, the weighted sum coefficient is calculated as:

[0121] Weighted sum coefficient = w1×normalized accuracy + w2×normalized loss value,

[0122] Among them, the normalized accuracy>0, the normalized loss value>0, w1 and w2 are two preset weighting coefficients and w1>0>w2;

[0123] Step B10, record the first model input tensor and the first model label tensor of each first data record of the first test set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding fourth prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding fourth prediction-label pair set from all the obtained fourth prediction-label pairs; and perform corresponding accuracy, precision, recall and F1 score calculations according to the accuracy calculation function, precision calculation function, recall calculation function and F1 score calculation function of the fourth prediction-label pair set and the first model evaluation function set to obtain the corresponding first accuracy, first precision, first recall and first F1 score;

[0124] Here, when the embodiment of the present invention performs corresponding accuracy, precision, recall and F1 score calculations according to the accuracy calculation function, precision calculation function, recall calculation function and F1 score calculation function of the fourth prediction-label pair set and the first model evaluation function set, a corresponding confusion matrix A2 is first constructed based on the fourth prediction-label pair set, and each matrix unit of the confusion matrix A2 corresponds to a fourth prediction-label pair; then the prediction purpose of the current large model is judged;

[0125] If the current large model is used for numerical prediction, the Monte Carlo method is used to calculate the corresponding credible interval according to the predicted value and label value of each matrix unit to obtain a credible prediction value range, and identify whether the predicted value of each matrix unit meets the corresponding credible prediction value range. If it meets, the true / false state corresponding to the current matrix unit is set to true, and if it does not meet, the true / false state corresponding to the current matrix unit is set to false. In this way, a confusion matrix B2 with true / false matrix unit values ​​can be reconstructed. Based on the confusion matrix B2, the calculation factors required for the conventional accuracy calculation function, precision calculation function, and recall calculation function can be statistically calculated to obtain the corresponding first accuracy, first precision, and first recall. Then, the obtained first accuracy, first precision, and first recall are brought into the conventional F1 score calculation function to obtain the corresponding first F1 score.

[0126] If the current large model is used for classification prediction, then the predicted value and label value of each matrix unit of the confusion matrix A2 is a pair of prediction type and label type. It is directly identified whether the predicted value of each matrix unit matches the corresponding label value. If it matches, the true / false state corresponding to the current matrix unit is set to true; if it does not match, the true / false state corresponding to the current matrix unit is set to false. In this way, a confusion matrix C2 with true / false matrix unit values ​​can be reconstructed. Based on the confusion matrix C2, the calculation factors required for the conventional accuracy calculation function, precision calculation function, and recall calculation function can be statistically calculated to obtain the corresponding first accuracy, first precision, and first recall rate. Then, the obtained first accuracy, first precision, and first recall rate are brought into the conventional F1 score calculation function to obtain the corresponding first F1 score.

[0127] Step B11, and identify whether the obtained first accuracy rate, first precision, first recall rate and first F1 score all meet the corresponding accuracy rate range, precision range, recall rate range and F1 score range in the first evaluation parameter set; if the first accuracy rate does not meet the corresponding accuracy rate range, or the first precision does not meet the corresponding precision range, or the first recall rate does not meet the corresponding recall rate range, or the first F1 score does not meet the corresponding F1 score range, then reduce the first round counter by 1, and use the first first batch data set as the new current batch data set, and reset the first iteration counter to 0, and return to step B5; if the first accuracy rate, the first precision, the first recall rate and the first F1 score all meet the corresponding accuracy rate range, precision range, recall rate range and F1 score range, then stop model training and solidify the model parameters of the current large model.

[0128] (IV) Model monitoring module 4

[0129] The model monitoring module 4 of the embodiment of the present invention is used to receive a first deployment configuration input by a user; and perform corresponding large model application instance deployment processing according to the first deployment configuration and the large model library 6 to obtain a corresponding first model instance; and create a corresponding first monitoring data list for the first model instance in the monitoring database 8; and perform data collection on the input / output data of the first model instance and the corresponding real observation data and update the corresponding first monitoring data list based on the collection results; and periodically perform model data extraction according to the first monitoring data list to obtain a corresponding first data packet and send it to the data collection module 2; and periodically perform model performance evaluation according to the first monitoring data list to obtain a corresponding first evaluation result and feedback to the user.

[0130] The first deployment configuration includes a third model identifier, a first running host address, first host software resource configuration parameters, and first host hardware resource configuration parameters.

[0131] Here, the running host mentioned in the embodiment of the present invention can be a local or remote virtual host / server / server / system / platform, or a computing unit of a network; the first running host address is a device address of a running host, which can be a local or remote MAC address, a local or remote IP address, a local or remote IP+port address, or a network address of a network computing unit. The first host software resource configuration parameters of the embodiment of the present invention are software resource configuration parameters that support the creation and operation of large model instances on the running host, and should at least include operating system configuration information, network configuration information, large model compilation / operation / storage environment configuration information, etc. The first host hardware resource configuration parameters of the embodiment of the present invention are hardware resource configuration parameters that support the creation and operation of large model instances on the running host, and should at least include memory hardware configuration information, network hardware configuration information, storage hardware configuration information, CPU / GPU resource configuration information, etc.

[0132] In another specific implementation method of the embodiment of the present invention, the model monitoring module 4 is specifically used to extract the corresponding third large model identifier, the first running host address, the first host software resource configuration parameters and the first host hardware resource configuration parameters from the first deployment configuration when the corresponding large model application instance deployment processing is performed according to the first deployment configuration and the large model library 6 to obtain the corresponding first model instance; and use the first large model that matches the first large model identifier and the third large model identifier in the large model library 6 as the corresponding current large model; and use the running host corresponding to the first running host address as the corresponding current host; and perform corresponding software and hardware resource configuration processing on the large model running environment of the current host based on the first host software resource configuration parameters and the first host hardware resource configuration parameters; and after the software and hardware resources are successfully configured, create a model instance of the current large model on the large model running environment of the current host and record it as the corresponding first model instance.

[0133] In another specific implementation of the embodiment of the present invention, the model monitoring module 4 is specifically used to collect data on the input / output data of the first model instance and the corresponding real observation data and update the corresponding first monitoring data list based on the collection result, and collect data on the model input tensor and the corresponding model output tensor of the first model instance at any time t, which are recorded as the corresponding input tensor x t and the output tensor y t ; and through the preset observation query interface, the input tensor x at any time t t The corresponding real observation data tensor is queried to obtain the corresponding real tensor z t ; and the input tensor x at each time t t , output tensor y t and the real tensor z tThe corresponding second model input tensor, the first model output tensor and the first real observation tensor form a corresponding first monitoring data record which is added to the corresponding first monitoring data list.

[0134] Here, the observation query interface of the embodiment of the present invention is a pre-set data query interface, and the data query interface can query the real observation values ​​of all data objects predicted by the first model instance at any time t.

[0135] In another specific implementation of the embodiment of the present invention, the model monitoring module 4 is specifically used to extract the first monitoring data records added in the first monitoring data list within the most recent first time period to form a corresponding first record set when periodically extracting model data according to the first monitoring data list to obtain the corresponding first data packet and sending it to the data acquisition module 2; and use the first large model corresponding to the first monitoring data list as the corresponding current large model; and use the first data set identifier corresponding to the current large model as the corresponding second data set identifier; and use the second model input tensor and the first real observation tensor of each first monitoring data record in the first record set as the corresponding first input data and the first label data to form a corresponding first acquisition data record; and all the first acquisition data records obtained this time form a corresponding first acquisition data list; and the second data set identifier and the first acquisition data list obtained this time form a corresponding first data packet to send to the data acquisition module 2.

[0136] Here, the first duration is a preset time length.

[0137] In another specific implementation of the embodiment of the present invention, the model monitoring module 4 is specifically configured to periodically perform a model performance evaluation according to the first monitoring data list to obtain a corresponding first evaluation result and provide feedback to the user:

[0138] Step C1, every second preset time period, the current time is taken as the corresponding third time, and the time obtained by subtracting the second time period from the third time is recorded as the corresponding second time, and the time obtained by subtracting the second time period from the second time is recorded as the corresponding first time; and the time period from the first time to the second time is recorded as the corresponding first time period; and the time period from the second time to the third time is recorded as the corresponding second time period;

[0139] Here, the second duration is a preset time length; the second duration>the first duration;

[0140] Step C2, extracting the first monitoring data record added in the first time period from the first monitoring data list to form a corresponding second record set; and extracting the first monitoring data record added in the second time period from the first monitoring data list to form a corresponding third record set;

[0141] Step C3, a corresponding fifth prediction-label pair is formed by the first model output tensor and the first true observation tensor of each first monitoring data record of the second record set, and a corresponding fifth prediction-label pair set is formed by all the obtained fifth prediction-label pairs; and a corresponding sixth prediction-label pair is formed by the first model output tensor and the first true observation tensor of each first monitoring data record of the third record set, and a corresponding sixth prediction-label pair set is formed by all the obtained sixth prediction-label pairs;

[0142] Step C4, and calculate the corresponding prediction error standard deviation according to the fifth and sixth prediction-label pair sets to obtain the corresponding first and second standard deviations; and calculate the corresponding first standard deviation change rate according to the first and second standard deviations; and identify whether the first standard deviation change rate meets the preset first change rate range; if so, set the corresponding first recognition state to no abnormality; if not, set the corresponding first recognition state to abnormality;

[0143] The first identification state includes no abnormality and abnormality; the first change rate range is a preset change rate range;

[0144] For example, it is known that the fifth prediction-label pair set includes n1 fifth prediction-label pairs (the first model output tensor ypr e ,i, the first real observation tensor ytag,i), 1≤i≤n1, n1 is a positive integer; the sixth prediction-label pair set includes n2 sixth prediction-label pairs (the first model output tensor ypr e ,j, the first real observation tensor ytag,j), 1≤j≤n2, n2 is a positive integer;

[0145] Then, the first standard deviation σ1, the second standard deviation σ2, and the first standard deviation change rate ρ1 are:

[0146]

[0147]

[0148] ρ1=σ2 / σ1,

[0149] Assuming that the first change rate ranges from 0 to 1, if the first standard deviation change rate ρ1 is greater than 1, it means that the model prediction accuracy has decreased in the most recent second period, and the corresponding first recognition state is abnormal; if the first standard deviation change rate ρ1 is less than 1, it means that the model prediction accuracy has not decreased in the most recent second period, and the corresponding first recognition state is normal;

[0150] Step C5, when the first model instance corresponding to the first monitoring data list is a classification model, the corresponding KL divergence calculation is performed according to the fifth and sixth prediction-label pair sets to obtain the corresponding first and second KL divergences; and the corresponding first divergence change rate is calculated according to the first and second KL divergences; and whether the first divergence change rate satisfies a preset second change rate range is identified; if satisfied, the corresponding second identification state is set to no abnormality; if not satisfied, the corresponding second identification state is set to abnormality;

[0151] The second identification state includes no abnormality and abnormality; the second change rate range is a preset change rate range;

[0152] For example, the fifth and sixth prediction-label pair set examples are shown above, and the total number of classifications of the first model instance is known to be M. Then the overall KL divergence of the model in the first period, i.e., the first KL divergence d1, can be obtained from the fifth prediction-label pair set. The overall KL divergence of the model in the second period, i.e., the second KL divergence d2, can be obtained from the sixth prediction-label pair set. The corresponding first divergence change rate ρ2 can be obtained from the first and second KL divergences d1 and d2:

[0153]

[0154]

[0155] ρ2=d2 / d1 ;

[0156] Among them, P(y pre,i )、Q(y tag,i ) is the first model output tensor y of the i-th fifth prediction-label pair in the fifth prediction-label pair set pre,i and the first real observation tensor y tag,i M types of probability distribution vectors; P(y pre,j )、Q(y tag,j ) is the first model output tensor y of the jth sixth prediction-label pair in the sixth prediction-label pair set pre,j and the first real observation tensor y tag,j M types of probability distribution vectors; D KL () is the KL divergence function;

[0157] Assuming that the second change rate range is between 0 and 1, then if the first divergence change rate ρ2 is greater than 1, it means that the model drift is obvious in the most recent second period, the prediction performance is reduced, and the corresponding second recognition state is abnormal; if the first divergence change rate ρ2 is less than 1, it means that the model has not drifted in the most recent second period, the prediction performance has not decreased, and the corresponding second recognition state is normal;

[0158] Step C6, when the first model instance corresponding to the first monitoring data list is a regression calculation model, the corresponding average confidence interval width is calculated according to the fifth and sixth prediction-label pair sets based on the Monte Carlo method to obtain the corresponding first and second average confidence interval widths; and the corresponding first interval width change rate is calculated according to the first and second average confidence interval widths; and whether the first interval width change rate satisfies a preset third change rate range is identified; if satisfied, the corresponding third identification state is set to no abnormality; if not satisfied, the corresponding third identification state is set to abnormality;

[0159] The third identification state includes no abnormality and abnormality; the third change rate range is a preset change rate range;

[0160] The first and second confidence intervals here are average confidence intervals;

[0161] For example, the fifth and sixth prediction-label pair set examples are shown above. Then, the first standard deviation σ1 of the model in the first period can be obtained from the fifth prediction-label pair set, and then the average confidence interval width of the model in the first period, i.e., the first average confidence interval width L1, can be calculated. The second standard deviation σ2 of the model in the second period can be obtained from the sixth prediction-label pair set, and then the average confidence interval width of the model in the second period, i.e., the first average confidence interval width L2, can be calculated. The first and second average confidence interval widths L1 and L2 can be used to obtain the corresponding first interval width change rate ρ3:

[0162]

[0163]

[0164] ρ3=L2 / L1;

[0165] Where α is the Z-Score parameter at a given confidence level (e.g. 95%);

[0166] Assuming that the third change rate range is between 0 and 1, then if the first interval width change rate ρ3 is greater than 1, it means that the confidence interval width of the model has become larger and the prediction performance has decreased in the most recent second period, and the corresponding third recognition state is abnormal; if the first interval width change rate ρ3 is less than 1, it means that the confidence interval width of the model has become narrower and the prediction performance has not decreased in the most recent second period, and the corresponding third recognition state is no abnormality;

[0167] Step C7, when the first model instance corresponding to the first monitoring data list is a classification model, the first evaluation result corresponding to the third time, the first recognition state and the second recognition state is fed back to the user; and when the first model instance corresponding to the first monitoring data list is a regression calculation model, the first evaluation result corresponding to the third time, the first recognition state and the third recognition state is fed back to the user.

[0168] Here, if the user finds that the first, second or third recognition status is abnormal when receiving the first evaluation result, he can send the training configuration to the model training module 3 to perform secondary training on the prototype large model of the first model instance in the large model library 6, and delete the running model instance on the running host of the first model instance after the training is completed, and create a new first model instance on the large model running environment of the current host based on the latest model parameters.

[0169] (V) Basic model library 5:

[0170] The basic model library 5 of the embodiment of the present invention includes multiple first basic models; each first basic model is a type of machine learning model or deep learning model.

[0171] (VI) Large Model Library 6:

[0172] The large model library 6 of the embodiment of the present invention includes multiple first large models; each first large model corresponds to a first large model identifier.

[0173] (VII) Model database 7:

[0174] The model database 7 of the embodiment of the present invention is used to store multiple first model data sets; the first model data sets correspond one-to-one to the first large model; each first model data set corresponds to a first data set identifier; the first model data set includes multiple first data records; the first data record includes a first model input tensor and a first model label tensor.

[0175] (VIII) Monitoring database 8:

[0176] The monitoring database 8 of the embodiment of the present invention includes multiple first monitoring data lists; the first monitoring data lists include multiple first monitoring data records; the first monitoring data record includes the second model input tensor, the first model output tensor and the first real observation tensor.

[0177] It should be noted that the division of the various system components / modules / libraries of the above system can be regarded as a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these system components / modules / libraries can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the model design module can be a separately established processing element, or it can be integrated in a chip of the above system for implementation. In addition, it can also be stored in the memory of the above system in the form of program code, and called and executed by a processing element of the above system. The implementation of other system components / modules / libraries is similar. In addition, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, the processing steps of each module of the aforementioned system can be completed by the hardware integrated logic circuit in the processor element or the instructions in the form of software.

[0178] For example, the system components / modules / libraries of the above system may be one or more integrated circuits configured to implement the functions of the corresponding components / modules / libraries, such as: one or more application specific integrated circuits (ASIC), or one or more digital signal processors (DSP), or one or more field programmable gate arrays (FPGA). For another example, when a system component / module / library of the above system is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0179] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the above system are generated. The above computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) methods. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, a hard disk, a tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0180] In summary, an embodiment of the present invention provides a large model training system; as can be seen from the above content, the system includes: a model design module, a data acquisition module, a model training module, a model monitoring module, a basic model library, a large model library, a model database and a monitoring database; wherein the model design module is used to perform corresponding large model assembly and model data set initialization according to the basic model library, the large model library and the model database; the data acquisition module is used to perform data update processing on the model database according to the data packet input by the user or the model monitoring module; the model training module is used to perform corresponding model training processing according to the training configuration, the model database and the large model library input by the user; the model monitoring module is used to perform corresponding large model application instance deployment according to the deployment configuration and the large model library input by the user, and create a corresponding monitoring data list for the model instance in the monitoring database, and collect data for the input / output data of the model instance and the corresponding real observation data and update the corresponding monitoring data list based on the collection results, and regularly extract model data according to the monitoring data list to obtain the corresponding data packet and send it to the data acquisition module, and regularly perform model performance evaluation according to the monitoring data list to obtain the corresponding evaluation results and feedback to the user. The embodiments of the present invention can assemble and customize a large model with specific functions based on conventional models, can fully train the customized model before it is put into use for the first time, can update the model data set used for training based on user input data and monitored model application data, can track and monitor the performance of the customized model in real time after it is put into use, and promptly inform relevant staff when the model performance is abnormal so as to start a new round of model training as soon as possible; the system solution provided by the embodiments of the present invention not only reduces the complexity of model customization, improves the real-time nature of training data and model training efficiency, but also improves the real-time tracking of model performance.

[0181] The professionals should further realize that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0182] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0183] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A large model training system, characterized in that: The system includes: a model design module, a data acquisition module, a model training module, a model monitoring module, a basic model library, a large model library, a model database and a monitoring database; The model design module is connected to the basic model library, the large model library and the model database respectively; the data acquisition module is connected to the model database; the model training module is connected to the model database and the large model library respectively; the model monitoring module is connected to the data acquisition module, the large model library and the monitoring database respectively; The model design module is used to assemble the corresponding large model and initialize the model data set according to the basic model library, the large model library and the model database; The data acquisition module is used to receive a first data packet input by a user or the model monitoring module; and perform data update processing on the model database according to the first data packet; The model training module is used to receive a first training configuration input by a user; and perform corresponding model training processing according to the first training configuration, the model database and the large model library; The model monitoring module is used to receive a first deployment configuration input by a user; and perform corresponding large model application instance deployment processing according to the first deployment configuration and the large model library to obtain a corresponding first model instance; and create a corresponding first monitoring data list for the first model instance in the monitoring database; and perform data collection on the input / output data of the first model instance and the corresponding real observation data and update the corresponding first monitoring data list based on the collection results; and periodically perform model data extraction according to the first monitoring data list to obtain the corresponding first data packet and send it to the data collection module; and periodically perform model performance evaluation according to the first monitoring data list to obtain the corresponding first evaluation result and feedback it to the user; The basic model library includes a plurality of first basic models; each of the first basic models is a type of machine learning model or a deep learning model; The large model library includes a plurality of first large models; each of the first large models corresponds to a first large model identifier; The model database is used to store multiple first model data sets; the first model data sets correspond to the first large model one by one; each of the first model data sets corresponds to a first data set identifier; the first model data set includes multiple first data records; the first data record includes a first model input tensor and a first model label tensor; The monitoring database includes a plurality of the first monitoring data lists; the first monitoring data lists include a plurality of first monitoring data records; the first monitoring data records include a second model input tensor, a first model output tensor and a first real observation tensor.

2. The large model training system according to claim 1, characterized in that: The model design module is specifically used for assembling the corresponding large model and initializing the model data set according to the basic model library, the large model library and the model database. Providing a user with a visual model assembly interface having a draggable toolbar, a draggable design window, and a model submission button; the draggable toolbar includes a plurality of draggable components, and the draggable components correspond one to one with the first basic model of the basic model library; When the user drags any of the draggable components into the drag-and-drop design window, a corresponding basic model implementation is created for the first basic model corresponding to the currently dragged draggable component; when the user selects any of the draggable components from the drag-and-drop design window, a corresponding component property configuration toolbar is provided for the user to configure the basic model implementation corresponding to the currently selected draggable component; and in the drag-and-drop design window, a corresponding component connection tool is provided for each of the draggable components to create a connection relationship between the current component and other components; When the user clicks the model submission button, a new model is constructed based on the configuration states and connection relationships of all the basic model embodiments and recorded as the corresponding current constructed model; and based on the preset key layer definition rules, a batch normalization layer is added to the output end of each key layer of the current constructed model; Adding the currently constructed model after adding the batch normalization layer as a new first large model to the large model library; and assigning a unique first large model identifier to the new first large model; and initializing an empty data set on the model database for the new first large model as the corresponding first model data set; And a unique first data set identifier is allocated to the first model data set initialized this time.

3. The large model training system according to claim 1, characterized in that: The first data packet is composed of a second data set identifier and a first collected data list; the first collected data list includes a plurality of first collected data records; the first collected data record includes first input data and first label data; The first training configuration includes a second large model identifier, a first training round threshold, a first split ratio, a first batch size, a first optimizer, a first initial learning rate, a first learning rate decay factor, a first loss function, a first regularization term type, a first model validation function, a first model evaluation function set, and a first evaluation parameter set; The first training round threshold is the maximum training round; The first split ratio is the ratio of the total number of records in the training set, the validation set, and the test set; The first batch size is the number of samples in a single batch; The first optimizer includes an SGD optimizer, an ADAM optimizer and an RMSprop optimizer; The first initial learning rate is an optimizer learning rate used by the optimizer in the first round; The first learning rate attenuation factor is the attenuation factor of the optimizer learning rate in the previous and next two rounds; The first regularization term type includes an L1 regularization term and an L2 regularization term; The first model verification function is an accuracy calculation function; The first model evaluation function set consists of four types of evaluation functions, namely, accuracy calculation function, precision calculation function, recall calculation function and F1 score calculation function; The first evaluation parameter set consists of four types of evaluation value ranges, namely, accuracy range, precision range, recall range and F1 score range; The first deployment configuration includes a third large model identifier, a first running host address, a first host software resource configuration parameter, and a first host hardware resource configuration parameter.

4. The large model training system according to claim 3, characterized in that: The data acquisition module is specifically configured to extract the corresponding second data set identifier and the first acquired data list from the first data packet as the corresponding current data set identifier and the current data list when the model database is updated according to the first data packet; and performing data cleaning, data standardization and data denoising processing on the current data list; and taking the first model data set whose first data set identifier matches the current data set identifier in the model database as the corresponding current model data set; And each of the first collected data records in the current data list is taken as the corresponding current record, and the first input data and the first label data of the current record are converted into corresponding data tensors to obtain the corresponding first model input tensor and the first model label tensor to form a corresponding first data record and add it to the current model data set.

5. The large model training system according to claim 3, characterized in that: The model training module is specifically used for performing corresponding model training processing according to the first training configuration, the model database and the large model library. Step 501, extracting the corresponding second large model identifier, the first training round threshold, the first segmentation ratio, the first batch size, the first optimizer, the first initial learning rate, the first learning rate decay factor, the first loss function, the first regularization term type, the first model validation function, the first model evaluation function set and the first evaluation parameter set from the first training configuration; Step 502, identifying the type of the first regularization term; if the first regularization term type is an L1 regularization term, the first loss function and an L1 regularization loss term are added to form the corresponding first model loss function; if the first regularization term type is an L2 regularization term, the first loss function and an L2 regularization loss term are added to form the corresponding first model loss function; Specifically: When the first regularization term type is L1 regularization term: When the first regularization term type is L2 regularization term: Among them, L M is the first model loss function; L A is the first loss function; P is the set of prediction-label pairs; {θ k } is the model parameter set of the current large model, θ k is the kth model parameter, 1≤k≤K, K is the total number of model parameters of the current large model; λ is the preset regularization loss term coefficient; Step 503: take the first large model whose first large model identifier matches the second large model identifier in the large model library as the corresponding current large model; and take the first model data set corresponding to the current large model in the model database as the corresponding current model data set; and randomly divide the current model data set into three sub-data sets based on the first division ratio, namely, a first training set, a first validation set and a first test set; and initialize the first round counter to 0; and take the first initial learning rate as the corresponding current learning rate; The first training set, the first validation set, and the first test set are all composed of a plurality of the first data records, and the ratio of the total number of records in the first training set, the first validation set, and the first test set satisfies the first segmentation ratio; Step 504: divide the first training set into batches based on the first batch size to obtain a plurality of corresponding first batch data sets; record the total number of records in the first batch data sets as the corresponding current iteration threshold; and use the first of the first batch data sets as the corresponding current batch data set; and initialize the first iteration counter to 0; wherein the first batch of data sets consists of the first data records of the first batch size; Step 505: record the first model input tensor and the first model label tensor of each of the first data records of the current batch data set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding first prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding first prediction-label pair set from all the obtained first prediction-label pairs; and set the learning rate parameter of the first optimizer based on the current learning rate; and bring the first prediction-label pair set into the first model loss function L M , and based on the first optimizer moving towards making the first model loss function L M Perform a round of model parameter modulation on the current large model in the direction of reaching the minimum value; Step 506, adding 1 to the first iteration counter; then identifying whether the first iteration counter is less than the current iteration threshold; if so, taking the next first batch data set as the new current batch data set and returning to step 505; if not, going to step 507; Step 507: record the first model input tensor and the first model label tensor of each of the first data records of the first validation set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding third prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding third prediction-label pair set from all the obtained third prediction-label pairs; and bring the third prediction-label pair set into the first model loss function L M Calculate the loss value to obtain the corresponding first-round loss value; and calculate the corresponding accuracy rate according to the third prediction-label pair set and the first model verification function to obtain the corresponding first-round accuracy rate; and use the model parameters of the current large model at the current moment as the corresponding first-round model parameters; and form a corresponding first-round record with the first-round loss value, the first-round accuracy rate and the first-round model parameters obtained this time; Step 508: add 1 to the first round counter; then use the current learning rate as the corresponding old learning rate; and reset the current learning rate according to the first learning rate attenuation factor, the first round counter and the old learning rate. now =a old ×d ep , a now is the current learning rate after reset, a old is the old learning rate, d is the first learning rate decay factor, and ep is the first round counter; and whether the first round counter is less than the first training round threshold is identified; if so, the first first batch data set is used as the new current batch data set, and the first iteration counter is reset to 0, and the process returns to step 505; if not, the process goes to step 509; Step 509, normalize the first-round loss values ​​and the first-round accuracy of all the first-round records obtained respectively; and calculate the corresponding weighted sum coefficient = w1×normalized accuracy+w2×normalized loss value based on the normalized loss value and normalized accuracy corresponding to each first-round record, the normalized accuracy>0, the normalized loss value>0, w1 and w2 are two preset weighting coefficients and w1>0>w2; and use the first-round model parameters of the first-round record with the largest weighted sum coefficient as the corresponding current optimal model parameters; and reset the model parameters of the current large model based on the current optimal model parameters; Step 510, record the first model input tensor and the first model label tensor of each of the first data records of the first test set as the corresponding current input tensor and current label tensor, input the current input tensor into the current large model for model prediction to obtain the corresponding current prediction tensor, and form a corresponding fourth prediction-label pair from the current prediction tensor and the corresponding current label tensor; and form a corresponding fourth prediction-label pair set from all the obtained fourth prediction-label pairs; and perform corresponding accuracy, precision, recall and F1 score calculations according to the fourth prediction-label pair set and the accuracy calculation function, the precision calculation function, the recall calculation function and the F1 score calculation function of the first model evaluation function set to obtain the corresponding first accuracy, first precision, first recall and first F1 score; Step 511, and identify whether the first accuracy rate, the first precision, the first recall rate and the first F1 score obtained all meet the corresponding accuracy rate range, the precision range, the recall rate range and the F1 score range in the first evaluation parameter set; if the first accuracy rate does not meet the corresponding accuracy rate range, or the first precision does not meet the corresponding precision range, or the first recall rate does not meet the corresponding recall rate range, or the first F1 score does not meet the corresponding F1 score range, then reduce the first round counter by 1, and use the first first batch data set as the new current batch data set, and reset the first iteration counter to 0, and return to step 505; if the first accuracy rate, the first precision, the first recall rate and the first F1 score all meet the corresponding accuracy rate range, the precision range, the recall rate range and the F1 score range, then stop model training and solidify the model parameters of the current large model.

6. The large model training system according to claim 3, characterized in that: The model monitoring module is specifically used for extracting the corresponding third large model identifier, the first running host address, the first host software resource configuration parameters and the first host hardware resource configuration parameters from the first deployment configuration when the corresponding large model application instance deployment processing is performed according to the first deployment configuration and the large model library to obtain the corresponding first model instance; and taking the first large model in the large model library whose first large model identifier matches the third large model identifier as the corresponding current large model; and taking the running host corresponding to the first running host address as the corresponding current host; and performing corresponding software and hardware resource configuration processing on the large model running environment of the current host based on the first host software resource configuration parameter and the first host hardware resource configuration parameter; After the software and hardware resources are configured successfully, a model instance of the current large model is created on the large model running environment of the current host and recorded as the corresponding first model instance.

7. The large model training system according to claim 1, characterized in that: The model monitoring module is specifically used to collect data on the model input tensor and the corresponding model output tensor of the first model instance at any time t when collecting data on the input / output data of the first model instance and the corresponding real observation data and updating the corresponding first monitoring data list based on the collection result, and record it as the corresponding input tensor x t and the output tensor y t ; and through the preset observation query interface, the input tensor x at any time t t The corresponding real observation data tensor is queried to obtain the corresponding real tensor z t ; and the input tensor x at each time t t , the output tensor y t and the real tensor z t The corresponding second model input tensor, the first model output tensor and the first real observation tensor form a corresponding first monitoring data record which is added to the corresponding first monitoring data list.

8. The large model training system according to claim 3, characterized in that: The model monitoring module is specifically configured to extract the first monitoring data records added in the first monitoring data list within the most recent first time period to form a corresponding first record set at every preset first time period when the corresponding first data packet is obtained by periodically extracting model data according to the first monitoring data list and sending it to the data acquisition module; and taking the first large model corresponding to the first monitoring data list as the corresponding current large model; and using the first data set identifier corresponding to the current large model as the corresponding second data set identifier; and using the second model input tensor and the first real observation tensor of each of the first monitoring data records in the first record set as the corresponding first input data and the first label data to form a corresponding first collected data record; and forming a corresponding first collected data list from all the first collected data records obtained this time; The second data set identifier obtained this time and the first collected data list form a corresponding first data packet and send it to the data collection module.

9. The large model training system according to claim 3, characterized in that: The model monitoring module is specifically used for periodically performing model performance evaluation according to the first monitoring data list to obtain a corresponding first evaluation result and feeding back to the user. At every preset second time duration, the current time is taken as the corresponding third time, and the time obtained by subtracting the second time duration from the third time is recorded as the corresponding second time, and the time obtained by subtracting the second time duration from the second time is recorded as the corresponding first time; and the time period from the first time to the second time is recorded as the corresponding first time period; and the time period from the second time to the third time is recorded as the corresponding second time period; and extracting the first monitoring data record added in the first time period from the first monitoring data list to form a corresponding second record set; and extracting the first monitoring data record added in the second time period from the first monitoring data list to form a corresponding third record set; A corresponding fifth prediction-label pair is formed by the first model output tensor and the first real observation tensor of each of the first monitoring data records in the second record set, and a corresponding fifth prediction-label pair set is formed by all the obtained fifth prediction-label pairs; a corresponding sixth prediction-label pair is formed by the first model output tensor and the first real observation tensor of each of the first monitoring data records in the third record set, and a corresponding sixth prediction-label pair set is formed by all the obtained sixth prediction-label pairs; And according to the fifth and sixth prediction-label pair sets, the corresponding prediction error standard deviation is calculated to obtain the corresponding first and second standard deviations; and according to the first and second standard deviations, the corresponding first standard deviation change rate is calculated; and whether the first standard deviation change rate satisfies a preset first change rate range is identified; if satisfied, the corresponding first recognition state is set to no abnormality; if not satisfied, the corresponding first recognition state is set to abnormality; wherein the first recognition state includes no abnormality and abnormality; When the first model instance corresponding to the first monitoring data list is a classification model, the corresponding KL divergence calculation is performed according to the fifth and sixth prediction-label pair sets to obtain the corresponding first and second KL divergences; and the corresponding first divergence change rate is calculated according to the first and second KL divergences; and whether the first divergence change rate satisfies a preset second change rate range is identified; if so, the corresponding second recognition state is set to no abnormality; if not, the corresponding second recognition state is set to abnormality; wherein the second recognition state includes no abnormality and abnormality; When the first model instance corresponding to the first monitoring data list is a regression calculation model, the corresponding average confidence interval width is calculated according to the fifth and sixth prediction-label pair sets based on the Monte Carlo method to obtain the corresponding first and second average confidence interval widths; and the corresponding first interval width change rate is calculated according to the first and second average confidence interval widths; and whether the first interval width change rate satisfies a preset third change rate range is identified; if so, the corresponding third identification state is set to no abnormality; if not, the corresponding third identification state is set to abnormality; wherein the third identification state includes no abnormality and abnormality; And when the first model instance corresponding to the first monitoring data list is a classification model, the first evaluation result corresponding to the third time, the first recognition state and the second recognition state is fed back to the user; and when the first model instance corresponding to the first monitoring data list is a regression calculation model, the first evaluation result corresponding to the third time, the first recognition state and the third recognition state is fed back to the user.

Images