Parameter updating method and device based on low-rank decomposition, equipment and storage medium

Abstract

The invention relates to a parameter updating method based on low-rank decomposition, and aims to solve the problems of high consumption of computing resources and difficulty in balancing model performance and complexity when large-scale financial data is processed in the prior art. According to the method, the pre-training model is determined, the source data set is obtained, and the source data set is utilized to train the pre-training model to obtain the source model; fixing parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model; and obtaining a training data set, training the inserted data processing layer by using the training data set, updating parameters of the data processing layer in an incremental mode, and finally obtaining a target model. The method is suitable for processing a scene of a large-scale data set, the problems that in the model training process, computing resource consumption is too large, and training time is too long are solved, the practicability and response speed of the model are remarkably improved, and meanwhile the performance and adaptability of the model on specific tasks are improved.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to a parameter updating method, device, equipment and storage medium based on low-rank decomposition. Background Art

[0002] In the modern financial industry, with the rapid development of big data and artificial intelligence technologies, financial institutions are increasingly relying on data-driven decision-making models. These models are widely used in key business areas such as risk management, credit assessment, market forecasting, fraud detection, etc. However, with the increasing scale and complexity of financial data, how to effectively train and optimize these models, especially how to reduce the consumption of computing resources while ensuring model accuracy, has become a technical problem that needs to be solved urgently.

[0003] Traditional financial models usually rely on a large amount of historical data for training, and the complexity and number of parameters of these models are often very large. This not only increases the cost of training and deployment, but also places higher demands on computing resources. In addition, due to the diversity and dynamics of financial data, the demand for model generalization and real-time updates is also increasing.

[0004] In order to solve these problems, model compression and optimization techniques based on low-rank decomposition have gradually attracted attention in recent years. Low-rank decomposition technology can reduce the number of model parameters without significantly reducing model performance, thereby reducing computational complexity, and is suitable for deploying financial models in resource-limited environments. In addition, low-rank decomposition can also be combined with other optimization techniques (such as incremental updates and learning rate decay) to further improve the real-time performance and accuracy of the model.

[0005] In the financial field, typical applications of model optimization using low-rank decomposition techniques include:

[0006] Credit scoring model optimization: By performing low-rank decomposition on the credit scoring model, the computational burden of the model can be effectively reduced and the real-time scoring capability of the model can be improved, which is especially important when processing massive amounts of user data.

[0007] Fraud detection: By applying low-rank decomposition to the fraud detection model, we can reduce the resource consumption of the model in actual operation while ensuring high accuracy, so that it can work effectively in a high-concurrency transaction environment. Summary of the invention

[0008] The main purpose of the present invention is to provide a parameter updating method, device, equipment and storage medium based on low-rank decomposition, aiming to solve the technical problems faced by the prior art in processing large-scale financial data, namely, high consumption of computing resources and difficulty in balancing model performance and complexity.

[0009] To achieve the above object, the present invention provides a parameter updating method based on low rank decomposition, and the parameter updating method based on low rank decomposition comprises the following steps:

[0010] Determine a pre-trained model, obtain a source data set, and use the source data set to train the pre-trained model to obtain a source model;

[0011] Fixing the parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model;

[0012] A training data set is obtained, the inserted data processing layer is trained using the training data set, parameters of the data processing layer are updated in an incremental manner, and a target model is obtained.

[0013] In one embodiment, the updating of the parameters of the data processing layer in an incremental manner includes:

[0014] Decomposing the weight matrix of the data processing layer into two matrices whose ranks are lower than the rank of the original weight matrix, respectively represented as a first matrix and a second matrix;

[0015] The dimension of the first matrix is ​​the same as the number of rows of the pre-trained weight matrix of the source model, and the dimension of the second matrix is ​​the same as the number of columns of the pre-trained weight matrix of the source model;

[0016] Keep the pre-trained weight matrix of the source model unchanged, and update the parameters of the first matrix and the second matrix of the newly added data processing layer;

[0017] The weight matrix of the updated data processing layer is composed of the product of the first matrix and the second matrix and the sum of the pre-trained weight matrix of the source model to form the final weight matrix of the target model.

[0018] In one embodiment, obtaining a training data set includes:

[0019] Select a candidate training data set, and use word embedding technology to convert the labels in the candidate training data set and the source data set into vector representations, which are represented as candidate label vector representations and source label vector representations respectively;

[0020] Calculating the label similarity between the candidate label vector representation and the source label vector representation using cosine similarity;

[0021] If the label similarity is higher than a similarity threshold, the candidate training data set is used as a training data set for training the data processing layer.

[0022] In one embodiment, the updating of the parameters of the data processing layer in an incremental manner includes:

[0023] Set the initial learning rate and learning rate decay strategy to divide the training data set into multiple small batches, each of which contains a fixed preset number of samples;

[0024] Each time, a small batch of data is extracted from the training data set and forward propagation is performed to calculate the model's prediction results;

[0025] Calculate the loss function based on the predicted results and the actual labels, calculate the gradient of the loss function relative to the parameters of the data processing layer through backpropagation, and update the parameters using the current learning rate;

[0026] After each mini-batch of data is used to update the parameters of the data processing layer, the learning rate is adjusted according to the current learning rate decay strategy until all mini-batches of data are processed.

[0027] In one embodiment, after obtaining the target model, the method further includes:

[0028] Obtain a validation data set and determine evaluation indicators based on the task requirements of the target model;

[0029] Calculating the prediction result of the target model through the verification data set, and performing quantitative evaluation according to the selected evaluation index to generate a comprehensive evaluation result;

[0030] Analyze the comprehensive evaluation results and adjust the parameters of the target model.

[0031] In one embodiment, inserting a preset number of data processing layers into the source model includes:

[0032] Determine the type of insertion layer based on the requirements of the target task, and design the preset number and depth of the insertion layer according to the type of insertion layer;

[0033] Analyzing the layer structure of the source model to determine the position of the insertion layer in the source model;

[0034] Inserting a data processing layer into the source model according to the type of the insertion layer, the preset number and depth of the insertion layer, and the insertion position;

[0035] An initialization strategy is determined according to the type of the inserted layer, and all weights and biases of the inserted data processing layer are initialized using the initialization strategy.

[0036] In one embodiment, determining a pre-trained model, acquiring a source data set, and using the source data set to train the pre-trained model to obtain a source model includes:

[0037] Determine the pre-training model based on the requirements of the target task and select the source dataset that matches the characteristics of the target task;

[0038] Set model training parameters, including learning rate and number of training rounds, according to the complexity of the target task and the size of the source dataset;

[0039] Performing data enhancement processing on the source data set and performing standardization or normalization processing;

[0040] The pre-trained model is trained based on the processed source data set and the parameters of the model training to obtain the source model.

[0041] In one embodiment, to achieve the above object, the present invention further provides a parameter updating device based on low-rank decomposition, the parameter updating device based on low-rank decomposition comprising:

[0042] A model pre-training module determines a pre-training model, obtains a source data set, and uses the source data set to train the pre-training model to obtain a source model;

[0043] A parameter fixing module, fixing the parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model;

[0044] The parameter updating module obtains a training data set, uses the training data set to train the inserted data processing layer, and updates the parameters of the data processing layer in an incremental manner to obtain a target model.

[0045] In one embodiment, to achieve the above-mentioned purpose, the present invention also provides a parameter updating device based on low-rank decomposition, wherein the parameter updating device based on low-rank decomposition comprises a memory, a processor, and a parameter updating program based on low-rank decomposition stored on the memory and executable on the processor, wherein the parameter updating program based on low-rank decomposition implements the steps of the parameter updating method based on low-rank decomposition as described above when executed by the processor.

[0046] In one embodiment, to achieve the above-mentioned purpose, the present invention also provides a storage medium, on which a parameter update program based on low-rank decomposition is stored, and when the parameter update program based on low-rank decomposition is executed by a processor, the steps of the parameter update method based on low-rank decomposition as described above are implemented.

[0047] Beneficial effects: The present invention relates to a parameter updating method based on low-rank decomposition, which aims to solve the problems of high consumption of computing resources and difficulty in balancing model performance and complexity faced by the prior art when processing large-scale financial data. The present invention determines a pre-trained model and obtains a source data set, and uses the source data set to train the pre-trained model to obtain a source model; fixes the parameters of all layers of the source model, and inserts a preset number of data processing layers into the source model; obtains a training data set and uses the training data set to train the inserted data processing layer, and updates the parameters of the data processing layer in an incremental manner to finally obtain a target model. The present invention is suitable for scenarios that process large-scale data sets, helps to solve the problems of excessive consumption of computing resources and long training time during model training, significantly improves the practicality and response speed of the model, and at the same time improves the performance and adaptability of the model on specific tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0049] Figure 1 It is a flow chart of an embodiment of a parameter updating method based on low-rank decomposition of the present invention;

[0050] Figure 2 It is a functional module diagram of a preferred embodiment of a parameter updating device based on low-rank decomposition of the present invention;

[0051] Figure 3 It is a structural diagram of the device hardware operating environment involved in the embodiment of the parameter updating device based on low-rank decomposition of the present invention. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0053] It should be noted that in the modern financial industry, with the rapid development of big data and artificial intelligence technologies, financial institutions are increasingly relying on data-driven decision-making models. These models are widely used in key business areas such as risk management, credit assessment, market forecasting, and fraud detection. However, with the increasing scale and complexity of financial data, how to effectively train and optimize these models, especially while ensuring model accuracy and reducing the consumption of computing resources, has become a technical problem that needs to be solved urgently.

[0054] Traditional financial models usually rely on a large amount of historical data for training, and the complexity and number of parameters of these models are often very large. This not only increases the cost of training and deployment, but also places higher demands on computing resources. In addition, due to the diversity and dynamics of financial data, the demand for model generalization and real-time updates is also increasing.

[0055] In order to solve these problems, model compression and optimization techniques based on low-rank decomposition have gradually attracted attention in recent years. Low-rank decomposition technology can reduce the number of model parameters without significantly reducing model performance, thereby reducing computational complexity, and is suitable for deploying financial models in resource-limited environments. In addition, low-rank decomposition can also be combined with other optimization techniques (such as incremental updates and learning rate decay) to further improve the real-time performance and accuracy of the model.

[0056] In the financial field, typical applications of model optimization using low-rank decomposition techniques include:

[0057] Credit scoring model optimization: By performing low-rank decomposition on the credit scoring model, the computational burden of the model can be effectively reduced and the real-time scoring capability of the model can be improved, which is especially important when processing massive amounts of user data.

[0058] Fraud Detection: By applying low-rank decomposition to the fraud detection model, we can reduce the resource consumption of the model in actual operation while ensuring high accuracy, enabling it to work effectively in a high-concurrency transaction environment.

[0059] See also Figure 1 , Figure 1 This is a flow chart of an embodiment of a parameter updating method based on low-rank decomposition provided by the present invention. It should be noted that although a logical sequence is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0060] like Figure 1 As shown, the parameter updating method based on low-rank decomposition proposed in the present invention comprises the following steps:

[0061] S10, determining a pre-trained model, obtaining a source data set, and using the source data set to train the pre-trained model to obtain a source model;

[0062] In this embodiment, a pre-trained model suitable for the current task is selected. A pre-trained model refers to a deep learning model trained on a large-scale data set and has a strong feature extraction capability. Common pre-trained models include ResNet, VGG, BERT, GPT, etc. For example, ResNet50 is selected as the pre-trained model.

[0063] Collect and prepare a large-scale source dataset, which is a large-scale dataset used to train pre-trained models. It usually contains a large amount of labeled data, which can help the model learn common features. For example, the ImageNet dataset contains millions of labeled images covering 1,000 categories.

[0064] The pre-trained model is trained using the source data set. Through training, the model can learn common features from the data set. The training process includes steps such as forward propagation, loss calculation, backward propagation and parameter update.

[0065] Example description:

[0066] Determine the pre-trained model: Select ResNet50 as the pre-trained model.

[0067] Get the source dataset: Use the ImageNet dataset, which contains more than 14 million labeled images covering 1,000 categories.

[0068] Training pre-trained model: Use the ImageNet dataset to train the ResNet50 model. The training process is as follows:

[0069] Data preprocessing: resize the image to a fixed size (e.g. 224x224) and normalize it.

[0070] Forward propagation: Input the image into the model and calculate the predicted probability for each class.

[0071] Loss calculation: The difference between the predicted value and the true label is calculated using the cross entropy loss function.

[0072] Backward propagation: Calculate the gradient of the loss function with respect to the model parameters.

[0073] Parameter update: Use optimization algorithms (such as SGD, Adam) to update model parameters.

[0074] Iteration: Repeat the above steps until the training process converges.

[0075] Through the above steps, the trained ResNet50 model can achieve a high accuracy on the ImageNet dataset and become a source model with strong feature extraction capabilities.

[0076] S20, fixing the parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model;

[0077] In this embodiment, fixing the parameters of all layers of the source model means that the weights and biases of these layers are not updated during the retraining process. By freezing these parameters, the features and knowledge learned by the pre-trained model on a large-scale dataset are retained. All layers of the source model are traversed and the parameters of each layer are set to a non-trainable state. This can be achieved by setting the properties of each layer to ensure that these layers will not be updated during the retraining process.

[0078] Data processing layers refer to newly added trainable layers based on the source model. These layers are used to process new task data so that the model can adapt to new task requirements. Data processing layers can be fully connected layers, convolutional layers, low-rank decomposition layers, etc. Several new data processing layers are inserted after the output layer of the source model. Depending on the task requirements, one or more layers can be inserted, and the structure and parameters of these layers can be defined. These newly added layers will participate in the retraining process to adapt to the new task.

[0079] Example description:

[0080] Suppose ResNet50 is selected as the source model and a new data processing layer is inserted, with the goal of performing a new image classification task.

[0081] Fix the parameters of all layers of the source model: Load the pre-trained ResNet50 model and set the parameters of all layers to be non-trainable, ensuring that these layers retain their features and knowledge learned on the ImageNet dataset during retraining.

[0082] Insert a preset number of data processing layers into the source model:

[0083] A global average pooling layer is inserted after the output layer of the ResNet50 model to convert the feature map into a fixed-size vector.

[0084] Insert a fully connected layer with 1024 neurons and use the ReLU activation function.

[0085] Insert another fully connected layer with 512 neurons, also using the ReLU activation function.

[0086] Finally, an output layer containing the number of target categories is inserted and classified using the softmax activation function.

[0087] Through the above steps, it is possible to ensure that the source model retains its pre-trained features during the retraining process, while enabling the model to adapt to new task requirements through the newly added data processing layer. This approach can significantly reduce the number of trainable parameters, reduce training time and computing resource requirements, and improve the adaptability and performance of the model.

[0088] S30, obtaining a training data set, using the training data set to train the inserted data processing layer, and updating the parameters of the data processing layer in an incremental manner to obtain a target model.

[0089] In this embodiment, a suitable training dataset is collected and prepared according to the target task. For example, for an image classification task, image data containing the target category can be collected and labeled. The training dataset is a dataset used to adjust the model parameters during the retraining process, and usually contains labeled data related to the target task. The dataset should contain sufficiently diverse samples so that the model can learn new features.

[0090] Input the training dataset into the modified model and only update the parameters of the newly added data processing layer. The layer parameters of the source model remain unchanged.

[0091] During the training process, incremental parameter updates are performed on the newly added data processing layers using mini-batch stochastic gradient descent (SGD) or other optimization algorithms. Only a mini-batch of data is calculated and updated each time until the entire data set is traversed. Incremental update is an optimization method that minimizes the loss function by gradually adjusting the model parameters. Only a small portion of the data is updated each time, thereby gradually improving the model performance.

[0092] Through the above steps, we can ensure that the source model retains its pre-trained features during the retraining process, and at the same time, through the newly added data processing layer and incremental update method, the model can adapt to new task requirements. This method can significantly reduce the number of trainable parameters, reduce training time and computing resource requirements, and improve the adaptability and performance of the model.

[0093] The present invention relates to a parameter updating method based on low-rank decomposition, aiming to solve the problems of high consumption of computing resources and difficulty in balancing model performance and complexity faced by the prior art when processing large-scale financial data. The present invention determines a pre-trained model and obtains a source data set, and uses the source data set to train the pre-trained model to obtain a source model; fixes the parameters of all layers of the source model, and inserts a preset number of data processing layers into the source model; obtains a training data set and uses the training data set to train the inserted data processing layer, and updates the parameters of the data processing layer in an incremental manner, and finally obtains a target model. The present invention is suitable for scenarios where large-scale data sets are processed, helps to solve the problems of excessive consumption of computing resources and long training time during model training, significantly improves the practicality and response speed of the model, and at the same time improves the performance and adaptability of the model on specific tasks.

[0094] In one embodiment, in the above S30, the updating of the parameters of the data processing layer in an incremental manner includes:

[0095] S301, decomposing the weight matrix of the data processing layer into two matrices whose ranks are lower than the rank of the original weight matrix, respectively represented as a first matrix and a second matrix;

[0096] S302, the dimension of the first matrix is ​​the same as the number of rows of the pre-trained weight matrix of the source model, and the dimension of the second matrix is ​​the same as the number of columns of the pre-trained weight matrix of the source model;

[0097] S303, keeping the pre-trained weight matrix of the source model unchanged, and updating the parameters of the first matrix and the second matrix of the newly added data processing layer;

[0098] S304, the weight matrix of the updated data processing layer is composed of the product of the first matrix and the second matrix and the sum of the pre-trained weight matrix of the source model to form the final weight matrix of the target model.

[0099] In this embodiment, the weight matrix of the data processing layer is decomposed into two low-rank matrices. The ranks of these two matrices are lower than the rank of the original weight matrix, which means that their dimensions are smaller, thereby reducing the amount of calculation. For example, if the original weight matrix is ​​a 1000x1000 matrix, it can be decomposed into two matrices with ranks of 100, which reduces the number of parameters. It is ensured that the dimensions of the two decomposed matrices are consistent with the original matrix in terms of the number of rows and columns. The purpose of this is to ensure that the matrix product can maintain the same dimensions as the original matrix, so as to seamlessly integrate with the pre-trained weight matrix of the source model.

[0100] When updating the data processing layer, the pre-trained weight matrix of the source model remains unchanged, ensuring that the basic features of the model are not affected. Instead, only the two low-rank matrices after decomposition are updated. This approach helps to improve the flexibility and adaptability of the model while maintaining the original features of the model. For example, in a financial market risk management model, the original market volatility features can remain unchanged, while the newly added matrix can be optimized and updated based on the latest market data.

[0101] The updated data processing layer weight matrix is ​​constructed by multiplying the two matrices of the low-rank decomposition plus the pre-trained weight matrix of the source model. This combination enables the model to add new features while maintaining the original features, thereby improving the model's predictive ability. For example, in a financial derivatives pricing model, the final weight matrix can take into account both historical price fluctuations and new market data, resulting in a more accurate pricing model.

[0102] In one specific implementation, in a financial market prediction scenario, a pre-trained model based on time series data is determined. A historical data set containing market trading volume and price fluctuations is obtained, and the data set is standardized and normalized. Subsequently, the pre-trained model is trained by setting a reasonable learning rate and number of training rounds to obtain an accurate source model. Then, all layer parameters of the source model are fixed, and several data processing layers are inserted to capture new trading features in the market. When training these data processing layers, their weight matrices are decomposed into two low-rank matrices, and the parameters of these matrices are updated according to market data, ultimately forming a target model that can quickly respond to market changes.

[0103] In addition, during the training process, mini-batch stochastic gradient descent (SGD) or other optimization algorithms are used to incrementally update the parameters of the newly added data processing layer. Only a mini-batch of data is calculated and updated each time until the entire data set is traversed.

[0104] The formula for updating parameters is:

[0105] ΔW = BA;

[0106] W = W0 + ΔW = W0 + BA;

[0107] Among them, B represents one of the weight matrices of the newly added network layer, B∈R d×r ; A represents the weight matrix of the newly added network layer, A∈R r ; W0 represents the pre-trained weight matrix of the source model, W0∈R d×k ; r, k, d represent the dimensions of the matrix; ΔW represents the weight change matrix of the newly added network layer; W represents the final network weight matrix after retraining, that is, the weight matrix of the target model.

[0108] During the training process, the parameters of the newly added data processing layer are updated by calculating the formula ΔW = BA. The weight matrix W0 of the source model and the weight change matrix ΔW of the newly added layer are used to construct the final target model weight matrix W.

[0109] Example description:

[0110] Suppose you want to perform a cat-dog classification task, select ResNet50 as the source model, and insert a new data processing layer.

[0111] Update the parameters of the data processing layer incrementally: Divide the training data set into batches, each batch contains 32 images. Perform forward propagation on each batch of data and calculate the loss value (such as cross entropy loss). Perform backward propagation and calculate the gradient of the loss function with respect to the parameters of the added layer. Use the mini-batch stochastic gradient descent (SGD) optimization algorithm to update the parameters of the added layer. Repeat the above process until all batches of data are traversed, the model loss value converges, and the performance reaches the expected level.

[0112] The formula for updating parameters is:

[0113] Assume that the pre-trained weight matrix W0 of the source model is a matrix of size d×k. The size of the weight matrix B of the newly added network layer is d×r, and the size of the weight matrix A is r×k;

[0114] Calculate the weight change matrix ΔW of the newly added layer: ΔW = BA;

[0115] Calculate the final network weight matrix W after retraining: W = W0 + ΔW.

[0116] This embodiment uses low-rank decomposition and incremental update methods to not only significantly reduce the computational complexity of the model and reduce the consumption of computing resources, but also maintain or even improve the model's prediction performance. At the same time, the newly added data processing layer can capture new features in financial scenarios, allowing the model to maintain efficient and accurate prediction capabilities when facing dynamic markets or new risk factors.

[0117] In one embodiment, in the above S30, the step of obtaining a training data set includes:

[0118] S305, selecting a candidate training data set, and using word embedding technology to convert the labels in the candidate training data set and the source data set into vector representations, which are represented as candidate label vector representations and source label vector representations respectively;

[0119] S306, calculating the label similarity between the candidate label vector representation and the source label vector representation using cosine similarity;

[0120] S307: If the label similarity is higher than the similarity threshold, use the candidate training data set as a training data set for training the data processing layer.

[0121] In this embodiment, the labels of the candidate training data set and the source data set are converted into vector form through word embedding technology. Word embedding technology maps text labels to a high-dimensional vector space so that the semantic relationship between labels can be reflected by the distance between vectors. For example, in the application of credit scoring, the "high risk" and "low risk" labels can be represented by different vectors through word embedding technology, and then their similarity can be analyzed through the relationship between vectors.

[0122] The cosine similarity between the candidate label vector and the source label vector is calculated to measure the semantic similarity between the two. Cosine similarity measures the similarity between two vectors by calculating the cosine value of the angle between them. The closer the value is to 1, the more similar the two are. In financial data processing, this method can help determine the semantic similarity between different financial products or behaviors, so as to screen out more relevant training data.

[0123] A similarity threshold is set as the screening criterion. Only when the similarity between the labels of the candidate training dataset and the labels of the source dataset exceeds this threshold, the candidate training dataset will be selected for the training data processing layer. This process ensures the consistency of the features of the dataset and the source model, thereby improving the generalization ability and prediction accuracy of the model.

[0124] In one specific implementation, in an anti-fraud detection scenario, a candidate training data set containing various transaction behaviors is selected, and the transaction behavior labels (such as "normal transaction" and "suspicious transaction") are converted into vector representations using word embedding technology, and the cosine similarity between them and the source data set labels (such as "confirmed fraud") is calculated. If the similarity between some candidate data set labels and the "confirmed fraud" label exceeds a set threshold, these data sets are used to update the detection layer of the anti-fraud model to improve its ability to detect new types of fraudulent behaviors.

[0125] This embodiment uses word embedding technology and cosine similarity calculation to effectively screen out training data sets that match the characteristics of the source model from a large number of candidate data sets, thereby improving the training efficiency and accuracy of the model. In particular, in financial scenarios, it can help the model better capture emerging market trends, credit risks or fraudulent behaviors, and improve its adaptability to new data while maintaining the generalization ability of the model.

[0126] In one embodiment, in the above S30, the updating of the parameters of the data processing layer in an incremental manner includes:

[0127] S308, setting an initial learning rate and a learning rate decay strategy, dividing the training data set into multiple small batches of data, each small batch of data containing a fixed preset number of samples;

[0128] S309, extracting a small batch of data from the training data set each time, and performing forward propagation to calculate the prediction result of the model;

[0129] S310, calculating a loss function according to the prediction result and the actual label, calculating the gradient of the loss function relative to the parameters of the data processing layer through back propagation, and updating the parameters using the current learning rate;

[0130] S311, after each time the small batch data is used to update the parameters of the data processing layer, the learning rate is adjusted according to the current learning rate decay strategy until all the small batch data are processed.

[0131] In this embodiment, when training the model, the initial learning rate determines the step size for each parameter update. The learning rate decay strategy is used to gradually reduce the learning rate during the training process to avoid excessive parameter updates when the model is close to the optimal solution. Dividing the training data set into multiple mini-batches is to improve the efficiency of training and enable the model to be progressively updated based on each mini-batch of data. For example, the initial learning rate can be set to 0.01, the batch size can be set to 32, and the learning rate decay can be reduced by 10% after every 100 mini-batches of data.

[0132] Forward propagation refers to the process of passing input data through each layer of the model to calculate the output results. After extracting a small batch of data from the training data set each time, the model's prediction results are obtained through forward propagation. This process is used to calculate the performance of the model under the current parameters.

[0133] The loss function measures the gap between the model's predictions and the actual labels. Commonly used loss functions include mean squared error (MSE) and cross entropy loss. Backpropagation is the process of calculating gradients, which passes the gradients of the loss function back to the layers of the model to guide parameter updates. These gradients are used to adjust the model parameters by multiplying the current learning rate.

[0134] After each parameter update using a mini-batch of data, the learning rate is adjusted according to the preset learning rate decay strategy. For example, the learning rate can be halved after a certain number of mini-batch trainings. The purpose of this is to make the model more stable when it is close to convergence and avoid excessive parameter updates causing the model to jump out of the optimal solution.

[0135] This embodiment updates the parameters of the data processing layer in an incremental manner, and combined with the initial learning rate and learning rate decay strategy, can effectively control the update step size of the model parameters and improve the convergence speed and stability of the model.

[0136] In one embodiment, in the above S30, after obtaining the target model, the method further includes:

[0137] S401, obtaining a verification data set, and determining an evaluation index according to the task requirements of the target model;

[0138] S402, calculating the prediction result of the target model through the verification data set, and performing quantitative evaluation according to the selected evaluation index to generate a comprehensive evaluation result;

[0139] S403: Analyze the comprehensive evaluation result and adjust the parameters of the target model.

[0140] In this embodiment, after the model training is completed, a data set (i.e., a validation data set) independent of the training process is obtained to evaluate the performance of the model. According to the specific task performed by the target model, an appropriate evaluation indicator is selected. These indicators may include accuracy, F1 score, mean square error (MSE), AUC, etc., to measure the performance of the model on the validation set. For example, in credit risk assessment, AUC (area under the curve) can be selected as an evaluation indicator to measure the model's ability to distinguish high-risk from low-risk customers.

[0141] The validation dataset is fed into the target model, the model's predictions are calculated, and then these predictions are quantitatively evaluated based on the previously selected evaluation metrics. Quantitative evaluation involves comparing the model's predictions with the true labels in the validation set to generate a comprehensive evaluation score. For example, in a financial market prediction model, the accuracy of the model's predictions of market prices can be evaluated by the mean squared error (MSE).

[0142] Analyze the generated comprehensive evaluation results to determine whether the model's performance meets the expected requirements. If the evaluation results show that the model's performance in some aspects is poor, you need to improve the performance by adjusting the model's parameters. Parameter adjustment may involve resetting the learning rate, increasing or decreasing the number of model layers, or using regularization techniques to avoid overfitting. For example, in an anti-fraud detection model, if the model lacks detection accuracy, you may need to improve the model's accuracy by adjusting the model's weights or increasing the amount of training data.

[0143] In one specific implementation, in the credit scoring system, independent customer historical data is used as a validation data set. According to the requirements of the credit scoring model, AUC (area under the curve) is selected as the main evaluation indicator. The credit risk prediction results of the model are calculated using the validation data set, and the AUC is used to quantitatively evaluate the model's ability to distinguish. If the AUC score of the model is lower than the set threshold, the regularization parameter of the model is adjusted to improve its ability to identify high-risk customers.

[0144] This embodiment ensures that the model has high reliability and accuracy in practical applications by performing verification and evaluation after obtaining the target model. According to the needs of different financial scenarios (such as credit scoring, market forecasting, fraud detection), appropriate evaluation indicators are selected and comprehensive evaluation is performed to enable the model to better adapt to specific tasks, and the model performance is further optimized through parameter adjustment. The application effect of the model in the real world is improved, and the risks caused by inaccurate models are reduced.

[0145] In one embodiment, in the above S20, inserting a preset number of data processing layers into the source model includes:

[0146] S201, determining the type of insertion layer based on the requirements of the target task, and designing the preset number and depth of the insertion layer according to the type of the insertion layer;

[0147] S202, analyzing the layer structure of the source model to determine the position of the insertion layer in the source model;

[0148] S203, inserting a data processing layer into the source model according to the type of the insertion layer, the preset number and depth of the insertion layer, and the insertion position;

[0149] S204, determining an initialization strategy according to the type of the inserted layer, and initializing all weights and biases of the inserted data processing layer using the initialization strategy.

[0150] In this embodiment, before inserting new data processing layers, it is necessary to determine the types of these inserted layers according to the specific requirements of the target task. For example, the task may require the insertion of a convolutional layer, a fully connected layer, or an attention mechanism layer. The number and depth of the inserted layers (such as the number of layers or the number of neurons) should be designed according to the complexity of the task and the requirements of the model. After determining the type of the inserted layer, the number and depth can also be designed according to the type of the inserted layer. For example, in complex financial market prediction tasks, multiple convolutional layers may be required to extract high-level features.

[0151] When determining where to insert a layer, the layer structure of the source model needs to be analyzed in detail. The choice of where to insert a layer has an important impact on the final performance of the model. For example, inserting a data processing layer after the feature extraction layer of the source model can improve the model's ability to understand complex data, while inserting it before the decision layer can enhance the model's prediction accuracy.

[0152] After completing the design of the type, number, depth and location of the inserted layers, these inserted layers are actually added to the source model. This process needs to ensure the compatibility between the newly inserted layers and the original layers of the source model to maintain the coherence of the model structure. For example, in a financial risk prediction model, the newly added layer may be inserted after the existing feature extraction layer to enhance the capture of specific risk factors.

[0153] To ensure the stability and convergence of model training, the weights and biases of the inserted layers need to be initialized through an appropriate strategy. Different inserted layer types (such as convolutional layers or fully connected layers) may require different initialization methods, such as Xavier initialization or He initialization. This process ensures that the new layer works effectively when added to the model and is gradually optimized during training.

[0154] In one specific implementation, in the credit scoring model, two fully connected layers are inserted according to the task requirements, each containing 128 neurons. The addition of these layers is intended to enhance the model's ability to learn complex behavior patterns. Before insertion, the structure of the source model is analyzed, and it is decided to insert the new layer between the feature extraction layer and the output layer of the existing model. The He initialization strategy is used to initialize the weights and biases of these fully connected layers to ensure that the model can converge quickly in the subsequent training and provide higher accuracy in the task of customer credit scoring.

[0155] This embodiment can significantly enhance the learning ability and adaptability of the model to complex tasks by inserting a properly designed data processing layer into the source model. The type, number, location of the inserted layer, and the reasonable design and application of the initialization strategy ensure the stability of the model during the training process and the improvement of the final performance. It is suitable for scenarios that require processing complex financial data, such as credit scoring, market risk prediction, and fraud detection, and can effectively improve the accuracy and practicality of the model.

[0156] In one embodiment, S10 includes:

[0157] S101, determining a pre-training model based on the requirements of the target task, and selecting a source data set that matches the characteristics of the target task;

[0158] S102, setting model training parameters according to the complexity of the target task and the size of the source data set, including a learning rate and a number of training rounds;

[0159] S103, performing data enhancement processing on the source data set, and performing standardization or normalization processing at the same time;

[0160] S104: training the pre-trained model based on the processed source data set and model training parameters to obtain the source model.

[0161] In this embodiment, during the model development process, it is first necessary to select a suitable pre-trained model according to the requirements of the target task (such as prediction, classification, regression, etc.). The selection of the pre-trained model should take into account its performance and applicability in similar tasks. For example, in a text classification task, BERT or GPT may be selected as a pre-trained model; in an image classification task, ResNet or EfficientNet may be selected. When selecting a source data set, it should be ensured that its features are highly matched with the features required for the target task to ensure that the model can learn and generalize effectively. For example, if the task is to detect financial fraud, the source data set should contain rich historical transaction data and related features.

[0162] The complexity of the task is usually determined by the architecture of the model (such as the number of layers, the number of neurons), the number of features in the data, and the difficulty of the target task (such as the number of classification categories, the noise level of the data, etc.). For complex tasks, such as processing high-dimensional data or predicting nonlinear relationships, the model needs to learn more and more complex patterns. For complex tasks, it is usually necessary to set a lower learning rate. This is because in complex tasks, the space of loss functions can be very complex, with many local minima and saddle points. If the learning rate is too high, the model may skip the optimal solution, resulting in an unstable training process. Therefore, a lower learning rate allows the model to gradually converge to the optimal solution in a complex loss space. For simpler tasks, a higher learning rate can speed up the training process because the space of loss functions is smoother and it is easier for the model to find the optimal solution. In complex tasks, a learning rate decay strategy can be used to gradually reduce the learning rate as the training progresses, so that it converges at a faster speed in the early stages of training and converges steadily when approaching the optimal solution.

[0163] The size of a dataset includes the number of samples and feature dimensions of the data. A large-scale dataset means more samples and higher feature dimensions, and the model takes longer to traverse all samples and learn effective patterns from them.

[0164] For large-scale datasets, due to the large number of data samples, the model requires more training rounds to fully learn the patterns in the data. More training rounds allow the model to traverse the entire dataset multiple times, thereby better learning complex feature relationships. For large-scale datasets, if there are too many training rounds, it may cause the model to overfit, that is, the model overfits to the training data and has difficulty generalizing to new data. Therefore, during the training process, a validation set is needed to monitor the model performance. When the validation set performance no longer improves, training can be stopped early.

[0165] For small datasets, fewer training rounds may be sufficient because there are fewer data samples and the model can learn the data patterns in fewer rounds.

[0166] For example, in financial market forecasting, predicting financial market trends is usually a highly complex task because it involves a large number of influencing factors (such as economic indicators, market sentiment, etc.) and complex nonlinear relationships. Financial market data is usually time series data with large data volume and high dimensionality. The parameters can be set as:

[0167] Learning rate: It is initially set to 0.001, and a learning rate decay strategy is used to gradually reduce the learning rate during training.

[0168] Number of training rounds: Set to 50 to 100 times to ensure that the model can fully learn the complex patterns in the data while preventing overfitting.

[0169] Data augmentation and standardization / normalization are important steps to improve the generalization ability of the model. Data augmentation techniques increase the diversity of data by performing random transformations on the data (such as rotation, flipping, scaling, etc.) to avoid overfitting of the model. For example, when dealing with time series data, the dataset can be enhanced by adding noise or randomly dropping data points. Standardization or normalization is to adjust the feature values ​​of the data to the same scale to ensure that different features have the same impact in model training. This is crucial to ensure the stability of the model under different data distributions.

[0170] After the data processing is completed and the training parameters are set, the pre-trained model is trained. By inputting the source data set that has been data-enhanced and standardized / normalized, the model will gradually optimize its parameters and finally obtain the trained source model. For example, in the credit scoring task, the trained source model can accurately distinguish between high-risk and low-risk customers, laying the foundation for subsequent model optimization and application.

[0171] In one specific implementation, in the credit risk assessment task, BERT is selected as the pre-training model, and a source data set containing historical credit data and behavioral data is used. According to the complexity of the task, the initial learning rate is set to 0.005 and the number of training rounds is 30. Before training, the source data set is subjected to data enhancement processing, including data noise addition and data smoothing processing, and is standardized so that all feature values ​​are within the same range. Then, the processed data set is used for training, and finally a source model that can accurately assess credit risk is obtained.

[0172] This embodiment can significantly improve the learning and generalization capabilities of the model by selecting a suitable pre-trained model based on task requirements and combining data enhancement and standardization processing. Reasonable setting of training parameters (such as learning rate and number of training rounds) ensures that the model can fully learn data features in complex tasks, thereby obtaining a source model with superior performance. It is suitable for tasks in the financial field that require processing large-scale complex data, such as credit scoring, market forecasting, and fraud detection.

[0173] The present invention also provides a parameter updating device based on low-rank decomposition.

[0174] Reference Figure 3 , Figure 3 Schematic diagram of functional modules of a first embodiment of a parameter updating device based on low-rank decomposition according to the present invention. The parameter updating device based on low-rank decomposition comprises:

[0175] A model pre-training module determines a pre-training model, obtains a source data set, and uses the source data set to train the pre-training model to obtain a source model;

[0176] A parameter fixing module, fixing the parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model;

[0177] The parameter updating module obtains a training data set, uses the training data set to train the inserted data processing layer, and updates the parameters of the data processing layer in an incremental manner to obtain a target model.

[0178] The specific implementation of the parameter updating device based on low-rank decomposition of the present invention is basically the same as the various embodiments of the parameter updating method based on low-rank decomposition mentioned above, and will not be repeated here.

[0179] The present invention also provides a parameter updating device based on low-rank decomposition, such as Figure 3 As shown, the parameter update device based on low-rank decomposition may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0180] Those skilled in the art will understand that Figure 3 The hardware structure of the parameter updating device based on low-rank decomposition shown in the figure does not constitute a limitation of the parameter updating device based on low-rank decomposition, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0181] like Figure 3As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a parameter update program based on low-rank decomposition. Among them, the operating system is a program for managing and controlling the parameter update device and software resources based on low-rank decomposition, supporting the operation of the network communication module, the user interface module, the parameter update program based on low-rank decomposition, and other programs or software; the network communication module is used to manage and control the network interface 1004; the user interface module is used to manage and control the user interface 1003.

[0182] exist Figure 3 In the hardware structure of the parameter update device based on low-rank decomposition shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client and communicate data with the client; the processor 1001 can call the parameter update program based on low-rank decomposition stored in the memory 1005, and perform the same operations as the parameter update method based on low-rank decomposition.

[0183] The specific implementation of the parameter updating device based on low-rank decomposition of the present invention is basically the same as the above-mentioned embodiments of the parameter updating method based on low-rank decomposition, and will not be repeated here.

[0184] In addition, an embodiment of the present invention also proposes a computer storage medium, on which a parameter update program based on low-rank decomposition is stored. When the parameter update program based on low-rank decomposition is executed by a processor, the steps of the parameter update method based on low-rank decomposition as described above are implemented.

[0185] The specific implementation of the computer storage medium of the present invention is basically the same as the above-mentioned embodiments of the parameter updating method based on low-rank decomposition, and will not be repeated here.

[0186] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.

[0187] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0188] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk.

[0189] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0190] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A parameter updating method based on low-rank decomposition, characterized in that: The following steps are involved: Determine a pre-trained model, obtain a source data set, and use the source data set to train the pre-trained model to obtain a source model; Fixing the parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model; A training data set is obtained, the inserted data processing layer is trained using the training data set, parameters of the data processing layer are updated in an incremental manner, and a target model is obtained.

2. The parameter updating method based on low-rank decomposition according to claim 1, characterized in that: The updating of the parameters of the data processing layer in an incremental manner includes: Decomposing the weight matrix of the data processing layer into two matrices whose ranks are lower than the rank of the original weight matrix, respectively represented as a first matrix and a second matrix; The dimension of the first matrix is ​​the same as the number of rows of the pre-trained weight matrix of the source model, and the dimension of the second matrix is ​​the same as the number of columns of the pre-trained weight matrix of the source model; Keep the pre-trained weight matrix of the source model unchanged, and update the parameters of the first matrix and the second matrix of the newly added data processing layer; The weight matrix of the updated data processing layer is composed of the product of the first matrix and the second matrix and the sum of the pre-trained weight matrix of the source model to form the final weight matrix of the target model.

3. The parameter updating method based on low-rank decomposition according to claim 1, characterized in that: The step of obtaining a training data set includes: Select a candidate training data set, and use word embedding technology to convert the labels in the candidate training data set and the source data set into vector representations, which are represented as candidate label vector representations and source label vector representations respectively; Calculating the label similarity between the candidate label vector representation and the source label vector representation using cosine similarity; If the label similarity is higher than a similarity threshold, the candidate training data set is used as a training data set for training the data processing layer.

4. The parameter updating method based on low-rank decomposition according to claim 1, characterized in that: The updating of the parameters of the data processing layer in an incremental manner includes: Set the initial learning rate and learning rate decay strategy to divide the training data set into multiple small batches, each of which contains a fixed preset number of samples; Each time, a small batch of data is extracted from the training data set and forward propagation is performed to calculate the model's prediction results; Calculate the loss function based on the predicted results and the actual labels, calculate the gradient of the loss function relative to the parameters of the data processing layer through backpropagation, and update the parameters using the current learning rate; After each mini-batch of data is used to update the parameters of the data processing layer, the learning rate is adjusted according to the current learning rate decay strategy until all mini-batches of data are processed.

5. The parameter updating method based on low-rank decomposition as claimed in claim 1, characterized in that: After obtaining the target model, the method further includes: Obtain a validation data set and determine evaluation indicators based on the task requirements of the target model; Calculating the prediction result of the target model through the verification data set, and performing quantitative evaluation according to the selected evaluation index to generate a comprehensive evaluation result; Analyze the comprehensive evaluation results and adjust the parameters of the target model.

6. The parameter updating method based on low-rank decomposition according to claim 1, characterized in that: Inserting a preset number of data processing layers into the source model includes: Determine the type of insertion layer based on the requirements of the target task, and design the preset number and depth of the insertion layer according to the type of insertion layer; Analyzing the layer structure of the source model to determine the position of the insertion layer in the source model; Inserting a data processing layer into the source model according to the type of the insertion layer, the preset number and depth of the insertion layer, and the insertion position; An initialization strategy is determined according to the type of the inserted layer, and all weights and biases of the inserted data processing layer are initialized using the initialization strategy.

7. The parameter updating method based on low-rank decomposition according to claim 1, characterized in that: Determining a pre-trained model, acquiring a source data set, and using the source data set to train the pre-trained model to obtain a source model includes: Determine the pre-training model based on the requirements of the target task and select the source dataset that matches the characteristics of the target task; Set model training parameters, including learning rate and number of training rounds, according to the complexity of the target task and the size of the source dataset; Performing data enhancement processing on the source data set and performing standardization or normalization processing; The pre-trained model is trained based on the processed source data set and the parameters of the model training to obtain the source model.

8. A parameter updating device based on low-rank decomposition, characterized in that: The parameter updating device based on low-rank decomposition comprises: A model pre-training module determines a pre-training model, obtains a source data set, and uses the source data set to train the pre-training model to obtain a source model; A parameter fixing module, fixing the parameters of all layers of the source model, and inserting a preset number of data processing layers into the source model; The parameter updating module obtains a training data set, uses the training data set to train the inserted data processing layer, and updates the parameters of the data processing layer in an incremental manner to obtain a target model.

9. A parameter updating device based on low-rank decomposition, characterized in that: The low-rank decomposition-based parameter updating device includes a memory, a processor, and a low-rank decomposition-based parameter updating program stored in the memory and executable on the processor. When the low-rank decomposition-based parameter updating program is executed by the processor, the steps of the low-rank decomposition-based parameter updating method as described in any one of claims 1 to 7 are implemented.

10. A computer storage medium, characterized in that: The storage medium stores a parameter updating program based on low-rank decomposition, and when the parameter updating program based on low-rank decomposition is executed by a processor, the steps of the parameter updating method based on low-rank decomposition as described in any one of claims 1-7 are implemented.

Images