Model pruning method, device and equipment and storage medium

Abstract

The invention discloses a model pruning method, device and equipment and a storage medium, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting a test data set into a trained to-be-pruned network model; for each network layer in the to-be-pruned network model, obtaining a network vector set output by the network layer based on each test data, and training an initial classifier which is not completely trained based on each test label and each network vector set to obtain a target classifier corresponding to the network layer; and performing pruning operation on the to-be-pruned network model based on the classification effect data respectively corresponding to the at least two target classifiers to obtain a pruned network model.According to the embodiment of the invention, the problem of low accuracy of a traditional pruning evaluation algorithm is solved, the model performance of the pruned network model is ensured, and the pruning efficiency of the pruned network model is improved. And the reasoning speed of the pruned network model is improved as much as possible.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model pruning method, device, equipment and storage medium. Background Art

[0002] Deep learning models are increasingly being used in various business scenarios, such as face recognition, image classification, image segmentation, etc. By preprocessing business data, selecting and processing business features, selecting training models, adjusting hyperparameters of training models, and performing iterative training, a deep learning model that can solve actual business needs is obtained.

[0003] There may be redundant network layers in the deep learning model. Even if the network layer is removed from the deep learning model, the model performance will hardly decrease. This phenomenon is called over-parameterization, and the technology to eliminate over-parameterization is called model pruning.

[0004] In the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art:

[0005] The traditional model pruning method uses an inaccurate evaluation algorithm for the network layer, which results in a decrease in the model performance of the pruned network model. Summary of the invention

[0006] The embodiments of the present invention provide a model pruning method, device, equipment and storage medium to solve the problem of low accuracy of traditional pruning evaluation algorithms, while ensuring the model performance of the pruned network model and improving the inference speed of the pruned network model as much as possible.

[0007] According to an embodiment of the present invention, a model pruning method is provided, the method comprising:

[0008] Inputting a test data set into the trained network model to be pruned; wherein the test data set contains test data corresponding to at least two test labels, and the network model to be pruned includes at least two network layers;

[0009] For each network layer, a network vector set outputted by the network layer based on each of the test data is obtained, and based on each of the test labels and each of the network vector sets, an untrained initial classifier is trained to obtain a target classifier corresponding to the network layer;

[0010] Based on the classification effect data respectively corresponding to at least two target classifiers, a pruning operation is performed on the network model to be pruned to obtain a pruned network model.

[0011] According to another embodiment of the present invention, a model pruning device is provided, the device comprising:

[0012] A test data set input module, used to input the test data set into the trained network model to be pruned; wherein the test data set contains test data corresponding to at least two test labels, and the network model to be pruned includes at least two network layers;

[0013] An initial classifier training module is used to obtain, for each network layer, a network vector set output by the network layer based on each of the test data, and to train an untrained initial classifier based on each of the test labels and each of the network vector sets to obtain a target classifier corresponding to the network layer;

[0014] The pruned network model determination module is used to perform a pruning operation on the network model to be pruned based on the classification effect data corresponding to at least two target classifiers to obtain a pruned network model.

[0015] According to another embodiment of the present invention, there is provided an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model pruning method described in any embodiment of the present invention.

[0019] According to another embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model pruning method described in any embodiment of the present invention when executed.

[0020] The technical solution of the embodiment of the present invention obtains, for each network layer in the network model to be pruned, a network vector set output by the network layer based on at least two test data input respectively, and based on the test labels and network vector sets corresponding to the test data respectively, trains an untrained initial classifier to obtain a target classifier corresponding to the network layer, and based on the classification effect data corresponding to the at least two target classifiers, performs a pruning operation on the network model to be pruned to obtain a pruned network model, thereby solving the problem of low accuracy of traditional pruning evaluation algorithms, and while ensuring the model performance of the pruned network model, maximizes the inference speed of the pruned network model.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A flow chart of a model pruning method provided by one embodiment of the present invention;

[0024] Figure 2 A flowchart of another model pruning method provided by one embodiment of the present invention;

[0025] Figure 3 A flowchart of a pruning process of a network model to be pruned provided by an embodiment of the present invention;

[0026] Figure 4 A flowchart of a pruning process of another network model to be pruned provided by an embodiment of the present invention;

[0027] Figure 5 A flowchart of a specific example of a model pruning method provided by one embodiment of the present invention;

[0028] Figure 6 A schematic diagram of label classification accuracy provided by an embodiment of the present invention;

[0029] Figure 7 A schematic diagram of the structure of a model pruning device provided by one embodiment of the present invention;

[0030] Figure 8 The present invention is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0031] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first", "second", "initial", "target", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0033] Figure 1 This is a flow chart of a model pruning method provided by an embodiment of the present invention. This embodiment is applicable to the case of pruning a deep learning model. The method can be executed by a model pruning device. The model pruning device can be implemented in the form of hardware and / or software. The model pruning device can be configured in a terminal device. Figure 1 As shown, the method includes:

[0034] S110, inputting the test data set into the trained network model to be pruned.

[0035] In this embodiment, the test data set includes test data corresponding to at least two test labels, wherein specifically, the test data includes at least one test sample data corresponding to the test label, and the test label is used to represent the classification category of the test data.

[0036] In a specific embodiment, the test data set can be pre-stored locally on the current terminal device, or in other storage devices or the cloud associated with the current terminal device, and when needed, the test data set can be searched and obtained from the local current terminal device, other storage devices or the cloud associated with the current terminal device.

[0037] In another specific embodiment, each test data in the test data set is pre-stored locally on the current terminal device, in other storage devices associated with the current terminal device, or in the cloud, and when necessary, the test data is searched and acquired from the local current terminal device, in other storage devices associated with the current terminal device, or in the cloud according to the mapping relationship between the test tag and the test data.

[0038] In order to effectively distinguish different test data and avoid repeated storage of the same test data, a unique test data identifier, such as a test number, may be established for different test data in the entire terminal device or device group. Alternatively, different test data may be stored in different storage devices.

[0039] It is understandable that in order to facilitate the search of test data, a hash table can be used for storage, that is, after the test tag is hashed, a storage address is obtained, and the test data corresponding to the test tag is stored in the storage space corresponding to the storage address. When necessary, the test tag is used as the key to query the hash table, and the test data corresponding to the test tag is obtained according to the query result.

[0040] In this embodiment, the network model to be pruned includes at least two network layers. Specifically, the network model to be pruned adopts a deep learning model for data classification. Exemplarily, the deep learning model includes but is not limited to a feedforward network model, a convolutional network model (CNN), a residual network model, etc. The deep learning model is not limited here and can be customized according to actual needs.

[0041] S120. For each network layer, obtain the network vector sets output by the network layer based on each test data, and based on each test label and each network vector set, train the untrained initial classifier to obtain the target classifier corresponding to the network layer.

[0042] Specifically, the network vector set includes network vectors corresponding to each test sample data in the test data. In a specific embodiment, for each test data, at least one test sample data in the test data is input into the network model to be pruned, and a hook function pre-registered in the network output module in the network layer is called to obtain the network vectors corresponding to each test sample data output by the network layer, and each network vector together constitutes the network vector set corresponding to the test data.

[0043] In a specific embodiment, based on each test label and each network vector set, an untrained initial classifier is trained to obtain a target classifier corresponding to the network layer, including: based on each test label, each network vector set is divided into a training vector set and a verification vector set; each training network vector in the training vector set is input into the untrained initial classifier to obtain output predicted classification label data; based on the predicted classification label data and each test label corresponding to the training vector set, the model parameters of the initial classifier are adjusted to obtain the target classifier corresponding to the network layer.

[0044] Specifically, the training vector set includes at least one training network vector corresponding to each test label, and the verification vector set includes at least one verification network vector corresponding to each test label. The training vector set is used to train the initial classifier that has not been trained, and the verification vector set is used to obtain the classification effect data of the target classifier corresponding to the network layer.

[0045] Among them, exemplary, the initial classifier includes but is not limited to SGD (stochastic gradient descent) classifier, SVM (Support Vector Machine) classifier, KNN classifier, etc. There is no limitation on the initial classifier used here, and the specific settings can be customized according to actual needs.

[0046] Specifically, the predicted classification label data includes predicted classification labels corresponding to each training network vector, and a classification loss function is determined based on each predicted classification label and a test label corresponding to each predicted classification label. The model parameters of the initial classifier are adjusted based on the classification loss function until the classification loss function converges, and the initial classifier is used as the target classifier corresponding to the network layer.

[0047] Among them, illustratively, the function type of the classification loss function includes but is not limited to the cross entropy loss function, the Kullback-Leibler divergence loss function, the exponential loss function, etc. The function type of the classification loss function is not limited here, and can be customized according to actual needs.

[0048] S130. Based on the classification effect data corresponding to at least two target classifiers, a pruning operation is performed on the network model to be pruned to obtain a pruned network model.

[0049] In a specific embodiment, the method also includes: for each target classifier, inputting each verification network vector in the verification vector set into the target classifier respectively to obtain output target classification label data; based on the target classification label data and each test label corresponding to the verification vector set, determining the classification effect data corresponding to the target classifier.

[0050] Specifically, the target classification label data includes target classification labels corresponding to each verification network vector, and the classification effect data is determined based on each target classification label and the test label corresponding to each target classification label.

[0051] In a specific embodiment, the classification effect data includes the label classification accuracy corresponding to at least one test label, or the classification effect data includes the overall classification accuracy of the target classifier.

[0052] Specifically, the label classification accuracy is used to characterize the proportion of test data corresponding to the test label that is the same as the test label, and the total classification accuracy is used to characterize the proportion of the test data set that is the same as the test label. For example, TP is used to represent the case where the target classification label is a positive example and the test label is a positive example, FP is used to represent the case where the target classification label is a positive example but the test label is a negative example, FN is used to represent the case where the target classification label is a negative example but the test label is a positive example, and TN is used to represent the case where the target classification label is a negative example and the test label is a negative example. Then the label classification accuracy OA x Satisfy the formula:

[0053]

[0054] Total classification accuracy OA n Satisfy the formula:

[0055]

[0056] In a specific embodiment, the classification effect data may also include the accuracy, F1 score, and AUC score of the target classifier. The embodiment of the present invention mainly uses the classification effect data as the classification accuracy for exemplary description.

[0057] The technical solution of this embodiment obtains, for each network layer in the network model to be pruned, a network vector set output by the network layer based on at least two types of input test data, and trains an untrained initial classifier based on the test labels and network vector sets corresponding to the test data to obtain a target classifier corresponding to the network layer. Based on the classification effect data corresponding to the at least two target classifiers, a pruning operation is performed on the network model to be pruned to obtain a pruned network model, which solves the problem of low accuracy of traditional pruning evaluation algorithms and maximizes the inference speed of the pruned network model while ensuring the model performance of the pruned network model.

[0058] Based on the above embodiment, specifically, the relevant information of the network model to be pruned and at least two initial classifiers can be deployed in a distributed server cluster. The relevant information of the network model to be pruned and each initial classifier is deployed in the storage units of different servers in the distributed server cluster, such as the server where the relevant information of the network model to be pruned is deployed is called a pruning server, and the server where the relevant information of the initial classifier is deployed is called a classification server.

[0059] Specifically, after receiving the pruning trigger instruction, the main control server first sends a calculation instruction to the pruning server, so that the pruning server inputs the obtained test data set into the trained network model to be pruned, and obtains the network vector set output by each network layer based on each test data, and sends each test label and each network vector set to the main control server. For each network layer, the main control server sends each test label and at least two network vector sets corresponding to the network layer to the classification server corresponding to the network layer in the distributed server cluster, so that the classification server trains the untrained initial classifier based on each test label and each network vector set to obtain the target classifier corresponding to the network layer, and sends the obtained classification effect data corresponding to the target classifier to the main control server. The main control server sends the received classification effect data to the pruning server, so that the pruning server performs a pruning operation on the network model to be pruned based on each classification effect data to obtain a pruned network model.

[0060] The above solution uses a distributed server cluster, which can simultaneously train the target classifiers corresponding to each network layer, thereby effectively improving the speed and efficiency of model pruning.

[0061] Figure 2 This is a flow chart of another model pruning method provided by an embodiment of the present invention. This embodiment further refines the above-mentioned embodiment of "practicing pruning operations on the network model to be pruned to obtain a pruned network model based on the classification effect data corresponding to at least two target classifiers". Figure 2 As shown, the method includes:

[0062] S210: Input the test data set into the trained network model to be pruned.

[0063] S220. For each network layer, obtain the network vector sets output by the network layer based on each test data, and based on each test label and each network vector set, train the untrained initial classifier to obtain the target classifier corresponding to the network layer.

[0064] S210-S220 in this embodiment are similar to the above Figure 1 S110 - S120 shown in FIG. 1 are identical or similar to each other, and will not be described in detail in this embodiment.

[0065] S230: Taking the network layer as the current network layer, and taking the classification effect data of the target classifier corresponding to the current network layer as the current classification effect data.

[0066] S240: Acquire the previous classification effect data of the previous network layer corresponding to the current network layer, and determine the difference effect data based on the current classification effect data and the previous classification effect data.

[0067] Specifically, the previous network layer is used to represent the previous network layer connected in series with the current network layer, and the input data of the current network layer includes the network vector output by the previous network layer.

[0068] In a specific embodiment, when the number of the previous network layer is one and the classification effect data includes the total classification accuracy of the target classifier, the difference effect data is used to represent the difference between the current total classification accuracy and the previous total classification accuracy. Exemplarily, the difference effect data Diff satisfies the formula:

[0069]

[0070] in, represents the total classification accuracy of the target classifier corresponding to the kth network layer, Represents the total classification accuracy of the target classifier corresponding to the k-1th network layer.

[0071] In another embodiment, when the number of previous network layers is multiple and the classification effect data includes the total classification accuracy of the target classifier, the difference effect data is used to characterize the statistical value of the difference between the current total classification accuracy and at least two previous total classification accuracies respectively. The statistical value may be the maximum value, the minimum value, the median value or the average value. Taking the average value as an example, the difference effect data Diff satisfies the formula:

[0072]

[0073] Among them, m represents the total number of the previous network layer, represents the total classification accuracy of the target classifier corresponding to the kth network layer, Represents the total classification accuracy of the target classifier corresponding to the i-th previous network layer.

[0074] In another specific embodiment, when the number of the previous network layer is one, and the classification effect data includes the label classification accuracy corresponding to at least one test label, the difference effect data is used to characterize the statistical value corresponding to at least one difference label classification accuracy. Exemplarily, the statistical value may be the maximum value, the minimum value, the median value or the average value. For example, taking the average value as an example, the difference effect data Diff satisfies the formula:

[0075]

[0076] Where n represents the total number of test labels, represents the label classification accuracy of the jth test label corresponding to the kth network layer, represents the label classification accuracy of the jth test label corresponding to the k-1th network layer, Represents the difference label classification accuracy corresponding to the j-th test label.

[0077] In another specific embodiment, when the number of previous network layers is multiple, and the classification effect data includes label classification accuracy corresponding to at least one test label, the difference effect data is used to characterize the statistical value of the difference label classification accuracy corresponding to each test label and each previous network layer. Exemplarily, the statistical value may be a maximum value, a minimum value, a median value, or an average value. Taking the average value as an example, the difference effect data Diff satisfies the formula:

[0078]

[0079] in, It represents the label classification accuracy of the jth test label corresponding to the i-th previous network layer.

[0080] In a specific embodiment, if the current network layer is the first network layer in the network model to be pruned, the previous classification effect data of the previous network layer can be a preset value. For example, the preset value can be 0 or 0.5. The preset value is not limited here and can be customized according to actual needs.

[0081] S250: Based on each difference effect data, a pruning operation is performed on the network model to be pruned to obtain a pruned network model.

[0082] In a specific embodiment, based on each difference effect data, a pruning operation is performed on the network model to be pruned to obtain a pruned network model, including: for each difference effect data, when the difference effect data is less than a preset difference threshold, the network layer corresponding to the difference effect data is deleted from the network model to be pruned to obtain a pruned network model.

[0083] Specifically, if the difference effect data is less than the preset difference threshold, it means that the network layer corresponding to the difference effect data is not helpful for improving the performance of the model and can be deleted from the network model to be pruned. If the difference effect data is greater than or equal to the preset difference threshold, it means that the network layer corresponding to the difference effect data is more helpful for improving the performance of the model and needs to be retained in the network model to be pruned.

[0084] For example, if the model accuracy requirement for the pruned network model is high and the model reasoning speed requirement is low, the preset difference threshold may be 5%, and if the model accuracy requirement for the pruned network model is low and the model reasoning speed requirement is high, the preset difference threshold may be 20%. The preset difference threshold is not limited here and can be customized according to actual needs.

[0085] On the basis of the above embodiments, specifically, the method also includes: inputting training data corresponding to at least two training labels into the pruned network model to obtain at least two model prediction labels as output; based on each model prediction label and the training labels corresponding to each model prediction label, adjusting the model parameters of the pruned network model to obtain a trained target network model.

[0086] Since the reasoning ability of the pruned network model on business data will change to a certain extent compared with the network model to be pruned after updating from the network model to be pruned to the pruned network model, the pruned network model is further trained, and the reasoning speed of the target network model obtained is faster and the reasoning performance is better, which improves the integrity of the model pruning method.

[0087] Figure 3 A flowchart of a pruning process of a network model to be pruned provided by an embodiment of the present invention. Specifically, the difference classification accuracy corresponding to each network layer in the network model to be pruned is obtained, and it is determined whether there is a network layer in the network model to be pruned whose difference classification accuracy is less than a preset difference threshold. If so, at least one network layer that meets the above judgment condition is deleted from the network model to be pruned to obtain a pruned network model, and the pruned network model is retrained to obtain a target network model, and the target network model is saved. If not, the network model to be pruned is saved as the target network model.

[0088] This embodiment uses the difference effect data corresponding to each network layer to accurately identify all redundant network layers in the network model to be pruned, and performs a one-time pruning operation on the network model to be pruned, saving the time of repeated pruning judgments during the iterative training process of the model, thereby saving the computing power required for model pruning optimization and improving the efficiency of model pruning.

[0089] In another specific embodiment, based on each difference effect data, a pruning operation is performed on the network model to be pruned to obtain a pruned network model, including: taking the smallest difference effect data among each difference effect data as the target difference effect data; when the target difference effect data is less than a preset difference threshold, deleting the network layer corresponding to the target difference effect data from the network model to be pruned to obtain an updated network model; taking the updated network model as the network model to be pruned, and training the network model to be pruned; returning to execute the step of inputting the test data set into the trained network model to be pruned, until the target difference effect data is greater than or equal to the preset difference threshold, and taking the network model to be pruned as the pruned network model.

[0090] Figure 4A flowchart of another pruning process of a network model to be pruned provided by an embodiment of the present invention. Specifically, the difference classification accuracy corresponding to each network layer in the network model to be pruned is obtained, and the minimum difference classification accuracy among the difference classification accuracy is used as the target difference classification accuracy, and it is determined whether the target difference classification accuracy is less than the preset difference threshold. If so, the network layer corresponding to the target difference classification accuracy is deleted from the network model to be pruned to obtain an updated network model, and the updated network model is used as the network model to be pruned, and the network model to be pruned is trained, and the step of obtaining the difference classification accuracy corresponding to each network layer in the network model to be pruned is returned. If not, the network model to be pruned is used as the pruned network model.

[0091] Since the reasoning ability of the updated network model for business data will change to a certain extent compared with the network model to be pruned after each model pruning, some network layers may appear redundant in the network model to be pruned, but appear important in the updated network model. One-time pruning may have the risk of reducing the performance of the network model. This embodiment can effectively avoid the above risks and ensure the model performance of the network model by performing pruning evaluation on the network model to be pruned after each training is completed.

[0092] The technical solution of this embodiment, by taking the network layer as the current network layer for each network layer, and taking the classification effect data of the target classifier corresponding to the current network layer as the current classification effect data, obtaining the previous classification effect data of the previous network layer corresponding to the current network layer, and determining the difference effect data based on the current classification effect data and the previous classification effect data, and performing a pruning operation on the network model to be pruned based on the difference effect data to obtain a pruned network model, thereby improving the pruning operation of the network model to be pruned, and while ensuring the model performance of the pruned network model, the inference speed of the pruned network model is improved as much as possible.

[0093] On the basis of the above embodiment, specifically, the data type of the test data is order data, the test label is an order risk label, and the network model to be pruned is a risk prediction model.

[0094] For example, the order data may include item order data or credit application order data, etc. Taking item order data as an example, the item order data includes but is not limited to order time, login address, payment type, installment ratio, item type, item quantity and total order price, etc. The item order data is not limited here and can be customized according to actual needs.

[0095] Exemplarily, the order risk label can be whether there is risk, or the risk level, or the risk type, such as risk types including but limited to information leakage risk, fraud risk, cash-out risk, and no risk, etc. The order risk label is not limited here, and can be customized according to actual needs.

[0096] In a specific embodiment, the network model to be pruned is a Tinybert model + CNN model, and the target classifier is an SGD classifier. The Tinybert model includes 4 transformer encoding layers, and the transformer encoding layer includes a position encoding layer, a multi-head attention layer, and a feedforward network layer. In this embodiment, the Tinybert model in the network model to be pruned is pruned.

[0097] Figure 5 A flowchart of a specific example of a model pruning method provided by an embodiment of the present invention. Specifically, at least two order risk labels and order data corresponding to each order risk label are obtained, and each order data is input into the trained Tinybert model + CNN model. Transformer coding layer 1-transformer coding layer 4 in the Tinybert model outputs network vector sets corresponding to each order data, namely network vector set 1-network vector set 4, and the corresponding SGD classifiers are trained based on network vector set 1-network vector set 4, and the label classification accuracy corresponding to the trained SGD classifier 1-SGD classifier 4 is obtained, namely label classification accuracy 1-label classification accuracy 4. Based on label classification accuracy 1-label classification accuracy 4, a pruning operation is performed on the Tinybert model to obtain a pruned network model + CNN model. Among them, the number of transformer coding layers in the pruned network model is less than 4.

[0098] by Figure 5 For example, Figure 6 A schematic diagram of label classification accuracy provided by an embodiment of the present invention. Specifically, Figure 6 The horizontal axis in the figure represents each transformer encoding layer in the Tinybert model, and the vertical axis represents the label classification accuracy. The darker curve represents the label classification accuracy of the target classifier of each transformer encoding layer and the "risk-free" label, and the lighter curve represents the label classification accuracy of the target classifier of each transformer encoding layer and the "risky" label.

[0099] from Figure 6It can be seen that in the TinyBERT model, the label classification accuracy corresponding to the target classifier of each transformer coding layer has almost no significant change. That is to say, after the network vector output by the previous transformer coding layer passes through the current transformer coding layer, the network vector output by the current transformer coding layer has no effect on improving the label classification accuracy of the target classifier, indicating that the current transformer coding layer is redundant. Deleting the current transformer coding layer from the TinyBERT model will not only not reduce the classification performance of the TinyBERT model, but also improve the inference speed of the TinyBERT model.

[0100] by Figure 5 For example, Table 1 shows the classification performance data corresponding to four network models provided by an embodiment of the present invention.

[0101] Network Model AUC KS Accuracy Recall Tinybert (1 layer transformer) + CNN 0.986 0.891 0.909 0.652 Tinybert (2-layer transformer) + CNN 0.987 0.891 0.909 0.660 Tinybert (3-layer transformer) + CNN 0.987 0.891 0.909 0.655 Tinybert (4-layer transformer) + CNN 0.985 0.887 0.9087 0.639

[0102] The Tinybert model (4-layer transformer) + CNN model in Table 1 is the network model to be pruned. The "Tinybert (3-layer transformer) + CNN" in Table 1 means the Tinybert model + CNN model containing 3 transformer coding layers obtained after the Tinybert model in the pruned network model is pruned. "Tinybert (2-layer transformer) + CNN" means the Tinybert model + CNN model containing 2 transformer coding layers obtained after the Tinybert model in the pruned network model is pruned. "Tinybert (1-layer transformer) + CNN" means the Tinybert model + CNN model containing 1 transformer coding layer obtained after the Tinybert model in the pruned network model is pruned. The performance of the above four network models is evaluated from four performance evaluation dimensions, namely, AUC score, KS score, precision and recall rate.

[0103] As can be seen from Table 1, the Tinybert model in the network model to be pruned is pruned from 4 transformer coding layers to 3 transformer coding layers, 2 transformer coding layers and 1 transformer coding layer respectively, and its AUC score, KS score, precision and recall rate all change little, indicating that the model performance of the network model to be pruned is almost not attenuated after being pruned.

[0104] by Figure 5For example, Table 2 shows the inference times corresponding to the four network models provided by an embodiment of the present invention.

[0105]

[0106] It can be seen from Table 2 that the more transformer encoding layers are pruned from the Tinybert model in the network model to be pruned, the shorter the inference time of the pruned network model, and the greater the improvement rate of the inference time.

[0107] The following is an embodiment of a model pruning device provided in an embodiment of the present invention. The device and the model pruning method of the above embodiment belong to the same inventive concept. For details not described in detail in the embodiment of the model pruning device, reference can be made to the contents about the model pruning method in the above embodiment.

[0108] Figure 7 The following is a schematic diagram of the structure of a model pruning device provided by one embodiment of the present invention. Figure 7 As shown, the device includes: a test data set input module 310, an initial classifier training module 320 and a pruning network model determination module 330.

[0109] The test data set input module 310 is used to input the test data set into the trained network model to be pruned; wherein the test data set includes test data corresponding to at least two test labels, and the network model to be pruned includes at least two network layers;

[0110] The initial classifier training module 320 is used to obtain, for each network layer, a network vector set output by the network layer based on each test data, and train an untrained initial classifier based on each test label and each network vector set to obtain a target classifier corresponding to the network layer;

[0111] The pruned network model determination module 330 is used to perform a pruning operation on the network model to be pruned based on the classification effect data corresponding to at least two target classifiers to obtain a pruned network model.

[0112] The technical solution of this embodiment solves the problem of low accuracy of the traditional pruning evaluation algorithm, and while ensuring the model performance of the pruned network model, it improves the inference speed of the pruned network model as much as possible.

[0113] In a specific embodiment, the data type of the test data is order data, the test label is an order risk label, and the network model to be pruned is a risk prediction model.

[0114] In a specific embodiment, the pruning network model determination module 330 includes:

[0115] A current classification effect data determination unit is used to, for each network layer, take the network layer as the current network layer, and take the classification effect data of the target classifier corresponding to the current network layer as the current classification effect data;

[0116] A difference effect data determining unit, used to obtain the last classification effect data of the last network layer corresponding to the current network layer, and determine the difference effect data based on the current classification effect data and the last classification effect data;

[0117] The pruned network model determination unit is used to perform a pruning operation on the network model to be pruned based on each difference effect data to obtain a pruned network model.

[0118] In a specific embodiment, the pruning network model determination unit includes:

[0119] The first pruned network model determination subunit is used to, for each difference effect data, delete the network layer corresponding to the difference effect data from the network model to be pruned when the difference effect data is less than a preset difference threshold, so as to obtain a pruned network model.

[0120] In a specific embodiment, the device further comprises:

[0121] A target network model determination module is used to input the training data corresponding to at least two training labels into the pruned network model to obtain at least two model prediction labels as output;

[0122] Based on the prediction labels of each model and the training labels corresponding to the prediction labels of each model, the model parameters of the pruned network model are adjusted to obtain the trained target network model.

[0123] In a specific embodiment, the pruning network model determination unit includes:

[0124] The second pruning network model determines a subunit, which is used to take the smallest difference effect data among the difference effect data as the target difference effect data;

[0125] When the target difference effect data is less than the preset difference threshold, the network layer corresponding to the target difference effect data is deleted from the network model to be pruned to obtain an updated network model;

[0126] The updated network model is used as the network model to be pruned, and the network model to be pruned is trained;

[0127] Return to the step of inputting the test data set into the trained network model to be pruned, until the target difference effect data is greater than or equal to the preset difference threshold, and use the network model to be pruned as the pruned network model.

[0128] In a specific embodiment, the initial classifier training module 320 is specifically used to:

[0129] Based on each test label, each network vector set is divided into a training vector set and a validation vector set;

[0130] Input each training network vector in the training vector set into the untrained initial classifier to obtain the output predicted classification label data;

[0131] Based on the predicted classification label data and the test labels corresponding to the training vector set, the model parameters of the initial classifier are adjusted to obtain the target classifier corresponding to the network layer;

[0132] The device also includes:

[0133] A classification effect data determination module is used to input each verification network vector in the verification vector set into the target classifier for each target classifier, and obtain output target classification label data;

[0134] Based on the target classification label data and the test labels corresponding to the verification vector set, the classification effect data corresponding to the target classifier is determined.

[0135] In a specific embodiment, the classification effect data includes the label classification accuracy corresponding to at least one test label, or the classification effect data includes the overall classification accuracy of the target classifier.

[0136] The model pruning device provided in the embodiment of the present invention can execute the model pruning method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0137] Figure 8 A schematic diagram of the structure of an electronic device provided for one embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0138] like Figure 8As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by the at least one processor 11, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0139] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as the model pruning method provided in the above embodiments.

[0141] In some embodiments, the model pruning method provided in the above embodiments may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps in the model pruning method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the model pruning method in any other appropriate manner (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] The computer programs for implementing the model pruning method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0144] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0147] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A model pruning method, characterized in that: include: Inputting a test data set into the trained network model to be pruned; wherein the test data set contains test data corresponding to at least two test labels, and the network model to be pruned includes at least two network layers; For each network layer, a network vector set outputted by the network layer based on each of the test data is obtained, and based on each of the test labels and each of the network vector sets, an untrained initial classifier is trained to obtain a target classifier corresponding to the network layer; Based on the classification effect data respectively corresponding to at least two target classifiers, a pruning operation is performed on the network model to be pruned to obtain a pruned network model.

2. The method according to claim 1, characterized in that The data type of the test data is order data, the test label is an order risk label, and the network model to be pruned is a risk prediction model.

3. The method according to claim 1, characterized in that The step of performing a pruning operation on the network model to be pruned based on the classification effect data respectively corresponding to at least two target classifiers to obtain a pruned network model comprises: For each network layer, the network layer is used as the current network layer, and the classification effect data of the target classifier corresponding to the current network layer is used as the current classification effect data; Acquire the last classification effect data of the last network layer corresponding to the current network layer, and determine the difference effect data based on the current classification effect data and the last classification effect data; Based on each of the difference effect data, a pruning operation is performed on the network model to be pruned to obtain a pruned network model.

4. The method according to claim 3, characterized in that The performing a pruning operation on the network model to be pruned based on each of the difference effect data to obtain a pruned network model includes: For each difference effect data, when the difference effect data is less than a preset difference threshold, the network layer corresponding to the difference effect data is deleted from the network model to be pruned to obtain a pruned network model.

5. The method according to claim 4, characterized in that The method further comprises: Inputting the training data corresponding to at least two training labels into the pruned network model to obtain at least two model prediction labels as output; Based on the model prediction labels and the training labels respectively corresponding to the model prediction labels, the model parameters of the pruned network model are adjusted to obtain a trained target network model.

6. The method according to claim 3, characterized in that The performing a pruning operation on the network model to be pruned based on each of the difference effect data to obtain a pruned network model includes: taking the smallest difference effect data among the difference effect data as the target difference effect data; When the target difference effect data is less than a preset difference threshold, the network layer corresponding to the target difference effect data is deleted from the network model to be pruned to obtain an updated network model; Using the updated network model as the network model to be pruned, and training the network model to be pruned; Return to the step of inputting the test data set into the trained network model to be pruned, until the target difference effect data is greater than or equal to the preset difference threshold, and use the network model to be pruned as the pruned network model.

7. The method according to claim 1, characterized in that The step of training an untrained initial classifier based on each of the test labels and each of the network vector sets to obtain a target classifier corresponding to the network layer includes: Based on each of the test labels, dividing each of the network vector sets into a training vector set and a verification vector set; Inputting each training network vector in the training vector set into an untrained initial classifier to obtain output predicted classification label data; Based on the predicted classification label data and each test label corresponding to the training vector set, adjusting the model parameters of the initial classifier to obtain a target classifier corresponding to the network layer; The method further comprises: For each target classifier, each verification network vector in the verification vector set is input into the target classifier to obtain output target classification label data; Based on the target classification label data and each test label corresponding to the verification vector set, classification effect data corresponding to the target classifier is determined.

8. The method according to any one of claims 1 to 7, characterized in that: The classification effect data includes label classification accuracy corresponding to at least one test label, or the classification effect data includes the overall classification accuracy of the target classifier.

9. A model pruning device, characterized in that: include: A test data set input module, used to input the test data set into the trained network model to be pruned; wherein the test data set contains test data corresponding to at least two test labels, and the network model to be pruned includes at least two network layers; An initial classifier training module is used to obtain, for each network layer, a network vector set output by the network layer based on each of the test data, and to train an untrained initial classifier based on each of the test labels and each of the network vector sets to obtain a target classifier corresponding to the network layer; The pruned network model determination module is used to perform a pruning operation on the network model to be pruned based on the classification effect data corresponding to at least two target classifiers to obtain a pruned network model.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model pruning method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model pruning method according to any one of claims 1 to 8 when executed.

Images