Model processing method and device, storage medium and electronic equipment
By clustering and unified processing of the parameter matrix of the deep learning model, and using preset identifiers for compression storage, the existing models' problems of high memory occupancy and high computing latency are solved, and more efficient inference computing and real-time requirements are achieved.
Patent Information
- Application Number
- CN202311103337.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-29
- Publication Date
- 2025-03-18
AI Technical Summary
The existing deep learning models have large amounts of parameters. The trained models occupy a lot of memory during inference calculations and have high computational delays, which cannot meet the real-time requirements of anti-fraud scenarios.
By extracting the parameter matrix of the text processing model, clustering and parameter unification, the updated parameter matrix is obtained, and the preset identifier is used for compression storage, reducing storage space and loading memory.
It effectively reduces the storage space and loading memory of the model, improves the efficiency of inference computing, and meets the real-time requirements of anti-fraud scenarios.
Smart Images
Figure CN119646218A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of deep learning technology, and in particular to a model processing method, device, storage medium and electronic device. Background Art
[0002] With the continuous development of deep learning technology, deep learning models are widely used. In the field of financial risk control, most of the models used in anti-fraud scenarios are time-series NLP models, such as pre-trained large models Bert and GPT.
[0003] In the process of implementing the present invention, it was found that there are at least the following technical problems in the prior art: the current deep learning model has a large number of parameters, the trained model occupies a lot of memory when performing inference calculations, and the calculation delay is high, which cannot meet the business needs of scenarios with high real-time requirements for anti-fraud. Summary of the invention
[0004] The present invention provides a model processing method, device, storage medium and electronic device to reduce the storage space occupied and the loading memory space of the text processing model.
[0005] According to one aspect of the present invention, a method for processing a model is provided, comprising:
[0006] A text processing model is obtained based on training of text sample data;
[0007] Extracting a parameter matrix of the text processing model, clustering the model parameters in the parameter matrix to obtain parameter clusters, and performing parameter unification processing on each of the parameter clusters to obtain an updated parameter matrix;
[0008] The update parameter matrix is compressed and stored based on a preset identifier.
[0009] Optionally, performing parameter unification processing on each parameter cluster to obtain an updated parameter matrix includes:
[0010] For each of the parameter clustering clusters, determining the cluster center data of the parameter clustering cluster;
[0011] The model parameters in the parameter clustering cluster are replaced based on the cluster center data to obtain updated model parameters, wherein the updated model parameters form the updated parameter matrix.
[0012] Optionally, the preset identifier includes a start identifier, a data identifier, a distance identifier, and a line break identifier;
[0013] The compressing and storing the update parameter matrix based on a preset identifier includes: setting a starting identifier respectively according to the starting data of the update parameter matrix; for each non-zero data of the update parameter matrix, determining the data identifier of the non-zero data, generating a distance identifier based on the distance between the non-zero data and adjacent non-zero data; and setting a line break identifier when there is a line break, so as to store the update parameter matrix.
[0014] Optionally, the training of a text processing model based on text sample data includes: creating an initial text processing model; based on the text sample data, performing knowledge distillation on the initial text processing model through a pre-trained teacher model to obtain a trained text processing model, wherein the model parameter amount of the text processing model is less than the model parameter amount of the teacher model.
[0015] Optionally, based on the text sample data, the initial text processing model is subjected to knowledge distillation processing through a pre-trained teacher model to obtain a trained text processing model, including: obtaining text sample data and a first label of the text sample data; inputting the text sample data into the teacher model to obtain a second label output by the teacher model, and obtaining a target label of the text sample data based on the first label and the second label; inputting the text sample data into the initial text processing model to obtain a prediction result, and updating parameters of the initial text processing model based on the prediction result and the target label.
[0016] Optionally, the method further includes: performing multiple random pruning processes on the text processing model to obtain candidate models; performing evaluation processes on the candidate models respectively to obtain evaluation data of the candidate models, and screening the target text processing model from the candidate models based on the evaluation data.
[0017] Optionally, after obtaining the text processing model, it also includes: acquiring text sample data of the target application scenario, and updating parameters of the text processing model based on the text sample data of the target application scenario; wherein the target application scenario is an abnormal address recognition scenario, and the text sample data of the target application scenario includes historical order address data.
[0018] According to another aspect of the present invention, there is provided a model processing device, comprising:
[0019] A model training module, used for training a text processing model based on text sample data;
[0020] A parameter updating module, used for extracting the parameter matrix of the text processing model, clustering the model parameters in the parameter matrix to obtain parameter clusters, and performing parameter unification processing on each of the parameter clusters to obtain an updated parameter matrix;
[0021] The compression storage module is used to compress and store the update parameter matrix based on a preset identifier.
[0022] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0023] at least one processor; and
[0024] a memory communicatively connected to the at least one processor; wherein,
[0025] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the model processing method described in any embodiment of the present invention.
[0026] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model processing method described in any embodiment of the present invention when executed.
[0027] The technical solution of the embodiment of the present invention extracts the model parameters of the trained text processing model to form a parameter matrix, clusters the model parameters in the parameter matrix, and unifies the model parameters in each parameter cluster cluster to reduce the number of model parameter values. The model parameters are replaced by preset identifiers to compress and store the updated parameter matrix, reduce the storage space occupied by the parameter matrix, and realize compressed storage of the text processing model.
[0028] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0030] Figure 1 is a flow chart of a model processing method provided by an embodiment of the present invention;
[0031] Figure 2 is a schematic diagram of parameter unified processing provided by an embodiment of the present invention;
[0032] Figure 3 is a schematic diagram of a compressed storage form of model parameters provided by an embodiment of the present invention;
[0033] Figure 4 It is a flowchart of a model processing method provided by an embodiment of the present invention;
[0034] Figure 5 is a schematic diagram of a knowledge distillation process provided by an embodiment of the present invention;
[0035] Figure 6 It is a flowchart of a model processing method provided by an embodiment of the present invention;
[0036] Figure 7 is a structural schematic diagram of a model processing device provided by an embodiment of the present invention;
[0037] Figure 8 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0038] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0039] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0040] Figure 1This is a flow chart of a model processing method provided by an embodiment of the present invention. This embodiment is applicable to the case of compressing and storing a neural network model. The method can be executed by a model processing device. The model processing device can be implemented in the form of hardware and / or software. The model processing device can be configured in an electronic device such as a computer, a server, etc. Figure 1 As shown, the method includes:
[0041] S110, obtaining a text processing model based on training of text sample data.
[0042] S120, extracting the parameter matrix of the text processing model, clustering the model parameters in the parameter matrix to obtain parameter clusters, and performing parameter unification processing on each of the parameter clusters to obtain an updated parameter matrix.
[0043] S130. Compress and store the update parameter matrix based on a preset identifier.
[0044] In this embodiment, the text processing model may be a neural network model. The network structure of the text processing model is not limited here. Exemplarily, the text processing model may be a convolutional neural network model, a recurrent neural network model, etc.
[0045] The text processing model is a neural network model trained with text sample data and having a text processing function. The text processing function can be trained based on the text sample data and the label data of the text sample data, for example, it can be, but not limited to, abnormal text recognition, text translation, keyword recognition, etc. The above text processing function can be determined according to user needs. Accordingly, the text processing model can be an abnormal text recognition model, a text translation model, a keyword recognition model, etc.
[0046] In this embodiment, the text processing model can be trained based on a supervised training method, and the training process of the text processing model is not limited here. The text sample data can be obtained from an open source database or an open source search engine according to training requirements, or automatically generated according to a sample generator, or screened from historical text data.
[0047] The text processing model may include multiple network layers, wherein the network layer includes an input layer, multiple hidden layers and an output layer, wherein the hidden layer includes but is not limited to a convolution layer, a pooling layer, an activation function layer, etc. Weights are set before different network layers for processing input information.
[0048] The weight parameters of multiple network layers in the text processing model are extracted, and the weight parameters of multiple network layers are formed into a parameter matrix, where the weight parameters of the same network layer can be used as a matrix row or matrix column, and when the number of weight parameters of different network layers is different, they can be padded with zeros. By storing the parameter matrix, the space required for model storage and the space required for loading the model into memory are reduced.
[0049] There are multiple network layers in the text processing model, and accordingly, the number of model parameters in the text processing model is large. Accordingly, the amount of data in the parameter matrix is large, resulting in a large amount of storage space. In view of the above problems, the model parameters in the parameter matrix are unified, and the different data values of the parameter matrix are reduced, so as to facilitate the unified storage of the parameter matrix. Among them, the unified processing of the model parameters can be achieved by clustering, and the model parameters in the same parameter clustering cluster are unified. Through clustering, the model parameters with small numerical differences are divided into a parameter clustering cluster, and accordingly, the model parameters in the parameter clustering cluster are unified. On the basis of ensuring the reduction of different data values, the numerical differences of the model parameters in the parameter clustering cluster are small, and the change of the data values of the unified processing is small, which reduces the change of the model parameters caused by the unified processing, and ensures the processing capacity of the text processing model.
[0050] The non-zero parameters in the parameter matrix are clustered by a clustering algorithm to obtain a plurality of parameter clusters. The clustering algorithm is not limited here, for example, it can be a K-maeans clustering algorithm or a clustering model. For any parameter cluster, a target parameter is determined based on the model parameters in the parameter cluster, and the target parameter is used as the unified parameter of the parameter cluster, that is, the model parameter of the parameter cluster is replaced by the target parameter. In some embodiments, the target parameter can be the cluster center data of the parameter cluster.
[0051] Optionally, parameter unification processing is performed on each of the parameter clustering clusters to obtain an updated parameter matrix, including: for each of the parameter clustering clusters, determining the cluster center data of the parameter clustering cluster; replacing the model parameters in the parameter clustering cluster based on the cluster center data to obtain updated model parameters, wherein the updated model parameters form the updated parameter matrix.
[0052] The cluster center data may be determined by: performing mean processing on the model parameters of the parameter clustering cluster to obtain the first mean of the parameter clustering cluster as the cluster center data. Alternatively, the maximum parameter and the minimum parameter in the parameter clustering cluster may be determined, and the maximum parameter and the minimum parameter may be averaged to obtain the second mean, and the second mean may be used as the cluster center data. Alternatively, the median or mode in the parameter clustering cluster may be determined as the cluster center data. Alternatively, the model parameters may be weighted according to the density of the model parameters in the parameter clustering cluster to obtain the cluster center data, wherein the weight of the model parameters may be positively correlated with the density of the model parameters.
[0053] By replacing different model parameters in parameter clustering clusters with cluster center data, the types of data values can be reduced, and the parameter clustering clusters can be stored uniformly to reduce storage space.
[0054] For example, see Figure 2 , Figure 2 It is a schematic diagram of parameter unification processing provided by an embodiment of the present invention. Among them, model parameters 1.02, 0.94 and 1.09 belong to a parameter clustering cluster, and the cluster center data is 1.01; model parameters 1.87, 1.99 and 1.92 belong to a parameter clustering cluster, and the cluster center data is 1.93; model parameters 0.54, 0.45 and 0.56 belong to a parameter clustering cluster, and the cluster center data is 0.52. Accordingly, the model parameters in the above-mentioned parameter clustering clusters are replaced with the corresponding cluster center data to obtain an updated parameter matrix, wherein the updated parameter matrix includes three different model parameters, and the numerical types of the model parameters in the model matrix before the update are reduced, which is convenient for compressing and storing the updated parameter matrix.
[0055] In order to further reduce the storage space of the update parameter matrix, the update parameter matrix is compressed. Specifically, the model parameters in the update parameter matrix can be converted into preset identifiers for storage, so as to convert the update parameter matrix into an addressing identifier form. The preset identifier can be an identifier composed of 0 and 1, which occupies a storage space smaller than the storage space of the model parameters, thereby achieving a compressed storage effect.
[0056] The preset identifiers include a start identifier, a data identifier, a distance identifier, and a line break identifier; the above preset identifiers can be represented by different numbers of bit positions of 01 data. Exemplarily, the start identifier can be a bit position of 0 or 1, wherein the start identifier of 0 indicates that the data of the first row and the first column in the update parameter matrix is zero, and the start identifier of 1 indicates that the data of the first row and the first column is non-zero data.
[0057] The data identifier may be a data representation of at least two bits of 01, and the bits of the data identifier may be determined according to the type data of the non-zero data in the update parameter matrix. For example, Figure 2 The non-zero data in the updated parameter matrix include three types: 1.01, 1.93 and 0.52. Correspondingly, the data identifier can be represented by two bits of 01 data, that is, 00, 01, 10. The corresponding relationship between the data representation and the non-zero data is pre-built, and a non-zero data is represented by a data identifier, which reduces the space occupied by the non-zero data. At the same time, there is no need to establish the data identifier corresponding to the model parameter 0, which reduces the storage space of the model parameter 0.
[0058] The distance indicator is used to represent the distance between two non-zero data. When two non-zero data are adjacent, the distance is 1. When the model parameter between two non-zero data is 0, the distance is 2, and so on. The distance indicator can be represented by at least two bits of 01 data, which is determined according to the amount of data in the matrix row / matrix column in the update parameter matrix. Figure 2 For example, the distance identifier can be represented by three bits of 01 data, indicating the distance 1-6.
[0059] The line break identifier is used to separate different matrix rows / matrix columns, characterize data line breaks / columns, and distinguish different matrix rows / matrix columns of the update parameter matrix. The line break identifier can be an identifier different from the above-mentioned start identifier, data identifier, and distance identifier, for example, the line break identifier can be 000.
[0060] Correspondingly, compressing and storing the update parameter matrix based on a preset identifier includes: setting a starting identifier according to the starting data of the update parameter matrix; for each non-zero data of the update parameter matrix, determining the data identifier of the non-zero data, and generating a distance identifier based on the distance between the non-zero data and adjacent non-zero data; and, setting a line break identifier when there is a line break, so as to store the update parameter matrix.
[0061] Traverse the model parameters in the update parameter matrix, set a preset flag based on each model parameter, and form stored data. The traversal can be performed in the order of rows or columns, which is not limited here. Taking row-order traversal as an example, for the starting data of the first row and the first column, if the starting data is zero, set the starting flag to 0, and if the starting data is non-zero data, set the starting flag to 1.
[0062] After setting the start mark, determine the non-zero data in the update parameter matrix in turn, determine the distance between the non-zero data and the previous non-zero data, set the distance mark, and determine the data mark corresponding to the non-zero data, set the data mark; determine the next non-zero data, set the distance mark and data mark of the next non-zero data, and so on. After the traversal of the matrix row is completed, set the line break mark, and traverse the next matrix row until the traversal of all model parameters in the update parameter matrix is completed. Optionally, after completing the traversal of the update parameter matrix, you can set the end mark to mark the end of data storage.
[0063] For example, see Figure 3 , Figure 3 It is a schematic diagram of the compressed storage form of the model parameters provided by an embodiment of the present invention. Figure 3 The stored data in Figure 2 The updated parameter matrix in is compressed and stored. A correspondence between data identifiers and model parameters is created in advance, where the data identifier of a model parameter 1.01 can be 0, the data identifier of a model parameter 1.93 can be 01, and the data identifier of a model parameter 0.52 can be 10.
[0064] according to Figure 3 It can be seen that the starting identifier is 1, indicating that the parameter of the first row and the first column is non-zero data, the distance identifier 001 indicates that the distance of the non-zero data is 1, and the data identifier is 00, indicating that the non-zero data is 1.01. It can be seen that the data of the first row and the first column is 1.01. Similarly, the distance identifier 010 indicates that the distance to the next non-zero data is 2, and the data identifier 00 indicates that the next non-zero data is 1.01, that is, the non-zero data after the data 1.01 in the first row and the first column is 1.01 with a model parameter zero interval. The line break identifier 000 indicates that after the above non-zero data is 1.01, there is no non-zero data, and a line break is required. The model parameters after the line break are similar.
[0065] The model parameters in the update parameter matrix are represented by the above preset identifiers, and the update parameter matrix is converted, which reduces the storage space occupied and realizes the compressed storage of the update parameter matrix.
[0066] The technical solution of this embodiment extracts the model parameters of the trained text processing model to form a parameter matrix, clusters the model parameters in the parameter matrix, and unifies the model parameters in each parameter cluster to reduce the number of model parameter values. The model parameters are replaced by preset identifiers to compress and store the updated parameter matrix, reduce the storage space occupied by the parameter matrix, and realize compressed storage of the text processing model.
[0067] Figure 4FIG. 1 is a flow chart of a model processing method provided by an embodiment of the present invention. Based on the above embodiment, the training process of the text processing model is optimized. Figure 4 , the method specifically comprises:
[0068] S210: Create an initial text processing model.
[0069] S220. Based on the text sample data, the initial text processing model is subjected to knowledge distillation processing through a pre-trained teacher model to obtain a trained text processing model, wherein the model parameter amount of the text processing model is smaller than the model parameter amount of the teacher model.
[0070] S230, extracting the parameter matrix of the text processing model, clustering the model parameters in the parameter matrix to obtain parameter clusters, and performing parameter unification processing on each of the parameter clusters to obtain an updated parameter matrix.
[0071] S240: compress and store the update parameter matrix based on a preset identifier.
[0072] The initial text processing model may be a model initialized with model parameters, and the structure of the initial text processing model is simple, and the amount of model parameter data included is small. The initial text processing model is trained to obtain a lightweight text processing model, which reduces the storage space occupied during the storage of the text processing model and reduces the memory space occupied during the loading of the text processing model. At the same time, the lightweight text processing model has high processing efficiency and fast reasoning speed.
[0073] The initial text processing model performs knowledge distillation based on the pre-trained high-precision teacher model to obtain a trained text processing model. Among them, the teacher model has a complex structure and high processing accuracy for text data. The initial text processing model is trained by knowledge distillation. On the basis of obtaining a lightweight text processing model, the training accuracy of the text processing model is guaranteed.
[0074] Optionally, based on the text sample data, the initial text processing model is subjected to knowledge distillation processing through a pre-trained teacher model to obtain a trained text processing model, including: obtaining text sample data, inputting the text sample data into the teacher model, obtaining a predicted label output by the teacher model, inputting the text sample data into the initial text processing model, obtaining a predicted result, and updating the parameters of the initial text processing model based on the predicted result and the predicted label. Specifically, a loss function may be generated based on the predicted result and the predicted label, and the parameters of the initial text processing model may be updated based on the loss function. The above training process is executed in a loop to obtain a trained text processing model. By training the initial text processing model in the above manner, there is no need to set labels for the text sample data, thereby simplifying the processing process of the text sample data.
[0075] Optionally, based on the text sample data, the initial text processing model is subjected to knowledge distillation processing through a pre-trained teacher model to obtain a trained text processing model, including: obtaining text sample data and a first label of the text sample data; inputting the text sample data into the teacher model to obtain a second label output by the teacher model, and obtaining a target label of the text sample data based on the first label and the second label; inputting the text sample data into the initial text processing model to obtain a prediction result, and updating the parameters of the initial text processing model based on the prediction result and the target label. For example, see Figure 5 , Figure 5 It is a flow chart of a knowledge distillation provided by an embodiment of the present invention. Among them, the large model A is the teacher model, and the small model B is the initial text processing model. The soft label is the second label, and the hard label is the first label. The target label is obtained by calculating the mean of the first label and the second label. The loss function is generated by the prediction result of the initial text processing model and the target label, and the parameters of the initial text processing model are updated. The above training process is executed repeatedly to obtain a trained text processing model.
[0076] In some embodiments, local text sample data in a text sample data set is set with a first label. For text sample data that is not set with the first label, the parameters of the initial text processing model can be updated based on the second label output by the teacher model and the prediction results of the initial text processing model. For text sample data that is set with a first label, the target label can be determined based on the first label and the second label, so as to update the parameters of the initial text processing model through the prediction results and the target label.
[0077] The initial text processing model is trained through knowledge distillation to obtain a lightweight text processing model and reduce the size of the model.
[0078] In the text processing model obtained in the above embodiment, the model size is reduced relative to the teacher model, but there is still redundancy in the model parameters, which occupy storage space and affect the reasoning efficiency. In view of the above problems, the trained text processing model is pruned to remove redundant model parameters to obtain a sparse model of the text processing model. The pruning process can be to prune the model parameters in each network layer at a preset ratio. Optionally, the absolute values of the model parameters in each network layer can be sorted from small to large, and the model parameters of the preset ratio before sorting can be removed, that is, the above-mentioned model parameters to be pruned are updated to zero. By pruning model parameters with small absolute values, model parameters with less impact on model operation are removed, thereby reducing the impact on model performance.
[0079] Optionally, the text processing model is subjected to multiple random pruning processes to obtain candidate models; the candidate models are evaluated respectively to obtain evaluation data of the candidate models, and the target text processing model is screened from the candidate models based on the evaluation data. The random pruning process improves the randomness and generalization of the candidate model. Multiple candidate models are obtained by multiple random pruning processes, which reduces the problem of poor model performance caused by a single random pruning process.
[0080] The evaluation process of multiple candidate models may be performed by verifying each candidate model based on a preset verification data set, determining the processing accuracy of each candidate model as evaluation data, and selecting the candidate model with the largest evaluation data as the target text processing model. The evaluation process of multiple candidate models may also be performed by determining the evaluation index of each candidate model, wherein the evaluation index includes but is not limited to the AOC index, etc., performing weighted processing on multiple evaluation indexes to obtain evaluation data, and selecting the candidate model with the largest evaluation data as the target text processing model.
[0081] On the basis of the above-mentioned embodiment, in the training process of the text processing model, in order to reduce the difficulty of obtaining text sample data, the scenario to which the text sample data belongs and the method of obtaining it are not limited. Accordingly, the text processing model trained based on the above-mentioned text sample data is not targeted at the application scenario. After obtaining the text processing model, the text sample data of the target application scenario is obtained, and the parameters of the text processing model are updated based on the text sample data of the target application scenario. The text processing model is further optimized and trained through the text sample data of the target application scenario to improve the processing accuracy of the text processing model on the data in the target application scenario. At the same time, the pruning of the text processing model has a certain impact on the processing accuracy of the text processing model. The pruned text processing model can be optimized and trained through the text sample data of the target application scenario to fine-tune the model parameters in the text processing model and improve the performance of the text processing model.
[0082] In some embodiments, the target application scenario is an abnormal address recognition scenario, and the text sample data of the target application scenario includes historical order address data. Accordingly, the text processing model is used to perform abnormal recognition on the order address data to obtain abnormal classification results of the order address data. In the e-commerce transaction business, the abnormal address data can be a fake order address, etc.
[0083] The technical solution provided in this embodiment obtains a high-precision teacher model through training, and uses the teacher model to perform knowledge distillation on a small-scale text processing model to obtain a small-scale text processing model, simplify the model structure, and achieve lightweight processing of the model.
[0084] On the basis of the above embodiments, the embodiment of the present invention further adopts a preferred example of a model processing method, see Figure 6, Figure 6 It is a flowchart of a model processing method provided by an embodiment of the present invention.
[0085] Through model distillation, the knowledge of the serialized large model A (i.e., the teacher model) is distilled into a relatively small model B (i.e., the text processing model). For model A, a complex network structure is used, such as training the model based on Bert+ downstream tasks. The purpose of this step is to train the best classification model in the anti-fraud scenario. There is no limit on the width and depth of the model. Generally, the number of parameters of model A in this step is generally at the billion level. By using the large model A as the teacher model to train the small model B, the small model B can make the effect of the small model B close to that of the large model A by learning the experience knowledge of the large model A and the knowledge of the sample data, but the number of parameters of the small model B can be reduced to 1 / 10 of the large model A or even smaller.
[0086] Fine-tune the parameters of model B based on the sample data of the target application scenario to improve the processing performance of model B. Specifically, optimize and train model B based on the text sample data and hard labels of the target application scenario, and then optimize and train model B based on the text sample data of the target application scenario through large model A.
[0087] After distillation, the small model B is randomly pruned at each layer to make the model more sparse. The purpose of random pruning is to reduce the overall number of parameters of model B. Based on the NAS concept, 40% is randomly pruned each time to check the classification effect of model B. The solution with better effect in 10 prunings is selected and the training results of this round of small model B are retained. For the non-zero model parameters of each layer in model B after random pruning, the density-first clustering algorithm is used for clustering, so that the remaining non-zero parameters can retain less data and the accuracy can be reduced from Float32 of large model A to Float16, so that the model storage and prediction are loaded value memory occupies less space.
[0088] After pruning, each layer of non-zero parameters is quantized separately (approximately identified), and the small model is fine-tuned again through the soft label of the large model A, so that the effect of the small model B is closer to or better than the large model A. This process is repeated several times until the effect of model B is better and the pruning amount reaches a certain threshold, then pruning is stopped. The pruned small model B is compressed and stored, which can minimize the disk space occupied by the compressed model B and the memory space occupied during inference.
[0089] Specifically, three bits are used to identify the distance between two data. Three bits only indicate that the distance between two data is 1-6. 000 indicates that a line break is required. 111 indicates that the distance between two data exceeds 6, and the subsequent 3-bit distance identifier needs to be added for calculation. After finding the matrix position with data through the interval, the data in the original data table is queried through the adjacent digital data to search for the original data. The number of bits in this part depends on the size of the original data table (generally not more than 16 bits). Figure 2 2 bits are sufficient. The first bit stored indicates whether the first data of the matrix is a number. If it is 1, it means that the first bit of the parameter matrix is data.
[0090] The model is compressed through distillation and pruning, and the compressed storage of sparse models makes the overall storage space (disk) of the model smaller, so that the model can be loaded with less memory, saving hardware resources, and can be used in certain resource-constrained scenarios.
[0091] Figure 7 Schematic diagram of a model processing device provided by an embodiment of the present invention. Figure 7 As shown, the device comprises:
[0092] A model training module 310 is used to train a text processing model based on text sample data;
[0093] A parameter updating module 320 is used to extract the parameter matrix of the text processing model, cluster the model parameters in the parameter matrix to obtain parameter clusters, and perform parameter unification processing on each parameter cluster to obtain an updated parameter matrix;
[0094] The compression storage module 330 is used to compress and store the update parameter matrix based on a preset identifier.
[0095] The technical solution of this embodiment extracts the model parameters of the trained text processing model to form a parameter matrix, clusters the model parameters in the parameter matrix, and unifies the model parameters in each parameter cluster to reduce the number of model parameter values. The model parameters are replaced by preset identifiers to compress and store the updated parameter matrix, reduce the storage space occupied by the parameter matrix, and realize compressed storage of the text processing model.
[0096] Based on the above embodiment, optionally, the parameter updating module 320 is used to:
[0097] For each of the parameter clustering clusters, determining the cluster center data of the parameter clustering cluster;
[0098] The model parameters in the parameter clustering cluster are replaced based on the cluster center data to obtain updated model parameters, wherein the updated model parameters form the updated parameter matrix.
[0099] Optionally, the preset identifier includes a start identifier, a data identifier, a distance identifier, and a line break identifier;
[0100] The compression storage module 330 is used to: set a starting identifier according to the starting data of the update parameter matrix; for each non-zero data of the update parameter matrix, determine the data identifier of the non-zero data, generate a distance identifier based on the distance between the non-zero data and the adjacent non-zero data; and, set a line break identifier when there is a line break, so as to store the update parameter matrix.
[0101] Based on the above embodiment, optionally, the model training module 310 is used to:
[0102] Create an initial text processing model;
[0103] Based on the text sample data, the initial text processing model is subjected to knowledge distillation processing through a pre-trained teacher model to obtain a trained text processing model, wherein the model parameter amount of the text processing model is less than the model parameter amount of the teacher model.
[0104] Optionally, the model training module 310 is further used for:
[0105] Acquire text sample data and a first label of the text sample data; input the text sample data into the teacher model to obtain a second label output by the teacher model, and obtain a target label of the text sample data based on the first label and the second label; input the text sample data into the initial text processing model to obtain a prediction result, and update the parameters of the initial text processing model based on the prediction result and the target label.
[0106] Optionally, the model training module 310 is further used for:
[0107] The text processing model is subjected to multiple random pruning processes to obtain candidate models; the candidate models are evaluated to obtain evaluation data of the candidate models, and the target text processing model is screened from the candidate models based on the evaluation data.
[0108] Optionally, the model training module 310 is further used for:
[0109] Acquire text sample data of a target application scenario, and update parameters of the text processing model based on the text sample data of the target application scenario;
[0110] Among them, the target application scenario is an abnormal address recognition scenario, and the text sample data of the target application scenario includes historical order address data.
[0111] The model processing device provided in the embodiment of the present invention can execute the model processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0112] Figure 8 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device 10 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.
[0113] like Figure 8 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0114] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0115] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as the processing method of the model.
[0116] In some embodiments, the processing method of the model may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the processing method of the model described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the processing method of the model in any other appropriate manner (e.g., by means of firmware).
[0117] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0118] The computer programs for implementing the processing methods of the models of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer programs are executed by the processor, the functions / operations specified in the flow charts and / or block diagrams are implemented. The computer programs may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0119] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to execute a model processing method, the method comprising:
[0120] A text processing model is obtained based on text sample data training; a parameter matrix of the text processing model is extracted, model parameters in the parameter matrix are clustered to obtain parameter clusters, parameters of each parameter cluster are unified to obtain an updated parameter matrix; and the updated parameter matrix is compressed and stored based on a preset identifier.
[0121] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0122] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).
[0123] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.
[0124] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.
[0125] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.
[0126] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for processing a model, characterized in that: include: A text processing model is obtained based on training of text sample data; Extracting a parameter matrix of the text processing model, clustering the model parameters in the parameter matrix to obtain parameter clusters, and performing parameter unification processing on each of the parameter clusters to obtain an updated parameter matrix; The update parameter matrix is compressed and stored based on a preset identifier.
2. The method according to claim 1, characterized in that The step of performing parameter unification processing on each parameter cluster to obtain an updated parameter matrix includes: For each of the parameter clustering clusters, determining the cluster center data of the parameter clustering cluster; The model parameters in the parameter clustering cluster are replaced based on the cluster center data to obtain updated model parameters, wherein the updated model parameters form the updated parameter matrix.
3. The method according to claim 1, characterized in that The preset marks include a start mark, a data mark, a distance mark, and a line break mark; The compressing and storing the update parameter matrix based on the preset identifier includes: Setting the starting identifiers respectively according to the starting data of the update parameter matrix; For each non-zero data of the update parameter matrix, determining a data identifier of the non-zero data, and generating a distance identifier based on a distance between the non-zero data and adjacent non-zero data; And, when a line break exists, a line break flag is set to store the update parameter matrix.
4. The method according to claim 1, characterized in that: The training of a text processing model based on text sample data includes: Create an initial text processing model; Based on the text sample data, the initial text processing model is subjected to knowledge distillation processing through a pre-trained teacher model to obtain a trained text processing model, wherein the model parameter amount of the text processing model is less than the model parameter amount of the teacher model.
5. The method according to claim 4, characterized in that The method of performing knowledge distillation processing on the initial text processing model based on the text sample data through a pre-trained teacher model to obtain a trained text processing model includes: Acquire text sample data and a first label of the text sample data; Inputting the text sample data into the teacher model to obtain a second label output by the teacher model, and obtaining a target label for the text sample data based on the first label and the second label; The text sample data is input into the initial text processing model to obtain a prediction result, and the parameters of the initial text processing model are updated based on the prediction result and the target label.
6. The method according to claim 4, characterized in that The method further comprises: Performing multiple random pruning processes on the text processing model to obtain a candidate model; The candidate models are evaluated respectively to obtain evaluation data of the candidate models, and a target text processing model is screened from the candidate models based on the evaluation data.
7. The method according to claim 4 or 6, characterized in that: After obtaining the text processing model, it also includes: Acquire text sample data of a target application scenario, and update parameters of the text processing model based on the text sample data of the target application scenario; Among them, the target application scenario is an abnormal address recognition scenario, and the text sample data of the target application scenario includes historical order address data.
8. A model processing device, characterized in that: include: A model training module, used for training a text processing model based on text sample data; A parameter updating module, used for extracting the parameter matrix of the text processing model, clustering the model parameters in the parameter matrix to obtain parameter clusters, and performing parameter unification processing on each of the parameter clusters to obtain an updated parameter matrix; The compression storage module is used to compress and store the update parameter matrix based on a preset identifier.
9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the model processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the model processing method described in any one of claims 1 to 7 when executed.