Model compression method and device, equipment and storage medium

Abstract

The invention provides a model compression method and device, equipment and a storage medium, and the method comprises the steps: obtaining a pre-trained complete teacher model and a plurality of first input values, and enabling the first input values to correspond to time sequence financial data at different moments, the teacher model is used for generating training data of the student model according to the first input value; inputting each first input value into a teacher model for calculation to obtain a predicted output value corresponding to each first input value; performing difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value through uniform distribution calculation; and selecting the first input value or a predicted output value corresponding to the first input value as a second input value according to a result of the difference analysis, and training a student model based on a plurality of second input values. The input value of the student model is dynamically adjusted through uniform distribution calculation, and the accuracy and generalization ability of the student model are improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model compression method, device, equipment and storage medium. Background Art

[0002] In recent years, the Internet financial market has developed rapidly. Compared with the traditional financial market, it has higher liquidity and volatility. Therefore, it is necessary to use neural networks to dynamically analyze time-series financial data to achieve dynamic prediction of the Internet financial market. Neural network models require massive amounts of financial data as training samples to obtain a neural network model with good performance and generalization ability. However, a large amount of training data often leads to highly redundant parameters of the neural network model. In order to improve the practicality of the neural network model, knowledge distillation can be used to compress and accelerate large neural network models.

[0003] Common knowledge distillation methods generally use the probability of categories output by the softmax layer of a teacher model with strong generalization ability as the input value for training the student model. However, since the accuracy of the output value of the teacher model is difficult to guarantee, once the output value of the teacher model is wrong, the parameters of the student model will also be affected, making it difficult to ensure the performance and accuracy of the student model obtained by distillation. Summary of the invention

[0004] The main purpose of this application is to provide a model compression method, device, equipment and storage medium, which aims to dynamically adjust the input value of the student model through uniformly distributed calculations, effectively reduce the inaccuracy of student model training data due to errors in the teacher model, improve the training efficiency of the student model, improve the accuracy and generalization ability of the student model, and thus improve the practicality of the model in the Internet financial market.

[0005] In a first aspect, the present application provides a model compression method, the method comprising:

[0006] Obtain a pre-trained teacher model and multiple first input values, wherein the multiple first input values ​​correspond to time series financial data at different times, the teacher model is used to generate training data for the student model according to the first input values, the teacher model is a neural network model trained according to the time series financial data, and the parameters of the student model are smaller than those of the teacher model;

[0007] Input each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values;

[0008] Performing a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value by uniform distribution calculation;

[0009] According to the result of the difference analysis, selecting the first input value or the predicted output value corresponding to the first input value as the second input value, to obtain a plurality of the second input values;

[0010] The student model is trained based on a plurality of the second input values.

[0011] In a second aspect, the present application also provides a compression device for a model, the device comprising:

[0012] A model data acquisition module, used to obtain a pre-trained teacher model and multiple first input values, wherein the multiple first input values ​​correspond to time series financial data at different moments, the teacher model is used to generate training data for a student model based on the first input value, the teacher model is a neural network model trained based on time series financial data, and the parameters of the student model are smaller than those of the teacher model; a predicted output calculation module, used to input each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values; a difference analysis result module, used to perform a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value through uniform distribution calculation; an input value determination module, used to select the first input value or the predicted output value corresponding to the first input value as the second input value based on the result of the difference analysis to obtain multiple second input values; a student model training module, used to train the student model based on multiple second input values.

[0013] In a third aspect, the present application further provides a computer device, the computer device comprising a memory and a processor;

[0014] The memory is used to store computer programs;

[0015] The processor is used to execute the computer program and implement any one of the model compression methods provided in the embodiments of the present application when executing the computer program.

[0016] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the model compression method described in any one of the embodiments of the present application.

[0017] The present application provides a compression method, device, equipment and storage medium for a model. The compression method for the model of the present application includes: obtaining a pre-trained teacher model and multiple first input values, wherein the multiple first input values ​​correspond to time series financial data at different times, the teacher model is used to generate training data for a student model according to the first input value, the teacher model is a neural network model trained according to time series financial data, and the parameters of the student model are smaller than the teacher model; inputting each first input value into the teacher model for calculation to obtain a predicted output value corresponding to each first input value; performing a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value through uniform distribution calculation; selecting the first input value or the predicted output value corresponding to the first input value as the second input value according to the result of the difference analysis to obtain multiple second input values; and training the student model based on multiple second input values. The purpose is to dynamically adjust the input value of the student model through uniform distribution calculation, improve the accuracy of the input value used to train the student model, and thereby improve the accuracy and generalization ability of the student model. It can achieve compression and acceleration of large neural network models while making effective and robust dynamic predictions of the Internet financial market, thereby improving the practicality of the model in the Internet financial market. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 It is a flow chart of a compression method of a model provided in an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of the operation of a teacher model provided in an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the operation of a student model provided in an embodiment of the present application;

[0022] Figure 4 is a schematic block diagram of a compression device of a model provided in an embodiment of the present application;

[0023] Figure 5 It is a schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0026] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] Financial data in the Internet financial market is usually at the millisecond level, including stocks, options, futures in the financial field, and data records from the opening to the closing of each trading day. The historical base and daily increment of such data are relatively large. At the same time, according to the different needs of financial transactions, hundreds of different indicators have been extended from basic financial data. These indicators have different focuses and can also be called financial data. In order to ensure the integrity and reliability of the data, these financial data are usually stored redundantly.

[0029] In order to effectively and robustly predict the dynamic changes in the Internet financial market, it is necessary to train a complete neural network model through massive and highly redundant financial data. The neural network can respond to the input of multiple time series at the same time, obtain financial data that changes over time, and predict the changes in a certain indicator or financial product price at the next moment.

[0030] In model training, complex models and a large amount of computing resources are often required to extract information from large and highly redundant financial data sets. In experiments, the best models are often large in scale, have many parameters, and are difficult to deploy, making them difficult to put into practical use. Therefore, it is necessary to reduce the number of model parameters while ensuring performance, achieve model compression, and thus improve the practicality of models in the Internet financial market.

[0031] The knowledge distillation method is an effective way to achieve model compression. It distills and trains the student model through a teacher model with strong generalization ability, and transfers the generalization ability of the teacher model to the student model. Specifically, in addition to positive examples, the output of the softmax layer also carries a lot of information in the negative label, which can provide more and richer information, such as the degree of similarity between different categories. Therefore, the probability of the category output by the softmax layer of the teacher model can be used as the input value of the student model. However, since the accuracy of the output value of the teacher model is difficult to guarantee, once the output value of the teacher model is wrong, the training parameters of the student model will also be affected, thereby ensuring the performance and accuracy of the student model that is difficult to distill.

[0032] See also Figure 1 , Figure 1 It is a flow chart of a compression method of a model provided in an embodiment of the present application. Among them, the compression method of the model can determine the possibility of error in the output value of the teacher model through uniform distribution calculation, and then dynamically adjust the input value of the student model. Based on this, it can effectively reduce the situation where the student model training data is inaccurate due to errors in the teacher model, improve the training efficiency of the student model, and improve the accuracy and generalization ability of the student model.

[0033] like Figure 1 As shown, the compression method of the model includes steps S101 to S105.

[0034] S101. Obtain a pre-trained teacher model and multiple first input values, wherein the multiple first input values ​​correspond to time series financial data at different times, and the teacher model is used to generate training data for the student model according to the first input values. The teacher model is a neural network model trained according to the time series financial data, and the parameters of the student model are smaller than those of the teacher model.

[0035] The first input value can be generated based on different training samples or can be dynamically generated based on time series financial data. Multiple first input values ​​are equivalent to multiple time series inputs. To obtain financial data that changes over time, each first input value can be distinguished by different moments. For example Figure 2 The y0 in is the first input value corresponding to time 0. It should be understood that financial data changes over time, and effective training data can be obtained by using time series financial data at different times as training samples.

[0036] The teacher model is a fully trained original model with good performance and generalization ability. It can be a relatively complex model or an integration of multiple separately trained models. For example, the teacher model can be a complete neural network model trained with massive and redundant time-series financial data.

[0037] See also Figure 2 , Figure 2 Schematic diagram of the operation of a teacher model provided in the embodiment of the present application. Figure 2 As shown, the teacher model can output P for the first input value, where P is the probability value of the corresponding category output after softmax mapping. It should be noted that the embodiment of the present application does not impose any restrictions on the teacher model in terms of model architecture, parameter quantity, and whether it is integrated.

[0038] Among them, the student model is a simplified model, which is a single model with a small number of parameters and a relatively simple model structure. Figure 3 , Figure 3 Schematic diagram of the operation of a student model provided in the embodiment of the present application. Figure 3 As shown, the student model can output P for the second input value, where P is the probability value of the corresponding category output after softmax mapping.

[0039] Specifically, a fully trained teacher model and a first input value are obtained to generate training data for a student model.

[0040] S102: Input each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values.

[0041] Among them, the predicted output value is the class with the largest predicted probability among the probability values ​​of the corresponding categories output by the teacher model.

[0042] Specifically, each first input value is input into the teacher model for calculation to obtain the first output value of the softmax layer output of the teacher model, and the predicted output value is determined by the probability value corresponding to the corresponding category in the first output value.

[0043] In some embodiments, the step of inputting each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values ​​includes: inputting the first input value into the teacher model to obtain a first output value output by the softmax layer of the teacher model; and calculating the first output value using the argmax function to obtain a predicted output value corresponding to the first input value.

[0044] Specifically, the input value is calculated by a fully trained teacher model to obtain a first output value, and the probability value of the corresponding category in the first output value is calculated using the argmax function to obtain an output value with the maximum probability value as the predicted output value.

[0045] like Figure 2As shown, the teacher model passes through the first input value y0, y1, y2, y3, y4, y5, outputs the first output value P0, P1, P2, P3, P4, P5, and uses the argmax function to calculate P0, P1, P2, P3, P4, P5 respectively to obtain the predicted output value corresponding to the first input value

[0046] S103: Perform difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value through uniform distribution calculation.

[0047] Specifically, in order to improve the accuracy of the input values ​​used to train the student model, the difference between the output prediction value of the teacher model at each moment and the first input value at each moment is calculated using uniform distribution to determine whether each time series financial data matches the output of the teacher model, and then determine whether the output of the teacher model is accurate.

[0048] In some embodiments, the first input value at each moment and the predicted output value corresponding to the first input value are subjected to difference analysis through uniform distribution calculation, including: obtaining the current moment corresponding to the first input value to be subjected to uniform distribution calculation, the multiple first input values ​​respectively corresponding to different moments; comparing whether the first input value at the current moment and the predicted output value corresponding to the first input value at all moments before the current moment are the same, and obtaining multiple comparison results; calculating the cumulative value at the current moment according to the multiple comparison results; and performing uniform distribution calculation based on the cumulative value to obtain the result of the difference analysis.

[0049] The accumulated value is used to characterize the accuracy of the comparison between the plurality of first input values ​​and the predicted output values ​​corresponding to the first input values.

[0050] Specifically, in order to verify the accuracy of the input values ​​used to train the student model, the first input values ​​at the current moment and the previous moment and the predicted output value output by the teacher model based on the first input value are compared respectively to determine whether the financial data at the current moment is the same as the predicted output value calculated by the teacher model. The comparison result is used to record whether the two are the same, and then the comparison results at the previous moment and the current moment are accumulated. Based on this, the average value of the comparison results at the previous moment and the current moment is calculated to obtain a cumulative value used to characterize the accuracy, so as to evaluate the time series financial data based on the financial data at the previous moment and the current moment, and then the cumulative value is uniformly distributed to obtain the difference between the output prediction value of the teacher model and the first input value, that is, to determine the difference between the financial data at the current moment and the predicted output value calculated by the teacher model.

[0051] For example, Figure 2 As shown, if the current time to be calculated is 2, compare y0 with y1 and y2 and And get multiple comparison results, then calculate y0 and y1 and y2 and The cumulative value of the comparison results.

[0052] It should be understood that quasi-cyclical phenomena are common in financial data, and their changing patterns essentially correspond to the cyclical patterns of social and economic development. By calculating the cumulative value of the comparison results at the current moment and the previous moment, the accuracy at the current moment can be obtained, and then the accuracy at the current moment can be used to predict whether the time series financial data at the current moment matches the output of the teacher model, which can effectively reduce the uncertainty of predictive calculations in the financial field.

[0053] In another embodiment, the uniform distribution calculation is used to perform a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value, including: comparing whether multiple first input values ​​and the predicted output values ​​corresponding to the first input value are the same, and obtaining multiple comparison results; obtaining the current moment corresponding to the first input value to be uniformly distributed, and the multiple first input values ​​correspond to different moments respectively; calculating the cumulative value of the current moment based on the comparison result between the current moment and all moments before the current moment; performing uniform distribution calculation based on the cumulative value to obtain the result of the difference analysis.

[0054] Specifically, in order to verify the accuracy of the input values ​​used to train the student model, the first input value and the predicted output value of the teacher model based on the first input value are compared respectively. The comparison result is used to record whether the two are the same. The comparison results of the previous moment and the current moment are accumulated. Based on this, the average value of the comparison results of the previous moment and the current moment is calculated to obtain the cumulative value used to characterize the accuracy rate. The cumulative value is evenly distributed to obtain the difference between the output prediction value of the teacher model and the first input value.

[0055] For example, Figure 2 As shown, y0 and y1 and y2 and y3 and y4 and y5 and And get multiple comparison results. If the current time to be calculated is 2, calculate y0 and y1 and y2 and The cumulative value of the comparison results.

[0056] It should be noted that in actual training, a complete teacher model can only be obtained by training with massive and redundant time-series financial data. Therefore, the number of first input values ​​can be determined according to actual needs. Figure 2 and Figure 3 It does not constitute a limitation on the number of first input values. It should be understood that as time gradually increases, the time series financial data continues to increase, the number of first input values ​​used to calculate the cumulative value and the number of predicted output values ​​corresponding to the first input values ​​is also increasing, and the overall situation of the corresponding multiple comparison results is changing. Therefore, the cumulative value is also constantly changing, and the cumulative values ​​at different times may be the same or different.

[0057] In some embodiments, the calculation formula of the cumulative value is:

[0058]

[0059] in, is the cumulative value of all moments before and including time t, T is the total number of moments before and including time t, y t is the first input value at time t, is the predicted output value corresponding to the first input value at time t, is an indicator function used to determine whether the first input value and the predicted output value corresponding to the first input value are the same.

[0060] Specifically, according to Get the comparison result of whether the financial data at the current moment and the previous moment are the same as the predicted output value calculated by the teacher model, accumulate the comparison results of the previous moment and the current moment, and based on this, calculate the average value of the comparison results of the previous moment and the current moment to get the cumulative value used to characterize the accuracy It is used to characterize the accuracy of the current moment, so as to evaluate the time series financial data based on the financial data of the previous moment and the current moment, and then to perform uniform distribution calculation. It should be noted that the result of the difference analysis depends not only on At time t, whether t Equal, and also related to the comparison results of all moments before moment t.

[0061] In some embodiments, the formula for calculating the uniform distribution is:

[0062]

[0063] Wherein, N is the calculation result of 0-1 distribution, which is used to determine the second input value of the student model, β is a parameter value of [0,1], which decreases dynamically with the increase of time, and μ is the statistical mean value of the random variable. is the accumulated value, and T is the time.

[0064] Specifically, the accumulated value As the variance, the uniform distribution is calculated to obtain the difference between the output prediction value of the teacher model and the first input value. The value of β will change linearly with time to dynamically adjust according to the time series financial data at different times, and then judge the current moment according to the calculated value. Is there a risk of error?

[0065] In some embodiments, the comparison of the first input value at the current moment and all moments before the current moment and the predicted output value corresponding to the first input value is performed to determine whether they are the same, to obtain multiple comparison results, including: inputting the first input value and the predicted output value corresponding to the first input value into the indicator function, to determine whether the first input value and the predicted output value corresponding to the first input value are the same; if the comparison result is the same, the output value of the indicator function is 1; if the comparison result is not the same, the output value of the indicator function is 0.

[0066] The indicator function is used to count the number of times a certain assertion or condition is met. When the input is True, the output result is 1, and when the input is False, the output result is 0.

[0067] Specifically, the first input value and the predicted output value corresponding to the first input value are input into the indicator function, and the indicator function determines whether the first input value and the predicted output value corresponding to the first input value are the same. If they are the same, it is True and the output is 1; if they are different, it is False and the output is 0. For example, y0 and Input the indicator function, the indicator function judges to be False, and the output is 0. Input the indicator function, if the indicator function is judged as True, the output is 1.

[0068] S104. According to the result of the difference analysis, select the first input value or the predicted output value corresponding to the first input value as the second input value, to obtain a plurality of the second input values.

[0069] Among them, the second input value is used to train the student model.

[0070] Specifically, according to the results of the difference analysis, it is judged whether the predicted output value of the teacher model is accurate. If it is accurate, the predicted output value is used as the second input value, which is conducive to the student model learning the generalization ability of the teacher model. If it is inaccurate, the original first input value is used as the second input value to ensure the accuracy of the training data. The second input value for training the student model can be flexibly selected based on uniform distribution calculation, which can improve the accuracy of the student model by improving the accuracy of the training data while improving the generalization ability of the student model.

[0071] In some embodiments, in order to dynamically adjust the input value of the student model, the first input value or the predicted output value corresponding to the first input value is selected as the second input value according to the result of the difference analysis, and multiple second input values ​​are obtained, including: if the result of the difference analysis is 0, the first input value is used as the second input value; if the result of the difference analysis is 1, the predicted output value corresponding to the first input value is used as the second input value.

[0072] Specifically, the difference analysis result of the uniform distribution calculation is determined to be the calculation result of the 0-1 distribution, and the calculation result is 1 or 0, representing yes or no. When the result of the difference analysis is 0, it is judged that the predicted output value of the teacher model may be wrong, and the first input value is used as the second input value to ensure the accuracy of the training data; if the result of the difference analysis is 1, it is judged that the predicted output value of the teacher model is unlikely to be wrong, and the predicted output value corresponding to the first input value is used as the second input value to improve the generalization ability of the student model.

[0073] For example, Figure 3 As shown, when t is 1, the result of the difference analysis is 0, and the first input value y1 is used as the second input value. When t is 3, the result of the difference analysis is 1, and the predicted output value and as the second input value. Determine each second input value according to the distribution of the difference analysis result, and then obtain the second input values ​​respectively as “y0, y2, y3, ". It can be seen that the second input value includes part of the first input value and part of the predicted output value. Under the premise of ensuring the accuracy of the training data, using the predicted output value output by the teacher model as training data to train the student model can effectively improve the generalization ability of the student model.

[0074] S105. Train the student model based on multiple second input values.

[0075] Specifically, multiple second input values ​​are input into the student model, and the parameters of the student model are continuously optimized and trained. Figure 3 As shown, multiple second input values ​​are respectively input into the student model, and the obtained second output values ​​of the student model are the same as the first output values ​​P0, P1, P2, P3, P4, and P5 of the teacher model. After dynamically adjusting the second input values ​​of the student model, the accuracy of the input values ​​used for training can be effectively improved, thereby improving the accuracy and generalization ability of the student model.

[0076] In some embodiments, the training of the student model based on multiple second input values ​​includes: inputting multiple second input values ​​into the student model to obtain a second output value output by the softmax layer of the student model; calculating the second output value using the argmax function to obtain a predicted output value corresponding to the second input value; training the parameters of the student model based on a first cross entropy loss function and a second cross entropy loss function, the first cross entropy loss function being used to calculate the error between each second input value and the predicted output value corresponding to the second input value, and the second cross entropy loss function being used to calculate the error between the first output value and the second output value.

[0077] Specifically, multiple second input values ​​are input into the student model, the second output value and the predicted output value corresponding to the second input value are obtained, the error between each second input value and the predicted output value corresponding to the second input value is calculated by the first cross entropy loss function, the error between the first output value and the second output value is calculated by the second cross entropy loss function, and the first cross entropy loss function and the second cross entropy loss function are added to train the parameters in the student model.

[0078] The student model obtained by distillation can effectively and robustly predict the dynamic changes of the Internet financial market. At the same time, it is much smaller than the teacher model, which can effectively improve the practicality of the model in the Internet financial market.

[0079] See also Figure 4 , Figure 4 It is a schematic block diagram of a model compression device provided in an embodiment of the present application. The model compression device can be configured in a server to execute the aforementioned model compression method.

[0080] like Figure 4 As shown, the compression device 200 of the model includes: a model data acquisition module 201, a prediction output calculation module 202, a difference analysis result module 203, an input value determination module 204, and a student model training module 205.

[0081] The model data acquisition module 201 is used to acquire a pre-trained teacher model and a plurality of first input values, wherein the plurality of first input values ​​correspond to time series financial data at different moments, the teacher model is used to generate training data of the student model according to the first input values, the teacher model is a neural network model trained according to the time series financial data, and the parameters of the student model are smaller than those of the teacher model;

[0082] A prediction output calculation module 202, used for inputting each of the first input values ​​into the teacher model for calculation to obtain a prediction output value corresponding to each of the first input values;

[0083] A difference analysis result module 203 is used to perform a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value by uniform distribution calculation;

[0084] An input value determination module 204, configured to select the first input value or a predicted output value corresponding to the first input value as a second input value according to the result of the difference analysis, and obtain a plurality of second input values;

[0085] The student model training module 205 is used to train the student model based on multiple second input values.

[0086] Exemplarily, the predicted output calculation module 202 further includes a first output value submodule and a second output value submodule.

[0087] A first output value submodule, used for inputting the first input value into the teacher model to obtain a first output value output by the softmax layer of the teacher model;

[0088] The first output value submodule is used to calculate the first output value using the argmax function to obtain a predicted output value corresponding to the first input value.

[0089] Exemplarily, the difference analysis result module 203 further includes a current moment submodule, a comparison result submodule, a cumulative value submodule, and a difference analysis submodule.

[0090] A current time submodule, used to obtain the current time corresponding to the first input value to be uniformly distributed, wherein the multiple first input values ​​correspond to different times respectively;

[0091] A comparison result submodule, used to compare whether the first input value at the current moment and the first input value at all moments before the current moment and the predicted output value corresponding to the first input value are the same, and obtain multiple comparison results;

[0092] The cumulative value submodule is used to calculate the cumulative value at the current moment according to multiple comparison results;

[0093] The difference analysis submodule is used to perform uniform distribution calculation based on the cumulative value to obtain the result of the difference analysis.

[0094] Exemplarily, the compression device of the model also includes a cumulative value calculation module and a uniform distribution calculation module.

[0095] The cumulative value calculation module is used for calculating the cumulative value as follows:

[0096]

[0097] in, is the cumulative value of all moments before and including time t, T is the total number of moments before and including time t, y t is the first input value at time t, is the predicted output value corresponding to the first input value at time t, is an indicator function, used to determine whether the first input value and the predicted output value corresponding to the first input value are the same;

[0098] Uniform distribution calculation module, the formula used for the uniform distribution calculation is:

[0099]

[0100] Wherein, N is the calculation result of 0-1 distribution, which is used to determine the second input value of the student model, β is a parameter value of [0,1], which decreases dynamically with the increase of time, and μ is the statistical mean value of the random variable. is the accumulated value, and T is the time.

[0101] Exemplarily, the comparison result submodule further includes an indication function submodule, a same result submodule, and a different result submodule.

[0102] an indication function submodule, configured to input the first input value and the predicted output value corresponding to the first input value into the indication function, and determine whether the first input value and the predicted output value corresponding to the first input value are the same;

[0103] The same result submodule is used for outputting the indicator function with a value of 1 if the comparison result is the same;

[0104] The result different submodule is used for, if the comparison result is different, the output value of the indicator function is 0.

[0105] Exemplarily, the input value determination module 204 further includes a first analysis result submodule and a second analysis result submodule.

[0106] a first analysis result submodule, configured to use the first input value as the second input value if the result of the difference analysis is 0;

[0107] The second analysis result submodule is used to use the predicted output value corresponding to the first input value as the second input value if the result of the difference analysis is 1.

[0108] Exemplarily, the student model training module 205 also includes a second output value submodule, a second prediction output submodule, and a loss function submodule.

[0109] A second output value submodule, used for inputting a plurality of the second input values ​​into the student model to obtain a second output value output by the softmax layer of the student model;

[0110] A second prediction output submodule, used to calculate the second output value using an argmax function to obtain a predicted output value corresponding to the second input value;

[0111] A loss function submodule is used to train the parameters of the student model based on a first cross entropy loss function and a second cross entropy loss function, wherein the first cross entropy loss function is used to calculate the error between each second input value and the predicted output value corresponding to the second input value, and the second cross entropy loss function is used to calculate the error between the first output value and the second output value.

[0112] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0113] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0114] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 5 Runs on the computer device shown.

[0115] See also Figure 5 , Figure 5 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device may be a terminal device or a server.

[0116] like Figure 5 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0117] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any model compression method.

[0118] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0119] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute the compression method of any model.

[0120] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0121] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0122] In some embodiments, the processor is used to run a computer program stored in the memory to implement the following steps:

[0123] Obtain a pre-trained teacher model and multiple first input values, wherein the multiple first input values ​​correspond to time series financial data at different times, the teacher model is used to generate training data for the student model according to the first input values, the teacher model is a neural network model trained according to the time series financial data, and the parameters of the student model are smaller than those of the teacher model;

[0124] Input each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values;

[0125] Performing a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value by uniform distribution calculation;

[0126] According to the result of the difference analysis, selecting the first input value or the predicted output value corresponding to the first input value as the second input value, to obtain a plurality of the second input values;

[0127] The student model is trained based on a plurality of the second input values.

[0128] In some embodiments, when the processor implements inputting each of the first input values ​​into the teacher model for calculation to obtain the predicted output value corresponding to each of the first input values, it is used to implement: inputting the first input value into the teacher model to obtain the first output value output by the softmax layer of the teacher model; using the argmax function to calculate the first output value to obtain the predicted output value corresponding to the first input value.

[0129] In some embodiments, when the processor performs a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value through uniform distribution calculation, it is used to implement: obtaining the current moment corresponding to the first input value to be subjected to uniform distribution calculation, the multiple first input values ​​respectively corresponding to different moments; comparing whether the first input value at the current moment and the predicted output value corresponding to the first input value at all moments before the current moment are the same, and obtaining multiple comparison results; calculating the cumulative value at the current moment based on the multiple comparison results; performing uniform distribution calculation based on the cumulative value to obtain the result of the difference analysis.

[0130] In some embodiments, when the processor compares the first input value at the current moment and all moments before the current moment and the predicted output value corresponding to the first input value to see whether they are the same, and obtains multiple comparison results, it is used to implement: inputting the first input value and the predicted output value corresponding to the first input value into the indicator function to determine whether the first input value and the predicted output value corresponding to the first input value are the same; if the comparison result is the same, the output value of the indicator function is 1; if the comparison result is not the same, the output value of the indicator function is 0.

[0131] In some embodiments, when the processor selects the first input value or the predicted output value corresponding to the first input value as the second input value according to the result of the difference analysis to obtain multiple second input values, it is used to implement: if the result of the difference analysis is 0, then the first input value is used as the second input value; if the result of the difference analysis is 1, the predicted output value corresponding to the first input value is used as the second input value.

[0132] In some embodiments, when the processor implements training the student model based on multiple second input values, it is used to implement: inputting multiple second input values ​​into the student model to obtain the second output value output by the softmax layer of the student model; calculating the second output value using the argmax function to obtain the predicted output value corresponding to the second input value; training the parameters of the student model based on the first cross entropy loss function and the second cross entropy loss function, the first cross entropy loss function is used to calculate the error between each second input value and the predicted output value corresponding to the second input value, and the second cross entropy loss function is used to calculate the error between the first output value and the second output value.

[0133] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, any model compression method provided in the embodiment of the present application is implemented.

[0134] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0135] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc.

[0136] The present invention refers to a new application model of computer technologies such as storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. of the blockchain language model. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the blockchain underlying platform, platform product service layer, and application service layer.

[0137] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A model compression method, characterized in that: The method comprises: Obtain a pre-trained teacher model and multiple first input values, wherein the multiple first input values ​​correspond to time series financial data at different times, the teacher model is used to generate training data for the student model according to the first input values, the teacher model is a neural network model trained according to the time series financial data, and the parameters of the student model are smaller than those of the teacher model; Input each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values; Performing a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value by uniform distribution calculation; According to the result of the difference analysis, selecting the first input value or the predicted output value corresponding to the first input value as the second input value, to obtain a plurality of the second input values; The student model is trained based on a plurality of the second input values.

2. The method according to claim 1, characterized in that The step of inputting each of the first input values ​​into the teacher model for calculation to obtain a predicted output value corresponding to each of the first input values ​​includes: Inputting the first input value into the teacher model to obtain a first output value output by the softmax layer of the teacher model; The first output value is calculated using the argmax function to obtain a predicted output value corresponding to the first input value.

3. The method according to claim 1, characterized in that The performing difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value by uniform distribution calculation includes: Obtaining a current time corresponding to a first input value to be subjected to uniform distribution calculation, wherein the plurality of first input values ​​correspond to different times respectively; Comparing whether the first input value at the current moment and at all moments before the current moment and the predicted output value corresponding to the first input value are the same, to obtain multiple comparison results; Calculating the cumulative value at the current moment according to the multiple comparison results; A uniform distribution calculation is performed based on the cumulative value to obtain the result of the difference analysis.

4. The method according to claim 3, characterized in that: The calculation formula of the cumulative value is: in, is the cumulative value of all moments before and including time t, T is the total number of moments before and including time t, y t is the first input value at time t, is the predicted output value corresponding to the first input value at time t, is an indicator function, used to determine whether the first input value and the predicted output value corresponding to the first input value are the same; The formula for calculating the uniform distribution is: Wherein, N is the calculation result of 0-1 distribution, which is used to determine the second input value of the student model, β is a parameter value of [0,1], which decreases dynamically with the increase of time, and μ is the statistical mean value of the random variable. is the accumulated value, and T is the time.

5. The method according to claim 4, characterized in that The comparing whether the first input value at the current moment and the first input value at all moments before the current moment and the predicted output value corresponding to the first input value are the same, obtaining multiple comparison results, including: Inputting the first input value and the predicted output value corresponding to the first input value into the indicator function, and determining whether the first input value and the predicted output value corresponding to the first input value are the same; If the comparison result is the same, the output value of the indicator function is 1; If the comparison result is not the same, the output value of the indicator function is 0.

6. The method according to claim 4, characterized in that The step of selecting the first input value or the predicted output value corresponding to the first input value as the second input value according to the result of the difference analysis, and obtaining a plurality of the second input values, comprises: If the result of the difference analysis is 0, the first input value is used as the second input value; If the result of the difference analysis is 1, the predicted output value corresponding to the first input value is used as the second input value.

7. The method according to claim 1, characterized in that The step of training the student model based on the plurality of second input values ​​comprises: Inputting a plurality of the second input values ​​into the student model to obtain a second output value output by the softmax layer of the student model; Calculating the second output value using the argmax function to obtain a predicted output value corresponding to the second input value; The parameters of the student model are trained based on a first cross entropy loss function and a second cross entropy loss function, wherein the first cross entropy loss function is used to calculate the error between each second input value and the predicted output value corresponding to the second input value, and the second cross entropy loss function is used to calculate the error between the first output value and the second output value.

8. A compression device for a model, characterized in that: include: A model data acquisition module, used to acquire a pre-trained teacher model and a plurality of first input values, wherein the plurality of first input values ​​correspond to time series financial data at different moments, the teacher model is used to generate training data for a student model according to the first input values, the teacher model is a neural network model trained according to the time series financial data, and the parameters of the student model are smaller than those of the teacher model; A prediction output calculation module, used for inputting each of the first input values ​​into the teacher model for calculation to obtain a prediction output value corresponding to each of the first input values; A difference analysis result module, used for performing a difference analysis on the first input value at each moment and the predicted output value corresponding to the first input value by uniform distribution calculation; an input value determination module, configured to select the first input value or a predicted output value corresponding to the first input value as a second input value according to a result of the difference analysis, and obtain a plurality of the second input values; A student model training module is used to train the student model based on multiple second input values.

9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is used to execute the computer program and implement the model compression method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the model compression method according to any one of claims 1 to 7.

Images