Data classification method, model compression method, device, equipment and program product

Abstract

The application discloses a data classification method, a model compression method, a device, equipment and a program product, and relates to the technical field of artificial intelligence. The model compression method comprises the following steps: acquiring a global model and local data, determining the input dimension of the global model, acquiring a feature expansion module related to the global model, performing dimensionality increasing processing on the local data according to the feature expansion module, obtaining feature expansion data with the same input dimension as the global model, determining an initial local model, and performing knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model, to obtain a target local model. The model compression method provided by the application can match the input dimension under the premise of not losing information through the feature expansion module when the model is compressed, and the generalization ability of the compressed model is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent classification, and in particular to a data classification method, a model compression method, an apparatus, a device and a program product. BACKGROUND

[0002] At present, in order to realize various intelligent classification systems, a classification model is usually constructed based on deep learning. Generally, a global model is first constructed. Since the model input dimension of a small device such as a portable device is generally different from the input dimension of the global model, the global model cannot be directly deployed on the portable device. Therefore, the global model needs to be compressed to obtain a compressed model, and the compressed model is applied to the local portable device. For example, in order to realize an intelligent classification system for cardiovascular diseases, a cardiovascular disease classification model is usually constructed based on deep learning. The model is finally deployed on different types of medical nodes to realize intelligent classification of cardiovascular diseases in various medical environments. The global model is usually a comprehensive detection of chest leads and limb leads. The portable device is usually a limb lead device. There is a difference in the model input dimension. The global model cannot be directly deployed on these devices. Therefore, the global model needs to be compressed by using a model compression mechanism and then applied to the portable device.

[0003] Traditional model compression techniques include model pruning and knowledge distillation. However, the model pruning technique generally only focuses on the model structure itself, and the knowledge distillation relies on the knowledge transfer of the teacher model. These model compression techniques are not sensitive to changes in the input dimension. When the input dimension of the required compressed model is different from that of the global model, important information is often lost during model compression, and the generalization ability of the compressed model is weak. SUMMARY

[0004] The main purpose of the present application is to provide a data classification method, a model compression method, an apparatus, a device and a program product, which aims to solve the technical problem of how to match the input dimension in the process of model compression without losing information, thereby improving the generalization ability of the compressed model.

[0005] To achieve the above-mentioned purpose, the embodiment of the present application provides a model compression method, which comprises the following steps:

[0006] obtaining a global model and local data, and determining the input dimension of the global model;

[0007] obtaining a feature expansion module related to the global model, performing dimensionality increasing processing on the local data according to the feature expansion module, and obtaining feature expansion data with the same input dimension as the global model;

[0008] An initial local model is obtained, and knowledge distillation is performed on the initial local model according to the local data, the feature expansion data, and the global model, to obtain a target local model.

[0009] Optionally, the global model is a global teacher model, and the step of obtaining the feature expansion module related to the global model comprises:

[0010] A convolutional neural network model and a local teacher model trained by the local data are obtained.

[0011] The local data is input into the local teacher model, and the output of the local teacher model is input into the convolutional neural network model and the global teacher model, respectively.

[0012] A first joint loss function is calculated according to the outputs of the convolutional neural network model and the global teacher model, and the convolutional neural network model is trained according to the first joint loss function, to obtain the feature expansion module.

[0013] Optionally, the initial local model is an initial local student model, and the step of performing knowledge distillation on the initial local model according to the local data, the feature expansion data, and the global model to obtain a target local model comprises:

[0014] The local data is input into the local teacher model and the initial local student model, respectively, and the feature expansion data is input into the global teacher model.

[0015] A second joint loss function is calculated according to the outputs of the local teacher model, the initial local student model, and the global teacher model.

[0016] The initial local student model is trained according to the second joint loss function, to obtain the target local model.

[0017] Optionally, the feature expansion module comprises a feature dimension increasing module, and the step of performing dimension increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model comprises:

[0018] The local data is input into the feature dimension increasing module for dimension increasing processing, to obtain the feature expansion data with the same dimension as the input dimension of the global model.

[0019] Alternatively, the feature expansion module comprises the feature dimension increasing module and a feature expansion module, and the step of performing dimension increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model comprises:

[0020] The local data is input into the feature dimension increasing module and the feature expansion module respectively for dimension increasing processing, the output data of the feature dimension increasing module and the feature expansion module are fused to obtain the feature expansion data with the same dimension as the input dimension of the global model.

[0021] Optionally, the step of obtaining the global model comprises:

[0022] obtaining a deep neural network model, and obtaining global data based on a federated learning mechanism;

[0023] training the deep neural network model based on standard supervised learning using a federated training framework according to the global data to obtain the global model.

[0024] The embodiment of the application further provides a data classification method, which comprises the following steps:

[0025] obtaining to-be-classified data;

[0026] inputting the to-be-classified data into a target local model created in advance to obtain a classification result, wherein the target local model is obtained by performing dimension increasing processing on local data using a feature expansion module to obtain feature expansion data with the same dimension as the input dimension of a global model, and then performing knowledge distillation on an initial local model according to the local data, the feature expansion data and the global model.

[0027] outputting the classification result.

[0028] Optionally, before the step of inputting the to-be-classified data into the target local model created in advance, the method further comprises:

[0029] obtaining the global model and the local data, and determining the input dimension of the global model;

[0030] obtaining the feature expansion module related to the global model, performing dimension increasing processing on the local data according to the feature expansion module to obtain the feature expansion data with the same dimension as the input dimension of the global model;

[0031] obtaining the initial local model, and performing knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain the target local model.

[0032] In addition, the embodiment of the application further provides a model compression device, which comprises:

[0033] an obtaining module, configured to obtain a global model and local data, and determine the input dimension of the global model;

[0034] The data dimension increasing module is configured to acquire a feature expansion module related to the global model, and perform dimension increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model.

[0035] The knowledge distillation module is configured to acquire an initial local model, and perform knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain a target local model.

[0036] In addition, the embodiment of the present application also provides a terminal device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the computer program is executed by the processor to implement the model compression method or the data classification method.

[0037] In addition, the embodiment of the present application also provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the model compression method or the data classification method.

[0038] The data classification method, the model compression method, the device, the equipment and the program product provided by the embodiment of the present application can acquire a global model and local data, determine the input dimension of the global model, acquire a feature expansion module related to the global model, perform dimension increasing processing on the local data according to the feature expansion module, obtain feature expansion data with the same input dimension as the global model, determine an initial local model, perform knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model, and obtain a target local model. The model compression method provided by the present application can match the input dimension under the premise of not losing information when the model is compressed, and the generalization ability of the compressed model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 It is a functional module schematic diagram of a terminal device to which the model compression device of the present application belongs.

[0040] Figure 2 It is a functional module schematic diagram of a terminal device to which the data classification device of the present application belongs.

[0041] Figure 3 It is a flowchart of the first embodiment of the model compression method of the present application.

[0042] Figure 4 It is a flowchart of the second embodiment of the model compression method of the present application.

[0043] Figure 5 It is a flowchart of the third embodiment of the model compression method of the present application.

[0044] Figure 6 a flowchart of a fifth embodiment of the model compression method of the present application;

[0045] Figure 7 a flowchart of a first embodiment of the data classification method of the present application;

[0046] Figure 8 a flowchart of a second embodiment of the data classification method of the present application;

[0047] Figure 9 a functional module schematic diagram of a preferred embodiment of the model compression device of the present application.

[0048] The implementation, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0050] The main solution of the embodiments of the present application is: obtaining a global model and local data, determining the input dimension of the global model, obtaining a feature expansion module related to the global model, performing dimensionality processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model, determining an initial local model, and performing knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain a target local model. The model compression method proposed in the present application can match the input dimension under the premise of not losing information when compressing the model, thereby improving the generalization ability of the compressed model.

[0051] The embodiments of the present application consider that in the prior related schemes, the traditional model compression technology includes model pruning technology and knowledge distillation, but the model pruning technology generally only focuses on the model structure itself, and the knowledge distillation relies on the knowledge transfer of the teacher model. These model compression technologies are not sensitive to the change of the input dimension, and when the input dimension of the compressed model needed is different from that of the global model, important information is often lost during model compression, and the generalization ability of the compressed model is weak.

[0052] Therefore, the embodiments of the present application propose a solution, which can match the input dimension under the premise of not losing information in the process of model compression, thereby improving the generalization ability of the compressed model.

[0053] Specifically, referring to Figure 1 , Figure 1The figure is a functional module diagram of a terminal device to which the model compression device belongs. The model compression device can be a device independent of the terminal device and capable of realizing model compression, which can be carried on the terminal device in the form of hardware or software. The terminal device can be a smart mobile terminal such as a mobile phone or a tablet computer, and can also be a network device such as a server.

[0054] In the embodiment, the terminal device to which the model compression device belongs at least includes an output module 110, a processor 120, a memory 130, and a communication module 140.

[0055] The memory 130 stores an operating system and a model compression program; the output module 110 can be a display screen, a loudspeaker, etc. The communication module 140 can include a WIFI module, a mobile communication module, a Bluetooth module, etc., and communicates with external devices or servers through the communication module 140.

[0056] As an embodiment, the model compression program in the memory 130 is executed by the processor to realize the following steps:

[0057] Obtain a global model and local data, and determine the input dimension of the global model;

[0058] Obtain a feature expansion module related to the global model, perform dimensionality expansion processing on the local data according to the feature expansion module, and obtain feature expansion data with the same input dimension as the global model;

[0059] Obtain an initial local model, perform knowledge distillation on the initial local model according to the local data, the feature expansion data, and the global model, and obtain a target local model.

[0060] Further, the model compression program in the memory 130 is executed by the processor to realize the following steps:

[0061] Obtain a convolutional neural network model and a local teacher model trained by the local data;

[0062] Input the local data into the local teacher model, and input the output of the local teacher model into the convolutional neural network model and the global teacher model, respectively;

[0063] Calculate a first joint loss function according to the outputs of the convolutional neural network model and the global teacher model, train the convolutional neural network model according to the first joint loss function, and obtain the feature expansion module.

[0064] Further, the model compression program in the memory 130 is executed by the processor to realize the following steps:

[0065] input the local data into the local teacher model and the initial local student model respectively, and input the feature expansion data into the global teacher model;

[0066] calculate a second joint loss function according to outputs of the local teacher model, the initial local student model and the global teacher model;

[0067] train the initial local student model according to the second joint loss function to obtain the target local model.

[0068] Further, the feature expansion module comprises a feature dimension increasing module, and the model compression program in the memory 130 is further implemented by the processor to implement the following steps:

[0069] input the local data into the feature dimension increasing module for dimension increasing processing to obtain the feature expansion data with the same input dimension as the global model;

[0070] Alternatively, the feature expansion module comprises the feature dimension increasing module and a feature expansion module, and the step of performing dimension increasing processing on the local data by the feature expansion module to obtain the feature expansion data with the same input dimension as the global model comprises:

[0071] input the local data into the feature dimension increasing module and the feature expansion module respectively for dimension increasing processing, fuse the output data of the feature dimension increasing module and the feature expansion module to obtain the feature expansion data with the same input dimension as the global model.

[0072] Further, the model compression program in the memory 130 is further implemented by the processor to implement the following steps:

[0073] obtain a deep neural network model, and obtain global data based on a federated learning mechanism;

[0074] train the deep neural network model based on standard supervised learning by using a federated training framework according to the global data to obtain the global model.

[0075] Specifically, referring to Figure 2 , Figure 2 is a functional module schematic diagram of a terminal device to which the data classification device belongs. The data classification device can be a device independent of the terminal device and capable of realizing data classification, which can be carried on the terminal device in the form of hardware or software. The terminal device can be a smart mobile terminal such as a mobile phone or a tablet computer, and can also be a network device such as a server.

[0076] In the embodiment, the terminal device to which the data classification apparatus belongs at least comprises an output module 110, a processor 120, a memory 130 and a communication module 140.

[0077] The memory 130 stores an operating system and a data classification program; the output module 110 can be a display screen, a loudspeaker or the like. The communication module 140 can comprise a WIFI module, a mobile communication module and a Bluetooth module or the like, and communicates with external devices or servers through the communication module 140.

[0078] As an embodiment, the data classification program in the memory 130 is executed by the processor to implement the following steps:

[0079] obtaining to-be-classified data;

[0080] inputting the to-be-classified data into a pre-created target local model to obtain a classification result, wherein the target local model is obtained by using a feature expansion module to perform dimensionality increasing processing on local data to obtain feature expansion data with the same input dimension as a global model, and then performing knowledge distillation on an initial local model according to the local data, the feature expansion data and the global model;

[0081] outputting the classification result.

[0082] Further, the data classification program in the memory 130 is executed by the processor to implement the following steps:

[0083] obtaining the global model and the local data, and determining the input dimension of the global model;

[0084] obtaining the feature expansion module related to the global model, performing dimensionality increasing processing on the local data according to the feature expansion module to obtain the feature expansion data with the same input dimension as the global model;

[0085] obtaining the initial local model, and performing knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain the target local model.

[0086] According to the above scheme, the global model and the local data are obtained, the input dimension of the global model is determined, the feature expansion module related to the global model is obtained, dimensionality increasing processing is performed on the local data according to the feature expansion module to obtain the feature expansion data with the same input dimension as the global model, the initial local model is determined, and knowledge distillation is performed on the initial local model according to the local data, the feature expansion data and the global model to obtain the target local model. The model compression method proposed in the present application can match the input dimension without losing information when compressing the model, thereby improving the generalization ability of the compressed model.

[0087] Based on the terminal device architecture described above but not limited to the above architecture, the method embodiment of the present application is proposed.

[0088] Referring to Figure 3 , Figure 3 The flowchart of the first embodiment of the model compression method of the present application is shown.

[0089] The embodiments of the model compression method are provided in the present application, and it should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be performed in an order different from that shown herein.

[0090] As Figure 3 shown, the model compression method of the present application embodiment includes the following steps:

[0091] Step S101, obtaining a global model and local data, and determining the input dimension of the global model;

[0092] Traditional model compression techniques include model pruning and knowledge distillation, but model pruning generally only focuses on the model structure itself, and knowledge distillation relies on the knowledge transfer of the teacher model. These model compression techniques are not sensitive to changes in input dimensions. When the input dimension of the compressed model needed is different from that of the global model, model compression often loses important information, and the generalization ability of the compressed model is weak.

[0093] To solve the technical problem of how to match the input dimension in the process of model compression without losing information, thereby improving the generalization ability of the compressed model. The model compression method of the present application embodiment aims to obtain a global model and local data, determine the input dimension of the global model, obtain a feature expansion module related to the global model, perform dimensionality processing on the local data according to the feature expansion module, obtain feature expansion data with the same input dimension as the global model, and then determine an initial local model. According to the local data, the feature expansion data and the global model, the knowledge of the initial local model is distilled to obtain a target local model. The model compression method proposed in the present application can match the input dimension without losing information when the model is compressed, thereby improving the generalization ability of the compressed model.

[0094] In the present embodiment, the global model is obtained. The global model usually has a complex structure but superior performance. The method can be, through a federated learning mechanism, obtaining high-dimensional global data, and simultaneously determining a deep neural network model, training the deep neural network model using a federated training framework to obtain the global model. The training of the global model adopts a standard supervised learning manner, that is, the data used for training contains real class labels; the global model can be obtained by directly obtaining a global model that has been trained by high-dimensional data; or the global model can be obtained by receiving a user input. Among them, a deep neural network model with a large number of parameters is generally determined. Meanwhile, local data is obtained, and the local data refers to a private data set Dp of the current node:

[0095] Dp={Dp i |1<i<N},Dp i ={x i ,y i}

[0096] Among them, Dp i represents the i-th data tuple in the data set, x i represents the i-th data, y i represents the corresponding class label, the local data is used to train the target local model, and the dimension of the local data is lower than that of the data used to train the global model, and is generally applied to small local devices such as portable devices. The input dimension of the global model is determined, that is, the dimension of the data used to train the global model.

[0097] Among them, the global teacher model of the electrocardiogram symptom classification can be constructed based on the electrocardiogram data of multiple first-class hospitals by using a federated learning mechanism, the electrocardiogram data of multiple first-class hospitals is high-dimensional global data, and is generally 12-lead electrocardiogram data with a data dimension of 12; the global teacher model is a global model. The local data can be a private data set of a hospital node, and is generally single-lead electrocardiogram data with a data dimension of 1.

[0098] In step S102, a feature expansion module related to the global model is obtained, the local data is processed by dimension increasing according to the feature expansion module, and feature expansion data with the same input dimension as the global model is obtained;

[0099] In this embodiment, since the dimension of the global data is generally much higher than that of the local data, a separate feature expansion module is designed for the global model, the local data is processed by dimension increasing by using the feature expansion module, and the feature expansion data is obtained, so that it can be used as the input of the feature expansion module.

[0100] The feature expansion module related to the global model can be obtained in the following manner: determining a convolutional neural network model, then determining a local teacher model trained by the local data, and continuously training the convolutional neural network model by the local data, the local teacher model and the global model, changing the parameter weights therein, so as to minimize the difference between the local teacher model and the global model, and obtaining the trained feature expansion module; or the feature expansion module related to the global model can be directly obtained.

[0101] The local data Dp is dimensionally upgraded by the feature expansion module, that is, the channel of the local data picture is increased, so as to obtain the feature expansion data Dp' and make the dimension of the feature expansion data Dp' conform to the input dimension of the global model.

[0102] The manner of dimensionally upgrading the local data by the feature expansion module can be that the feature expansion module comprises a feature dimension upgrading module. The feature dimension upgrading module comprises a 1*1 convolutional module, the input dimension of the global model, that is, the dimension of the data for training the global model is determined, the local data is input into the feature dimension upgrading module, the channel of the local data is dimensionally upgraded by the 1*1 convolutional module in the feature dimension upgrading module, and the feature expansion data with the same dimension as the input dimension of the global model is obtained.

[0103] Further, the feature dimension upgrading module can further comprise a multi-head attention mechanism module, the data obtained by dimensionally upgrading the local data by the 1*1 convolutional module is input into the multi-head attention mechanism module, and the feature expansion data is obtained again, and the multi-head attention mechanism can improve the effect of data dimension upgrading.

[0104] Further, the manner of dimensionally upgrading the local data by the feature expansion module can also be that the feature expansion module can simultaneously comprise a feature dimension upgrading module and a feature expansion module, the local data is input into the feature dimension upgrading module and the feature expansion module respectively, and the output of the feature dimension upgrading module and the feature expansion module is fused to obtain the feature expansion data.

[0105] In step S103, an initial local model is obtained, and the initial local model is knowledge distilled according to the local data, the feature expansion data and the global model to obtain a target local model.

[0106] In the embodiment, the initial local model is obtained in the following manner: The initial local model is generally a small model, that is, a neural network model with a relatively small depth. The local data is input into the initial local model, the feature expansion data is input into the global model, a joint loss function is calculated according to the output of the initial local model and the global model, the initial local model is knowledge distilled according to the joint loss function, and the initial local model is continuously updated to obtain the target local model.

[0107] The initial local model The process of knowledge distillation is to input the feature expansion data Dp' as the input of the global model , input the local data Dp as the input of the initial local model , and output two sets of label results l1 and l2 respectively.

[0108]

[0109] Wherein, and respectively represent the soft label output and the hard label output of the model, wherein the soft label is the probability value output by the model, and the hard label is the real class label of the input data, that is, y i in Dp i ={x i ,y i}.

[0110] The loss function Loss1 of the distillation process of the global model is set as:

[0111]

[0112] Wherein, CE represents the cross-entropy loss function, represents the soft label output after the i-th data in the feature expansion data Dp' is input into the global model , represents the class label of the i-th data in Dp', that is, y i .

[0113] represents the soft label output after the i-th data in the local data Dp is input into the initial local model, represents the class label of the i-th data in Dp, that is, y i . The i-th data in Dp' is obtained by dimensionally upgrading the i-th data in Dp.

[0114] Then, the loss function Loss2 representing the difference between the output of the initial local model and the real label is determined:

[0115]

[0116] Finally, the two sets of loss functions Loss1 and Loss2 are integrated to obtain the final target function Loss E , which is used as the total loss function for training the initial local model:

[0117] Loss E =w1Loss1+w2Loss2

[0118] Loss E The parameter weights of the initial local model are constantly updated as the total loss function of the initial local model until the model converges, the knowledge distillation is completed, and the final target local model is obtained, which is obtained by compressing the global model. Similarly, a certain number of times can also be set in advance, and when the number of iterations of the initial local model reaches the preset number of times, the training is completed, and the target local model is obtained. Wherein, the sum of w1 and w2 is 1, and the value of w1 and w2 can be changed constantly to make the initial local model obtain the best training effect.

[0119] After obtaining the target local model, data classification can be performed through the target local model to obtain the classification data, which is generally a local low-dimensional image. The feature expansion data is obtained by dimensionally upgrading the local data in the process of compressing the global model, and the feature expansion data is used as the input of the global model, thereby avoiding the loss of information and enabling the target local model obtained by compressing the global model to have good classification effect on the local low-dimensional data. The classification result can be obtained by inputting the classification data into the target local model, and the classification result can be output.

[0120] The present application obtains the global model and the local data, determines the input dimension of the global model, obtains the feature expansion module related to the global model, dimensionally upgrades the local data according to the feature expansion module, obtains the feature expansion data with the same input dimension as the global model, and then determines the initial local model. The initial local model is distilled according to the local data, the feature expansion data and the global model to obtain the target local model. The model compression method proposed in the present application can match the input dimension without losing information when the model is compressed, thereby improving the generalization ability of the compressed model.

[0121] Reference Figure 4 , Figure 4 The flowchart of the second embodiment of the model compression method of the present application is shown.

[0122] Further, as shown in Figure 4 , the second embodiment of the model compression method of the present application is proposed, and in the step S102, the global model is a global teacher model, and the step of obtaining the feature expansion module related to the global model comprises:

[0123] Step S10211, obtaining a convolutional neural network model and a local teacher model trained by the local data;

[0124] Step S10212, inputting the local data into the local teacher model, and then inputting the output of the local teacher model into the convolutional neural network model and the global teacher model respectively;

[0125] Step S10213, a first joint loss function is calculated according to the output of the convolutional neural network model and the global teacher model, the convolutional neural network model is trained according to the first joint loss function, and the feature expansion module is obtained.

[0126] In the embodiment, the global model is a global teacher model, and a local teacher model trained by local data is obtained. Specifically, a deep neural network model is obtained, the deep neural network model is trained by standard supervised learning using local data Dp, and the local teacher model is obtained. Generally, a deep neural network model with a large number of parameters is selected. The structure of the local teacher model and the global teacher model can be the same or different.

[0127] The feature expansion module can be obtained in the following manner. A convolutional neural network model is obtained, the local data is input into the local teacher model, the soft label output of the local teacher model is taken as the input of the convolutional neural network model and the global teacher model, a first joint loss function is calculated according to the output of the convolutional neural network model and the global teacher model, the convolutional neural network model is updated according to the first joint loss function, the parameter weight of the convolutional neural network model is adjusted, and the difference Loss between the local teacher model and the global teacher model is minimized. and the global teacher model between the local teacher model and the global teacher model is minimized, and the accuracy of dimensionality expansion of the local data is improved. E

[0128] The embodiment trains the convolutional neural network model according to the local teacher model and the global teacher model to obtain the feature expansion module, minimizes the difference between the local teacher model and the global teacher model, and improves the accuracy of dimensionality expansion of the local data.

[0129] Referring to Figure 5 , Figure 5 is a flowchart of the third embodiment of the model compression method of the present application.

[0130] Further, as Figure 5 shown, the third embodiment of the model compression method of the present application is proposed. In step S103, the initial local model is an initial local student model, and the step of knowledge distillation of the initial local model according to the local data, the feature expansion data and the global model to obtain a target local model includes:

[0131] Step S1031, the local data is input into the local teacher model and the initial local student model, respectively, and the feature expansion data is input into the global teacher model.

[0132] ​Step S1032, calculating a second joint loss function according to outputs of the local teacher model, the initial local student model and the global teacher model;

[0133] Step S1033, training the initial local student model according to the second joint loss function to obtain the target local model.

[0134] In the embodiment, the initial local model is the initial local student model. In order to improve the effect of model distillation and the recognition accuracy of the local model on the local data, the target local model can be constructed by distilling the multi-level teacher model, that is, by distilling the global teacher model and the local teacher model at the same time.

[0135] The local teacher model is obtained, specifically, a deep neural network model is obtained, the deep neural network model is trained by using the local data Dp through a standard supervised learning manner to obtain the local teacher model, wherein a deep neural network model with a large number of parameters is generally selected. The structure of the local teacher model can be the same as or different from that of the global teacher model.

[0136] The local data is input into the local teacher model and the initial local student model, the feature expansion data is input into the global teacher model, and then a second joint loss function is calculated according to outputs of the local teacher model, the initial local student model and the global teacher model. The initial local student model is trained according to the second joint loss function, the parameter weight of the initial local student model is continuously updated, and the target local model is obtained.

[0137] The global teacher model distillation is to distill the knowledge of the complex model input by the high-dimensional data, and the local teacher model distillation is to distill the knowledge of the complex model input by the low-dimensional data in the local area, to guide the construction of the simplified local student model. The ultimate goal is to construct a simple model that can be deployed in a portable device.

[0138] The initial local student model is distilled. The feature expansion data Dp' is input into the global teacher model , the local data Dp is input into the initial local student model and the local teacher model , and three sets of label results l1, l2 and l3 are output respectively.

[0139]

[0140] and respectively represent the soft label output and the hard label output of the model, wherein the soft label is the probability value output by the model, and the hard label is the real class label of the input data, that is, y in the data Dp.

[0141] Different loss functions Loss1 and Loss3 are set for the distillation process of the two teacher models.

[0142]

[0143]

[0144] wherein CE represents a cross-entropy loss function, represents the soft label output by the global teacher model after inputting the i-th data in the feature expansion data Dp' into the global teacher model . represents the class label y of the i-th data in Dp' i .

[0145] and represents the soft label output by the initial local student model and the local teacher model after inputting the i-th data in the local data Dp into the initial local student model and represents the class label y of the i-th data in Dp i . The i-th data in Dp' is obtained by dimensionally upgrading the i-th data in Dp.

[0146] The loss function Loss2 representing the difference between the output of the initial local student model and the real label is determined again:

[0147]

[0148] Finally, a plurality of sets of loss functions are integrated to obtain the final target function Loss e as the total loss function for training the initial local student model, and the total loss function Loss e is the second joint loss function:

[0149] Loss e = w1Loss1 + w2Loss2 + w3Loss3

[0150] Loss eThe total loss function of the initial local student model continuously updates the parameter weight of the initial local student model until the model converges, the knowledge distillation is completed, and the final target local model is obtained, and the target local model is obtained by simultaneously compressing the global teacher model and the local teacher model. Similarly, a certain number of times can be set in advance, and when the number of iterations of the initial local student model reaches the preset number of times, the training is completed, and the target local model is obtained. Wherein, the sum of w1, w2 and w3 is 1, and the value of w1, w2 and w3 can be changed to obtain the best training effect of the initial local student model.

[0151] The embodiment improves the accuracy of the target local model in identifying local data by distilling the multi-level teacher model, that is, distilling the initial local student model by the global teacher model and the local teacher model to obtain the target local model.

[0152] Further, the fourth embodiment of the model compression method of the present application is provided, the feature expansion module includes a feature dimension increasing module, and the step S102 includes the following steps:

[0153] In step S1022, the local data is input into the feature dimension increasing module for dimension increasing processing to obtain the feature expansion data with the same dimension as the input dimension of the global model.

[0154] In the embodiment, the feature expansion module includes a feature dimension increasing module, and the feature dimension increasing module includes a 1*1 convolution module, which can increase or decrease the dimension of the data through the 1*1 convolution module.

[0155] The input dimension of the global model is determined, the local data is input into the 1*1 convolution module, the dimension of the local data can be increased to the input dimension of the global model, and the feature expansion data with the same output dimension as the input dimension of the global model is output.

[0156] Further, the feature dimension increasing module can also include a multi-head attention mechanism module, and the output data obtained after the local data is input into the 1*1 convolution module is input into the multi-head attention mechanism module, and the effect of increasing the dimension of the data can be improved through the multi-head attention mechanism.

[0157] Alternatively, the feature expansion module includes the feature dimension increasing module and a feature expansion module, and the step S102 includes the following steps:

[0158] Step S1023, input the local data into the feature dimension increasing module and the feature expansion module respectively for dimension increasing processing, fuse the output data of the feature dimension increasing module and the feature expansion module to obtain the feature expansion data with the same dimension as the input dimension of the global model.

[0159] The feature expansion module includes the feature dimension increasing module and the feature expansion module. The manner of increasing the dimension of the local data to the feature expansion data can be that the local data is input into the feature dimension increasing module and the feature expansion module respectively for dimension increasing processing, and then the obtained output data is fused to obtain the feature expansion data.

[0160] The main function of the feature expansion module is to expand the dimension size of the input local data to the input dimension size of the global model through the full connection layer, and then perform Gaussian noise processing on the expanded data to obtain the output data of the feature expansion module. Finally, the output data obtained by inputting the local data into the feature dimension increasing module is fused with the output data obtained by inputting the local data into the feature expansion module, that is, the output data is directly added to obtain the feature expansion data.

[0161] For example, if the local data is a single lead electrocardiogram data, the data dimension is (5000, 1), indicating that there are 5000 data, and the channel of the data is 1. After the data enters the feature expansion module, it enters the feature dimension increasing module and the feature expansion module respectively. In the feature dimension increasing module, the electrocardiogram data is increased to 12 channels through a 1*1 convolution module, and further processed by using a multi-head attention mechanism. In the feature expansion module, the channel of the single lead data is expanded to 12 through a full connection layer, and then Gaussian noise processing is performed. Finally, the features output by the two modules are fused, and the dimension of the output data of the feature expansion module is increased to (5000, 12) to simulate the input of 12-lead data.

[0162] The embodiment increases the dimension of the local data through the feature module to obtain feature expansion data that meets the input dimension of the global model, and can realize input dimension matching without losing information.

[0163] Referring to Figure 6 , Figure 6 FIG. 5 is a flowchart of a fifth embodiment of the model compression method.

[0164] Further, as Figure 5 shown, the fifth embodiment of the model compression method is proposed. In step S101, the step of obtaining the global model includes:

[0165] Step S1001, obtaining a deep neural network model, and obtaining global data based on a federated learning mechanism.

[0166] Step S1002: Using a federated training framework based on standard supervised learning, train the deep neural network model according to the global data to obtain the global model.

[0167] In this embodiment, before obtaining the global model, a global model can be constructed through federated learning. Specifically, a deep neural network model is obtained, typically one with a large number of parameters. Then, a large amount of global data is acquired based on the federated learning mechanism. This global data is generally high-dimensional. The federated training framework is used to train the deep neural network model using this global data. The training process employs standard supervised learning, continuously updating the deep neural network model until it converges, resulting in the global model. This global model is a complex but high-performance classification model.

[0168] This embodiment constructs a global model through federated learning, which can obtain a global model with complex structure but superior performance, resulting in a compressed model with better classification performance after the global model is compressed.

[0169] In the field of electrocardiogram (ECG) diagnosis, the above model compression method can be implemented in the following ways:

[0170] First, based on the federated learning mechanism, data from multiple top-tier hospitals was acquired, and a global teacher model for ECG symptom classification was trained using a federated training framework. The global teacher model, also known as the global model, refers to high-dimensional global data, typically 12-lead ECG data, generated from multiple top-tier hospitals. This allows for the determination of the global teacher model. The input dimension is 12.

[0171] Next, obtain the local private data Dp. Dp represents a set of private data for a hospital node, which is usually single-lead ECG data.

[0172] Dp={Dp i |1 <i<N},Dp i ={x i ,y i}

[0173] Among them, Dp i Let x represent the i-th data tuple in the ECG dataset. i Let y represent the i-th electrocardiogram (ECG) data. i This represents its corresponding category label. A private teacher model is trained using local private data. Both the global teacher model and the local teacher model are trained using standard supervised learning methods. and The model structure can be consistent or inconsistent, but generally a deep neural network model with a large number of parameters is adopted.

[0174] An independent feature expansion module M is designed for the global teacher model E Since the dimension of the global data is much higher than the dimension of the local data deployed on the mobile end by the local node, the M E is used to expand the local data, so that it can be used as the input of the global teacher model.

[0175] A convolutional neural network model is obtained, the local data is input into the local teacher model, the soft label output of the local teacher model is used as the input of the convolutional neural network model and the global teacher model, and then the joint loss function is calculated according to the output of the convolutional neural network model and the global teacher model. The joint loss function is used to update the convolutional neural network model, adjust the parameter weight of the convolutional neural network model, minimize the difference Loss between the local teacher model and the global teacher model E , and finally obtain the trained feature expansion module.

[0176] The step of inputting the local data into the feature expansion module for dimension expansion processing is that the local data is single-lead data, and after the data enters the feature expansion module, it enters the feature dimension expansion module and the feature expansion module. In the feature dimension expansion module, the channel of the electrocardiogram data is expanded to 12 by a 1*1 convolution module, and further processed by a multi-head attention mechanism; in the feature expansion module, the main function of the feature expansion module is to expand the dimension of the single-lead data to 12 through a full connection layer, and then perform Gaussian noise processing. Finally, the features output by the two modules are fused to obtain feature expansion data, and the dimension of the output data is expanded to 12, which is used to simulate the input of 12-lead data.

[0177] Finally, the feature expansion data is used as the input of the global teacher model , the local data Dp is used as the input of the initial local student model and the local teacher model , and three sets of label results l1, l2 and l3 are output respectively.

[0178]

[0179] Among them, and represent the soft label output and hard label output of the model respectively, wherein the soft label is the probability value output by the model, and the hard label is the true class label of the input data, that is, y in the data Dp.

[0180] Different loss functions Loss1 and Loss3 are set for the distillation process of the two teacher models:

[0181]

[0182]

[0183]

[0184] The above formula represents the basic constraint specification of distilling the knowledge of the teacher model to the student model. The loss function Loss2 represents the difference between the output of the initial local student model and the real label.

[0185] Wherein, CE represents the cross-entropy loss function, represents the soft label output after the i-th data in the feature expansion data inputting the global teacher model , represents the class label of the i-th data in the feature expansion data, that is, y i .

[0186] and represents the soft label output after the i-th data in the local data Dp inputting the initial local student model and the local teacher model, and represents the class label of the i-th data in Dp, that is, y i . The i-th data in the feature expansion data is obtained by dimensionally upgrading the i-th data in Dp.

[0187] Finally, the final target function Loss e is obtained by integrating multiple sets of loss functions as the total loss function for training the initial local student model:

[0188] Loss e = w1Loss1 + w2Loss2 + w3Loss3

[0189] According to the total loss function Loss e , the initial local student model is trained to obtain the target local student model, that is, the target local model. Through the above method, the knowledge of the global teacher model, that is, the 12-lead complex model, can be distilled. The purpose of the local teacher model is to distill the knowledge of the local single-lead complex model to guide the construction of the local student simplified model, and the ultimate purpose is to construct a simple model that can be deployed in a portable device.

[0190] Referring to Figure 7 , Figure 7 is a flowchart of the first embodiment of the data classification method of the present application.

[0191] Embodiments of the data classification method are provided. It should be noted that although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that shown.

[0192] The data classification method comprises:

[0193] In step S201, the data to be classified is obtained.

[0194] In this embodiment, the data to be classified is obtained, which refers to local image data to be classified, generally low-dimensional data, and the channel of the image, i.e., the number of image channels, is low, for example, the number of channels of a grayscale image is 1, which is low-dimensional data.

[0195] In step S202, the data to be classified is input into a target local model created in advance to obtain a classification result, wherein the target local model is obtained by using a feature expansion module to perform dimensionality increasing processing on local data to obtain feature expansion data with the same input dimension as a global model, and then performing knowledge distillation on an initial local model according to the local data, the feature expansion data, and the global model.

[0196] In this embodiment, the data to be classified is input into a target local model created in advance to obtain a classification result.

[0197] The target local model is obtained by compressing the global model through knowledge distillation. The global model is generally trained by high-dimensional data and has a complex structure and superior performance. The training data of the target local model is generally low-dimensional data. Therefore, in order to match the input dimension in the model compression process, the training data of the target local model is dimensionally increased to obtain high-dimensional data, i.e., the local data is dimensionally increased to feature expansion data with the same input dimension as the global model. Then, the high-dimensional feature expansion data is input into the global model, and the low-dimensional local data is input into the initial local model. The joint loss function is calculated through the outputs of the global model and the initial local model. The initial local model is trained through knowledge distillation according to the joint loss function to obtain the target local model.

[0198] In step S203, the classification result is output.

[0199] In this embodiment, after obtaining the classification result of the data, the classification result of the data is output.

[0200] According to the present application, the data to be classified is input into the target local model obtained by compressing the global model, and the classification result of the data can be accurately output.

[0201] Reference Figure 8 , Figure 8 FIG. 2 is a flowchart of a second embodiment of the data classification method of the present application.

[0202] Further, as shown in Figure 7 The second embodiment of the data classification method of the present application is proposed, and before the step of inputting the data to be classified into the target local model in step S202, the step further comprises:

[0203] In step S2011, the global model and the local data are obtained, and the input dimension of the global model is determined.

[0204] In step S2012, the feature expansion module related to the global model is obtained, and the local data is processed by dimensionality expansion according to the feature expansion module, so as to obtain the feature expansion data with the same input dimension as the global model.

[0205] In step S2013, the initial local model is obtained, and the initial local model is distilled according to the local data, the feature expansion data and the global model, so as to obtain the target local model.

[0206] In this embodiment, the global model is obtained. The global model usually has a complex structure but superior performance. The method can be that the high-dimensional global data is obtained through the federated learning mechanism, and the deep neural network model is determined. The deep neural network model is trained by using the federated training framework, so as to obtain the global model. The global model is trained by using the standard supervised learning method, that is, the data for training contains real class labels. The global model can also be obtained directly by training the high-dimensional data. The global model can also be obtained by receiving the input of the user. Generally, the deep neural network model with a large number of parameters is determined.

[0207] The local data is obtained at the same time. The local data refers to the private data set Dp of the current node:

[0208] Dp = {Dp i |1 < i < N}, Dp i = {x i ,y i}

[0209] Wherein, Dp i represents the i-th data tuple in the data set, x i represents the i-th data, y i represents the corresponding class label. The local data is used to train the target local model. The dimension of the local data is lower than that of the data for training the global model, and is generally applied to small local devices such as portable devices. The input dimension of the global model, that is, the dimension of the data for training the global model, is determined.

[0210] The global teacher model of the electrocardiogram symptom classification can be constructed based on a federated learning mechanism using electrocardiogram data of multiple first-class hospitals, and the electrocardiogram data of the multiple first-class hospitals is high-dimensional global data, generally 12-lead electrocardiogram data with a data dimension of 12; the global teacher model is a global model. The local data can be a private data set of a hospital node, generally single-lead electrocardiogram data with a data dimension of 1.

[0211] Since the dimension of the global data is generally much higher than that of the local data, a separate feature expansion module is designed for the global model, and the feature expansion module is used to upgrade the local data to obtain feature expansion data, so that the feature expansion data can be used as the input of the feature expansion module.

[0212] The feature expansion module related to the global model is obtained. The way of obtaining can be to determine a convolutional neural network model, and then determine a local teacher model trained by the local data. The convolutional neural network model is continuously trained by the local data, the local teacher model and the global model, the parameter weights are changed, the difference between the local teacher model and the global model is minimized, and the trained feature expansion module is obtained. The way of obtaining can also be to directly obtain the feature expansion module related to the global model which has been trained.

[0213] The local data Dp is upgraded by the feature expansion module, that is, the channel of the local data picture is increased, to obtain the feature expansion data Dp', so that the dimension of the feature expansion data Dp' can meet the input dimension of the global model.

[0214] The way of upgrading the local data by the feature expansion module can be that the feature expansion module includes a feature upgrading module. The feature upgrading module includes a 1*1 convolution module. The input dimension of the global model is determined, that is, the dimension of the data used to train the global model. The local data is input into the feature upgrading module, and the channel of the local data is upgraded by the 1*1 convolution module in the feature upgrading module to obtain feature expansion data with the same dimension as the input dimension of the global model.

[0215] Further, the feature upgrading module can further include a multi-head attention mechanism module. The data obtained by inputting the local data into the 1*1 convolution module is input into the multi-head attention mechanism module, and then the feature expansion data is obtained. The multi-head attention mechanism can improve the effect of data upgrading.

[0216] Further, the way of upgrading the local data by the feature expansion module can also be that the feature expansion module can include a feature upgrading module and a feature expansion module at the same time. The local data is input into the feature upgrading module and the feature expansion module respectively, and the outputs of the feature upgrading module and the feature expansion module are fused to obtain the feature expansion data.

[0217] Get the initial local model The initial local model is generally a small model, i.e. a shallow neural network model. Local data is input into the initial local model, and the extended feature data is input into the global model. Then, the joint loss function is calculated based on the outputs of the initial local model and the global model. Knowledge distillation is then performed on the initial local model based on the joint loss function. The initial local model is continuously updated to obtain the target local model.

[0218] For the initial local model The process of knowledge distillation involves using the extended feature data Dp′ as a global model. The input is the local data Dp, which is used as the initial local model. Given the input, output two sets of label results, l1 and l2, respectively.

[0219]

[0220] in, and These represent the soft-label output and hard-label output of the model, respectively. The soft label is the probability value output by the model, and the hard label is the true class label of the input data, i.e., Dp. i ={x i ,y i y in} i .

[0221] The loss function Loss1 is set for the distillation process of the global model:

[0222]

[0223] Where CE represents the cross-entropy loss function, This represents the i-th data point in the extended feature data Dp′ input to the global model. The soft tag output later, This represents the category label of the i-th data in Dp′, i.e., y i .

[0224] This represents the soft label output after the i-th data in the local data Dp is input into the initial local model. This represents the category label of the i-th data in Dp, i.e., y i The i-th data in Dp′ is obtained by increasing the dimensionality of the i-th data in Dp.

[0225] Further determine the initial local model Loss2 is the loss function that determines the difference between the output and the true label.

[0226]

[0227] Finally, the two sets of loss functions Loss1 and Loss2 are integrated to obtain the final target function Loss E , as the total loss function of the initial local model for training:

[0228] LosS E =w1Loss1+w2Loss2

[0229] The Loss E is continuously updated as the total loss function of the initial local model to update the parameter weight of the initial local model until the model converges, the knowledge distillation is completed, and the final target local model is obtained. The target local model is obtained by compressing the global model. Similarly, a certain number of times can also be set in advance, and when the number of iterations of the initial local model reaches the preset number of times, the training is completed, and the target local model is obtained. Wherein, the sum of w1 and w2 is 1, and the value of w1 and w2 can be changed to obtain the best training effect of the initial local model.

[0230] In this embodiment, the global model and the local data are obtained, the input dimension of the global model is determined, the feature expansion module related to the global model is obtained, the local data is processed by dimensionality expansion according to the feature expansion module, the feature expansion data with the same input dimension as the global model is obtained, the initial local model is determined, and the initial local model is distilled according to the local data, the feature expansion data and the global model to obtain the target local model. The model compression method provided by the application can match the input dimension under the premise of not losing information when the model is compressed, and the generalization ability of the compressed model is improved.

[0231] The application also provides a model compression device.

[0232] Referring to Figure 9 , Figure 9 is a schematic diagram of the functional modules of the first embodiment of the model compression device of the application. The model compression device comprises:

[0233] The acquisition module 10 is configured to acquire a global model and local data, and determine an input dimension of the global model.

[0234] The data dimensionality expansion module 20 is configured to obtain a feature expansion module related to the global model, and perform dimensionality expansion on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model.

[0235] The knowledge distillation module 30 is configured to obtain an initial local model, and perform knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain a target local model.

[0236] Furthermore, the present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the model compression method or the data classification method as described above.

[0237] The computer program product embodiment of the present application is basically the same as the above-mentioned model compression method or the above-mentioned data classification method, and will not be described here.

[0238] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of another identical element in the process, method, article or system including the element.

[0239] The above-mentioned embodiment number of the present application is only for description, not representing the pros and cons of the embodiment.

[0240] Through the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by means of software and necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application or the part which contributes to the prior art can be embodied in the form of software product, which is stored in the above-mentioned storage medium (such as ROM / RAM, magnetic disc, optical disc), including a plurality of instructions for making a terminal device (which can be a mobile phone, computer, server, controlled terminal or network device, etc.) execute the method of each embodiment of the present application.

[0241] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent flow transformation made by using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A model compression method, characterized in that, The model compression method comprises the following steps: obtaining a global model and local data, and determining an input dimension of the global model; obtaining a feature expansion module related to the global model, performing dimensionality increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model; obtaining an initial local model, and performing knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain a target local model; The global model is a global teacher model, and the initial local model is an initial local student model. The step of performing knowledge distillation on the initial local model according to the local data, the feature expansion data and the global model to obtain a target local model comprises: inputting the local data into a local teacher model trained by the local data and the initial local student model respectively, and inputting the feature expansion data into the global teacher model; calculating a second joint loss function according to the outputs of the local teacher model, the initial local student model and the global teacher model; training the initial local student model according to the second joint loss function to obtain the target local model.

2. The model compression method of claim 1, wherein, The step of obtaining the feature expansion module related to the global model comprises: obtaining a convolutional neural network model and a local teacher model trained by the local data; inputting the local data into the local teacher model, and inputting the output of the local teacher model into the convolutional neural network model and the global teacher model respectively; calculating a first joint loss function according to the outputs of the convolutional neural network model and the global teacher model, and training the convolutional neural network model according to the first joint loss function to obtain the feature expansion module.

3. The model compression method of claim 1, wherein, The feature expansion module comprises a feature dimension increasing module. The step of performing dimensionality increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model comprises: inputting the local data into the feature dimension increasing module for dimensionality increasing processing to obtain the feature expansion data with the same input dimension as the global model; Alternatively, the feature expansion module comprises the feature dimension increasing module and a feature expansion module. The step of performing dimensionality increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model comprises: inputting the local data into the feature dimension increasing module and the feature expansion module respectively for dimensionality increasing processing, fusing the output data of the feature dimension increasing module and the feature expansion module to obtain the feature expansion data with the same input dimension as the global model.

4. The model compression method of claim 1, wherein, The step of obtaining the global model comprises: obtaining a deep neural network model, and obtaining global data based on a federated learning mechanism; training the deep neural network model based on standard supervised learning by using a federated training framework according to the global data to obtain the global model.

5. A data classification method characterized by, The data classification method comprises the following steps: obtaining to-be-classified data; inputting the to-be-classified data into a pre-created target local model to obtain a classification result, the target local model being obtained by using a feature expansion module to perform dimensionality increasing processing on local data to obtain feature expansion data with the same input dimension as a global teacher model, and then using an output of a local teacher model trained by the local data, an output of an initial local student model inputting the local data, and an output of the global teacher model inputting the feature expansion data to calculate a second joint loss function, and then using the second joint loss function to train the initial local student model to obtain the target local model; outputting the classification result.

6. A model compression apparatus, comprising: The model compression device comprises: an acquisition module configured to acquire a global model and local data, and determine an input dimension of the global model, the global model being a global teacher model; a data dimensionality increasing module configured to acquire a feature expansion module related to the global model, and perform dimensionality increasing processing on the local data according to the feature expansion module to obtain feature expansion data with the same input dimension as the global model; a knowledge distillation module configured to acquire an initial local model, and perform knowledge distillation on the initial local model according to the local data, the feature expansion data, and the global model to obtain a target local model, the initial local model being an initial local student model; the knowledge distillation module is further configured to input the local data into a local teacher model trained by the local data and the initial local student model respectively, and input the feature expansion data into the global teacher model, calculate a second joint loss function according to outputs of the local teacher model, the initial local student model, and the global teacher model, and train the initial local student model according to the second joint loss function to obtain the target local model.

7. A terminal device, characterized by comprising: The terminal device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is executed by the processor to implement the model compression method of any one of claims 1-4 or the data classification method of claim 5.

8. A computer program product, characterised in that, The computer program product comprises a computer program, and the computer program is executed by the processor to implement the model compression method of any one of claims 1-4 or the data classification method of claim 5.

Images