Neural network model compression method, device, storage medium and chip

Abstract

The present application provides a method for compressing a neural network model in the field of artificial intelligence, including: a server obtains a first neural network model uploaded by a user device and training data of the first neural network; a PU classifier is obtained according to the training data of the first neural network and unlabeled data stored by the server; the PU classifier is used to select extended data from the unlabeled data stored by the server, and the extended data has similar properties and distribution to the training data of the first neural network model; according to the extended data, a second neural network model is trained using a knowledge distillation KD method, the first neural network model is used as a teacher network model, and the second neural network model is used as a student network model. The PU classifier is used to select data with similar properties and distribution to the training data of the first neural network model from the unlabeled data, thereby improving the accuracy of the compression of the neural network model and avoiding the transmission of a large amount of positive sample data.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for compressing a neural network model. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] Computer vision is an integral part of various intelligent / autonomous systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is a science about how to use cameras / camcorders and computers to obtain the data and information of the objects we need. Figuratively speaking, it is to install eyes (cameras / camcorders) and brains (algorithms) on computers to replace human eyes to identify, track and measure targets, so that computers can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multidimensional data. In general, computer vision uses various imaging systems to replace visual organs to obtain input information, and then computers replace the brain to complete the processing and interpretation of these input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.

[0004] Convolutional neural network models usually have a large number of redundant parameters. In order to apply CNN networks to terminal devices with limited computing power such as smartphones, the existing convolutional neural network models can be compressed and accelerated. Neural network model compression technology requires a large amount of training data to enable the network to converge to a better result. However, it is very time-consuming for users to upload a large amount of data to the cloud, which will cause a decline in user experience. Some neural network model compression technologies only use a small amount of training data to compress the model, but the compressed neural network model is difficult to achieve satisfactory results. Summary of the invention

[0005] The present application provides a method, device, storage medium and chip for compressing a neural network model to reduce the amount of transmitted data and improve user experience.

[0006] In a first aspect, a method for compressing a neural network model is provided, comprising: a server obtains a first neural network model uploaded by a user device and training data of the first neural network; the server obtains a PU classifier using a positive class and an unlabeled PU learning algorithm based on the training data of the first neural network and unlabeled data stored by the server; the server uses the PU classifier to select extended data from the unlabeled data stored by the server, wherein the extended data is data having similar attributes and distribution to the training data of the first neural network model; the server trains a second neural network model using a knowledge distillation KD method based on the extended data, wherein the first neural network model serves as a teacher network model of the KD method, and the second neural network model serves as a student network model of the KD method.

[0007] The training data and unlabeled data of the first neural network are used to train a PU classifier, and the unlabeled data is classified to obtain data with similar properties and distribution to the training data of the first neural network model. Based on these data with similar properties and distribution to the training data of the first neural network model, the neural network model can be compressed, and the amount of transmitted data is reduced while ensuring the compression accuracy of the neural network model.

[0008] In combination with the first aspect, in some possible implementations, the server obtains a PU classifier using a positive class and unlabeled PU learning algorithm based on the training data of the first neural network, the unlabeled data stored by the server, and the proportion information. The loss function of the PU learning algorithm is the expectation of the training loss of the training data of the first neural network and the unlabeled data stored by the server. The proportion information is used to indicate the proportion of the extended data in the unlabeled data stored by the server, and the proportion information is used to calculate the expectation.

[0009] According to the proportion information of the positive sample data in the second data, the expectation of the training loss of the first data and the second data is calculated as the loss function of the PU learning algorithm, so as to train the PU classifier.

[0010] In combination with the first aspect, in some possible implementations, the training data of the first neural network model is part of the training data used to train the first neural network model.

[0011] The user device uploads part of the training data used to train the first neural network model, thereby reducing the amount of uploaded data and improving user experience.

[0012] In combination with the first aspect, in some possible implementations, the portion of training data includes data of each category in a plurality of categories output by the first neural network.

[0013] The training data of the first neural network model uploaded by the user includes data of each category of the multiple categories of data that the first neural network can process, so that the trained second neural network can process the multiple categories of data, thereby improving the accuracy of compression of the first neural network model.

[0014] In combination with the first aspect, in some possible implementations, the PU classifier is obtained based on the first feature and the proportion information, the first feature is obtained based on the fusion of multiple third features, the multiple third features are obtained by extracting features from the training data of the first neural network and the unlabeled data stored by the server using the first neural network model, the multiple third features correspond one-to-one to multiple layers of the first neural network, and the server uses the PU classifier to select extended data from the unlabeled data stored by the server, including: the server uses the first neural network model to extract features from the unlabeled data stored by the server to obtain a second feature; the server inputs the second feature into the PU classifier to determine the extended data. The feature extraction network for the data for training the PU classifier is performed through the first neural network model, which reduces the time for training the PU classifier and improves efficiency.

[0015] In combination with the first aspect, in some possible implementations, the first feature is obtained by fusing multiple third features after a first weight adjustment, the first weight adjustment is based on the proportion information, the second feature is obtained by fusing multiple fourth features using the first weight, and the multiple fourth features correspond one-to-one to multiple layers of the first neural network.

[0016] By adjusting the weights of features output by different layers of the first neural network model, the extracted features are fused according to the adjusted weights to improve the accuracy of the classification results of the PU classifier.

[0017] In combination with the first aspect, in some possible implementations, the server trains a second neural network model using a KD method based on the extended data, including: the server inputs the extended data into the first neural network model to classify the extended data to obtain multiple categories of extended data and a second weight for each category of the multiple categories of extended data; the server minimizes the loss function of the KD algorithm to obtain a trained second neural network model, wherein the loss function of the KD algorithm is the sum of the products of the training errors of each category of the multiple categories of extended data and the second weights of each category of the extended data.

[0018] According to the number of positive sample data in the unlabeled data in the categories that the first neural network model can handle, the weight corresponding to the data of each class in the loss function of the KD algorithm is adjusted. When the positive sample data is unevenly distributed in different categories, the trained neural network model can obtain better classification results for the data of each class.

[0019] In combination with the first aspect, in some possible implementations, the second weights of each type of extended data include multiple perturbed weights obtained by randomly perturbing the initial weights of each type of extended data, the loss function of the KD algorithm includes multiple loss functions corresponding to the multiple perturbed weights, wherein the initial weights of each type of extended data are negatively correlated with the number of each type of extended data, and the server minimizes the loss function of the KD algorithm to obtain a trained second neural network model, including: the server minimizes the maximum value of the multiple loss functions to obtain the trained second neural network model.

[0020] By randomly perturbing the weights in the loss function of the KD algorithm, the loss function of the KD algorithm of the neural network model under different perturbations is calculated, and the neural network model that minimizes the maximum value of the multiple loss functions is used as the trained neural network model. The influence of the classification error of the teacher network model on the accuracy of the trained neural network model is reduced, and the accuracy of the trained neural network model is improved.

[0021] In a second aspect, a neural network model compression device is provided, which includes various modules for executing the method in the first aspect.

[0022] In a third aspect, a computer device is provided, comprising a memory for storing programs; and a processor for executing the programs stored in the memory. When the programs stored in the memory are executed, the processor is used to execute the method in the first aspect.

[0023] It should be understood that the computer device may be a server, which may be deployed in the cloud. The computer device has a certain computing capability.

[0024] According to a fourth aspect, a computer storage medium is provided, wherein the computer readable storage medium stores program code, wherein the program code includes instructions for executing the steps in the method according to the first aspect.

[0025] According to a fifth aspect, a computer program product comprising instructions is provided, and when the computer program product is run on a computer, the computer is enabled to execute the method in the first aspect.

[0026] In a sixth aspect, a chip is provided, the chip comprising at least one processor, and when program instructions are executed in the at least one processor, the chip executes the method described in the first aspect.

[0027] Optionally, as an implementation method, the chip may further include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the method in the first aspect.

[0028] The above chip may specifically be a field programmable gate array FPGA or an application specific integrated circuit ASIC.

[0029] It should be understood that in the present application, the method of the first aspect may specifically refer to the method in the first aspect and any one of the various implementations of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the system architecture provided by the embodiment of the present application.

[0031] Figure 2 It is a schematic diagram of a convolutional neural network model.

[0032] Figure 3 A schematic diagram of a neural network model compression method.

[0033] Figure 4 It is a schematic flowchart of a method for compressing a neural network model provided by an embodiment of the present application.

[0034] Figure 5 It is a schematic flowchart of a method for compressing a neural network model provided by another embodiment of the present application.

[0035] Figure 6 It is a schematic flowchart of a method for expanding positive sample data provided by an embodiment of the present application.

[0036] Figure 7 This is a schematic diagram of a multi-feature fusion model with an attention mechanism provided by an embodiment of the present application.

[0037] Figure 8 It is a schematic diagram of a knowledge distillation method provided by an embodiment of the present application.

[0038] Fig. 9 This is a schematic flowchart of a method for compressing a neural network model provided by yet another embodiment of the present application.

[0039] Fig.10It is a schematic structural diagram of a device for compressing a neural network model provided by an embodiment of the present application.

[0040] Fig.11 This is a schematic diagram of the hardware structure of a device for compressing a neural network model provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0042] Since the embodiments of the present application involve the application of a large number of neural network models, in order to facilitate understanding, the relevant terms and related concepts such as neural network models involved in the embodiments of the present application are first introduced below.

[0043] (1) Neural network model

[0044] A neural network model can be composed of neural units, where a neural unit can be a s and intercept b as inputs, the output of the operation unit can be:

[0045]

[0046] Where, s = 1, 2, ... n, n is a natural number greater than 1, W s For x s The weight of the neural unit, b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network model to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer. The activation function can be a sigmoid function. The neural network model is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0047] (2) Deep Neural Network Model

[0048] A deep neural network model (DNN), also known as a multi-layer neural network model, can be understood as a neural network model with many hidden layers. There is no special metric for "many" here. Based on the position of different layers of DNN, the neural network model inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. For example, in a fully connected neural network model, the layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network model, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network model is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network model (a weight matrix formed by many layers of vectors W).

[0049] (3) Convolutional Neural Network Model

[0050] The convolutional neural network model (CNN) is a deep neural network model with a convolutional structure. The convolutional neural network model contains a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network model that performs convolution processing on the input signal. In the convolution layer of the convolutional neural network model, a neuron can only be connected to some neurons in the adjacent layer. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in a part can also be used in another part. Therefore, the same learned image information can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0051] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network model. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network model, while reducing the risk of overfitting.

[0052] (4) Loss Function

[0053] In the process of training a deep neural network model, because we hope that the output of the deep neural network model is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network model according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network model). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network model can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network model becomes a process of minimizing this loss as much as possible.

[0054] (5) Residual Network

[0055] When the depth of the neural network model is continuously increased, the problem of degradation will occur. That is, as the depth of the neural network model increases, the accuracy first increases, then reaches saturation, and continuing to increase the depth will cause the accuracy to decrease. The biggest difference between the ordinary direct-connected convolutional neural network model and the residual network (ResNet) is that ResNet has many bypass branches that connect the input directly to the subsequent layers. By directly bypassing the input information to the output, the integrity of the information is protected and the degradation problem is solved. The residual network includes convolutional layers and / or pooling layers.

[0056] The residual network can be: In addition to the layer-by-layer connection between multiple hidden layers in the deep neural network model, for example, the first hidden layer is connected to the second hidden layer, the second hidden layer is connected to the third hidden layer, and the third hidden layer is connected to the fourth hidden layer (this is a data operation path of the neural network model, which can also be figuratively called neural network model transmission), the residual network also has an additional direct branch, which is directly connected from the first hidden layer to the fourth hidden layer, that is, skipping the processing of the second and third hidden layers, and directly transmitting the data of the first hidden layer to the fourth hidden layer for operation. The highway network can be: In addition to the operation path and direct connection branch described above, the deep neural network model also includes a weight acquisition branch, which introduces a transmission gate (transform gate) to obtain the weight value, and outputs the weight value T for the subsequent operation of the above operation path and direct connection branch.

[0057] (6) Back propagation algorithm

[0058] The convolutional neural network model can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0059] (7) Attention Mechanism

[0060] The attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensations to increase the observation precision of some areas, and can use limited attention resources to quickly filter out high-value information from a large amount of information. The attention mechanism can quickly extract important features of sparse data, and is therefore widely used in natural language processing tasks, especially machine translation. The self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features. The essential idea of ​​the attention mechanism can be rewritten as the following formula:

[0061]

[0062] Among them, Lx=||Source|| represents the length of Source. The formula means that the elements in Source are imagined as a series of<Key,Value> Data pairs are composed. At this time, given an element Query in the target Target, the weight coefficient of the Value corresponding to each Key is obtained by calculating the similarity or correlation between the Query and each Key, and then the Value is weighted and summed to obtain the final Attention value. Therefore, the Attention mechanism is essentially a weighted sum of the Value values ​​of the elements in the Source, and the Query and Key are used to calculate the weight coefficient of the corresponding Value. Conceptually, Attention can be understood as selectively screening out a small amount of important information from a large amount of information and focusing on these important information, ignoring most of the unimportant information. The process of focusing is reflected in the calculation of the weight coefficient. The larger the weight, the more it focuses on its corresponding Value value, that is, the weight represents the importance of the information, and the Value is the corresponding information. The self-attention mechanism can be understood as internal Attention (intra attention). The Attention mechanism occurs between the Query element of the Target and all the elements in the Source. The self-attention mechanism refers to the Attention mechanism that occurs between the internal elements of the Source or the internal elements of the Target. It can also be understood as the attention calculation mechanism in the special case of Target = Source. The specific calculation process is the same, but the calculation object has changed.

[0063] (8) Pixel value

[0064] The pixel value of an image can be a red, green, and blue (RGB) color value, and the pixel value can be a long integer representing the color. For example, the pixel value is 256×Red+100×Green+76×Blue, where Blue represents the blue component, Green represents the green component, and Red represents the red component. In each color component, the smaller the value, the lower the brightness, and the larger the value, the higher the brightness. For a grayscale image, the pixel value can be a grayscale value.

[0065] (9) Knowledge Distillation

[0066] Although larger and more complex networks usually have good performance, they also have a lot of redundant information, so the amount of computation and resource consumption is very high. The knowledge distillation of a neural network model is to extract useful information from a complex network and migrate it to a smaller network, so that the learned small network can have performance effects close to that of a large complex network, and also greatly save computing resources. This complex network can be called a teacher network model, and the small network can be called a student network model.

[0067] (10) Positive Class and Unlabeled Learning

[0068] Positive-unlabeled learning (PU learning) is a semi-supervised machine learning method. In this method, only some positive sample data are labeled in the given data, and the rest of the positive sample data and all negative sample data are unlabeled. In this case, through learning, a two-class classifier is trained to classify the unlabeled data and determine the positive sample data and negative sample data.

[0069] The above is a brief introduction to some basic contents of neural networks. The following is an introduction to some specific neural networks that may be used in image data processing.

[0070] Combine the following Figure 1 The system architecture of the embodiment of the present application is introduced in detail.

[0071] Figure 1 Schematic diagram of the system architecture of the embodiment of the present application. Figure 1 As shown, the system architecture 100 includes an execution device 110 , a training device 120 , a database 130 , a client device 140 , a data storage system 150 , and a data collection system 160 .

[0072] In addition, the execution device 110 includes a calculation module 111, an I / O interface 112, a preprocessing module 113 and a preprocessing module 114. The calculation module 111 may include the target model / rule 101, and the preprocessing module 113 and the preprocessing module 114 are optional.

[0073] The data collection device 160 is used to collect training data. In the embodiment of the present application, the training data includes: training data of the first neural network, the first neural network model and unlabeled data stored in the server. The training data is stored in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130.

[0074] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The above-mentioned target model / rule 101 can be used to implement the method for compressing the neural network model in the embodiment of the present application, that is, the training data of the first neural network and the first neural network model are input into the target model / rule 101 to obtain the trained second neural network model. The target model / rule 101 in the embodiment of the present application can specifically be a neural network. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, but may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 based entirely on the training data maintained by the database 130, and may also obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application. The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as in Figure 3 The execution device 110 shown in the figure may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud. Figure 3 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data can include: training data of the first neural network and the first neural network model in the embodiment of the present application. The client device 140 here can specifically be a user device.

[0075] The preprocessing module 113 and the preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by the I / O interface 112. In the embodiment of the present application, there may be no preprocessing module 113 and the preprocessing module 114 or only one preprocessing module. When there is no preprocessing module 113 and the preprocessing module 114, the computing module 111 may be directly used to process the input data.

[0076] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0077] Finally, the I / O interface 112 presents the processing results, such as the trained second neural network model calculated by the target model / rule 101, to the client device 140, thereby providing it to the user.

[0078] Specifically, the trained second neural network model obtained by processing the target model / rule 101 in the computing module 111 can be processed by the preprocessing module 113 (and can also be processed by the preprocessing module 114) and the processing results can be sent to the I / O interface, and then the I / O interface can send the processing results to the client device 140.

[0079] It should be understood that when the preprocessing module 113 and the preprocessing module 114 do not exist in the above-mentioned system architecture 100, the computing module 111 can also transmit the processed trained second neural network model to the I / O interface, and then the I / O interface sends the processing result to the client device 140 for display.

[0080] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0081] exist Figure 3 In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0082] It is worth noting that Figure 1 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 1In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0083] like Figure 3 As shown, the target model / rule 101 obtained by training with the training device 120 may be a neural network in an embodiment of the present application. Specifically, the neural network provided in the embodiment of the present application may be a CNN and a deep convolutional neural network (DCNN), etc.

[0084] Since CNN is a very common neural network, Figure 2 The focus is on a detailed introduction to the structure of CNN. As mentioned in the previous basic concept introduction, the convolutional neural network model is a deep neural network model with a convolution structure, and is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network model, in which each neuron can respond to the image input into it.

[0085] like Figure 2 As shown, the convolutional neural network model (CNN) 200 may include an input layer 210, a convolution layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network model layer 230.

[0086] Convolutional layer / pooling layer 220:

[0087] Convolutional Layer:

[0088] like Figure 2 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0089] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0090] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0091] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network model 200 can make correct predictions.

[0092] When the convolutional neural network model 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network model 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0093] Pooling layer:

[0094] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. Figure 1Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the only purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0095] Neural network model layer 230:

[0096] After being processed by the convolution layer / pooling layer 220, the convolution neural network model 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network model 200 needs to use the neural network model layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network model layer 230 may include multiple hidden layers (such as Figure 1 231, 232 to 23n) and output layer 240 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to relevant training data of a specific task type, for example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc. ...

[0097] After the multiple hidden layers in the neural network model layer 230, that is, the last layer of the entire convolutional neural network model 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network model 200 (such as Figure 1 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 1 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network model 200 and the error between the result output by the convolutional neural network model 200 through the output layer and the ideal result.

[0098] It should be noted that if Figure 2 The convolutional neural network model 200 shown is only an example of a convolutional neural network model. In specific applications, the convolutional neural network model can also exist in the form of other network models. CNN networks are widely used in the field of computer vision and have been successful in many practical applications such as image classification, target detection, and semantic segmentation. In order to obtain higher classification accuracy, CNN networks usually have a large number of redundant parameters. Many studies have shown that these redundant parameters can be removed without affecting the classification performance of the network. At the same time, in order to apply CNN networks to terminal devices with limited computing power such as smartphones, we need to compress and accelerate the existing CNN networks. Considering that there are a large number of computing resources in the cloud, we can provide users with cloud model compression services.

[0099] Through CNN network compression and acceleration technology, for example, the neural network model can be compressed using methods such as knowledge distillation (KD), pruning similar neurons, weight discretization, hashing, etc., and model compression and acceleration services can be provided to users in the cloud.

[0100] Figure 3 A schematic diagram of a neural network model compression method.

[0101] The training data is input into the teacher network model and the student network model. According to the output of the teacher network model and the output of the student network model, the cross entropy loss (cross entropy loss), that is, the loss function L of the KD algorithm, is determined. KD The loss function L of the KD algorithm is KD It can be used to measure the similarity of the processing results of the teacher network model and the student network model for the training data. Adjust the parameters of the student network model so that the loss function L of the KD algorithm KD Minimum:

[0102]

[0103] Where n is the number of training data, and For a given input x i Output of the post-teacher network model and the student network model; is the training loss of the i-th data in the training data.

[0104] The KD algorithm can also be called a CNN model compression and acceleration algorithm. By adjusting the parameters of the student network model so that the loss function L KD Minimum, that is, the output of the student network model is as consistent as possible with the output of the teacher network model, so that the student network model learns the relevant characteristics of the teacher network model.

[0105] The teacher network model is obtained based on the original training data, and the output of the teacher network model for the original training data is relatively accurate. The training of the student network model can be completed based on the original training data.

[0106] When training the student network model, it is necessary to obtain all the training data required for training the teacher network model so that the compressed network can have good classification performance. However, compared with the teacher network model, the file size of the training data is very large. For example, the ResNet-50 network only takes up 95MB of space to store the model parameters, but the training data set (image net) required to train the ResNet-50 network has more than 1.2 million training images and requires more than 120GB of space to store. Therefore, in the case of limited transmission speed, users need to spend a long time to upload training data to the cloud. Providing a large amount of training data can make the network converge to a better result, but it is very time-consuming for users to upload a large amount of data to the cloud, which will bring a very bad user experience.

[0107] The teacher network model can obtain more accurate output results for images of the same class as the original data. By training the student network model with images of the same class as the original data, the teacher network model can be effectively utilized to obtain a more accurate student network model.

[0108] A way to obtain images with the same category distribution as the original training data can be found in "Data-Free Learning of Student Networks" (Chen H, Wang Y, Xu C, et al. 2019.), which generates images through generative adversarial networks.

[0109] Generative adversarial networks (GAN) are a type of deep learning model. The model includes at least two modules: one is the generative model, and the other is the discriminative model. These two modules learn from each other through competition, thereby producing better output. The basic principle of GAN is as follows: Taking the GAN that generates images as an example, assume that there are two networks, G (generator) and D (discriminator), where G is a network that generates images. It receives a random noise z and generates images through this noise, denoted as G(z); D is a discriminative network that is used to determine whether a picture is "real". Under ideal conditions, G can generate a picture G(z) that is "real" enough, while D has difficulty determining whether the picture generated by G is real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0110] The teacher network model is used as the discriminant model. The generative model generates a set of images based on random signals, and these images are input into the teacher network model to obtain the categories to which these images belong. For example, the output of the teacher network model is the probability that the image belongs to each category, and the sum of the probabilities that the image belongs to all categories is 1.

[0111] The teacher network model can only process images of specific categories. If an image that does not belong to the category that the teacher network model can process is input into the teacher network model, the output result of the teacher network model is still the probability that the image belongs to all the categories that the teacher network model can process.

[0112] If the probability that an image belongs to a certain category is greater than a preset value, the image is considered to belong to that category. If the preset value is high, the image is more likely to belong to that category, and the possibility of classifying an image that does not belong to that category as that category is low. However, many images that belong to that category are considered not to belong to that category because the teacher network model determines that the probability of belonging to that category is less than the preset value. In other words, if the preset value is high, the images obtained for training the student network model are only part of the images that the teacher network model can process. The student network model trained with these images cannot reflect the full performance of the teacher network model.

[0113] If the preset value is low, the possibility that the image belongs to the class is low, and the possibility that the image that does not belong to the class is classified as the class is high. Using images that the teacher network model cannot process for the student network model cannot accurately obtain a student network model that can process images of a specific class.

[0114] Therefore, judging whether the image belongs to the category that the teacher network model can process based on the processing result of the teacher network model is not accurate, which has an adverse effect on the training of the student network model, and the effect of the compressed neural network model is difficult to be satisfactory.

[0115] Figure 4 It is a schematic flow chart of a neural network model compression method provided in an embodiment of the present application.

[0116] In step S301, the server obtains a first neural network model and training data of the first neural network uploaded by a user device.

[0117] The training data of the first neural network may also be referred to as first data. The first data includes all or part of the training data of the first neural network model.

[0118] The user device uploads part of the training data used to train the first neural network model, thereby reducing the amount of uploaded data and improving user experience.

[0119] The portion of training data includes data of each category in a plurality of categories output by the first neural network.

[0120] The first neural network model is used to classify the input data into at least one of N categories. The first data may include data of each category in the N categories. That is, the first data may include data of each category in the categories that the first neural network model can process. The first neural network model is a neural network model that needs to be compressed. The first neural network model can be used for data classification. The first data may include data of each category in the categories to which all data used to train the first neural network model belong. In an embodiment of the present application, the positive sample data belongs to a category that the first neural network model can process.

[0121] The training data of the first neural network model uploaded by the user includes data of each category of the multiple categories of data that the first neural network can process, so that the trained second neural network can process the multiple categories of data, thereby improving the accuracy of compression of the first neural network model.

[0122] The first neural network model can realize the classification of input data. For example, the first neural network model can classify text, speech, images, features, etc. For example, it can classify the part of speech of each word in an input sentence (noun, verb...), or it can judge the emotion of a person when speaking based on an input speech, or it can determine the category of a person or object in an input picture, or classify the extracted features, etc.

[0123] In step S302, the server obtains a PU classifier using a positive-unlabeled (PU) learning algorithm according to the training data of the first neural network and the unlabeled data stored in the server.

[0124] Data having similar properties and distribution to the training data of the first neural network model may be referred to as positive class data or positive sample data. The first neural network model may be used to process positive sample data. The first data is positive sample data. Data having similar properties and distribution to the training data of the first neural network model may be referred to as positive sample data.

[0125] The unlabeled data stored by the server may also be referred to as second data. The second data may include positive sample data and data other than positive sample data, namely negative sample data. Unlabeled data means that it is not yet determined whether the data is positive sample data.

[0126] The server obtains the PU classifier. The server may receive the PU classifier sent by the user equipment or other servers, or obtain the PU classifier from the memory. Fig. 9 , the training of the PU classifier can be performed by other devices. The training of the PU classifier can also be performed by the server that compresses the first neural network model.

[0127] The server may train the PU classifier according to the first data, the second data, and the ratio information of positive sample data in the second data.

[0128] The server may train the PU classifier through a PU learning algorithm. The loss function of the PU learning algorithm may be an expectation of the training loss of the first data and the second data, and the ratio information is used to calculate the expectation.

[0129] The server may obtain information about the proportion of positive sample data in the second data. The proportion may also be referred to as prior probability information, which is used to indicate the proportion of positive sample data in the second data.

[0130] The PU learning algorithm provides a semi-supervised learning method. The server can classify the unlabeled data according to the labeled positive sample data and determine the positive sample data and negative sample data in the unlabeled data. Among them, the positive sample data can be data that can be processed by the first neural network model, and the negative sample data is data outside the data that can be processed by the first neural network model. The positive sample data belongs to the category that can be processed by the first neural network model, and the negative sample data does not belong to the category that can be processed by the first neural network model.

[0131] Using the PU learning algorithm, the server can input the first data and the second data into the PU classifier to be trained, and adjust the parameters of the PU classifier to be trained so that the classification result of the PU classifier on the second data satisfies the proportion information of the positive sample data in the second data, thereby obtaining the adjusted PU classifier. The adjusted PU classifier is the trained PU classifier. The principle of the PU learning algorithm can be found in Figure 6 Description.

[0132] In step S303, the server uses the PU classifier to select extended data from the unlabeled data stored in the server.

[0133] The extended data is data having similar attributes and distribution to the training data of the first neural network model, that is, the extended data is positive class data.

[0134] The unlabeled data used to obtain the extended data may be the same as the unlabeled data used to train the PU classifier, and the unlabeled data used to obtain the extended data may also include data different from the unlabeled data for training the PU classifier. The unlabeled data used to obtain the extended data may be referred to as third data. The third data may include all or part of the second data, or may be other data other than the second data. The amount of data stored in the server is limited. When training the PU classifier and using the PU classifier, the data input into the PU classifier is the same, which can increase the amount of data used to train the second neural network and improve the accuracy of training the second neural network.

[0135] The third data is input into the PU classifier, and the PU classifier classifies the third data to determine the positive sample data and the negative sample data therein. The extended data is the positive sample data in the third data.

[0136] The PU classifier can classify data or the features of data. The object of classification of the PU classifier is related to the training process of the PU classifier.

[0137] The first data and the second data may be input into a feature extraction network to be trained to obtain a first feature. The feature extraction network may be used to extract features from the input data to obtain a first feature. The first feature is input into a PU classifier to be trained. According to the proportion information of the positive sample data in the second data and the extracted first feature, the parameters of the feature extraction network to be trained and the PU classifier to be trained are adjusted to obtain a feature extraction network and a PU classifier.

[0138] The first features include features of each data in the first data extracted by the feature extraction network and features of each data in the second data.

[0139] According to the PU classifier, the positive sample data in the stored third data, i.e., the extended data, can be determined. The third data can be input into a feature extraction network to obtain a second feature. The second feature can be input into the PU classifier to determine the extended data.

[0140] The first feature may be obtained by fusing a plurality of third features output from a plurality of layers of the feature extraction network. The plurality of third features correspond one-to-one to the plurality of layers of the feature extraction network. The first weight adjustment is performed based on the proportion information.

[0141] The third feature includes the feature of each data in the first data extracted by the feature extraction network and the feature of each data in the second data.

[0142] The feature extraction network can be a CNN network.

[0143] There are many parameters in the feature extraction network, and adjusting the parameters of the feature extraction network requires more data and takes a long time.

[0144] The first neural network can be used as a feature extraction network. The first data and the second data can be input into the first neural network model to extract multiple third features output by multiple layers of the first neural network model, and the multiple third features are fused to obtain the first feature. The PU classifier is obtained based on the first feature and the proportion information of the positive sample data in the second data.

[0145] That is to say, the PU classifier is obtained based on the first feature and the proportion information of the positive sample data in the second data. The first feature is obtained based on the fusion of multiple third features. The multiple third features are obtained by extracting features from the first data and the second data using the first neural network model. The multiple third features correspond one-to-one to the multiple layers of the first neural network.

[0146] The first neural network model may be used to perform feature extraction on the third data to obtain a second feature. The second feature is obtained by fusing multiple fourth features using the first weight. The multiple fourth features correspond one-to-one to multiple layers of the first neural network. The second feature is input into the PU classifier to determine the extended data.

[0147] The first neural network model is a trained neural network model used to process positive sample data, so it can extract features that are effective for classifying positive sample data. The first neural network model can be used as a feature extraction network to reduce the number of parameters that need to be adjusted in the PU learning algorithm and improve the efficiency of obtaining positive sample data from unlabeled data stored in the cloud.

[0148] According to the features output by the first neural network model, a PU classifier can be trained using a positive class and an unlabeled PU learning algorithm.

[0149] To classify data such as text and images, the features of the data can be extracted through the feature extraction network. By using the first neural network model as the feature extraction network, there is no need to train the feature extraction network, which reduces the time for classifying unlabeled data and improves extraction efficiency.

[0150] Multiple features output by multiple layers of the feature extraction network. Multiple features output by multiple layers of the feature extraction network can be fused, and the fusion result can be input into the PU classifier. Features output by multiple layers of the feature extraction network can be input into the PU classifier with the same weight. The weights of features of multiple layers can also be adjusted through the multi-feature network of the attention mechanism.

[0151] The first feature is obtained by fusing multiple third features after first weight adjustment, wherein the first weight adjustment is performed based on the proportion information of positive sample data in the second data. The multiple third features correspond one-to-one to multiple layers of the feature extraction network.

[0152] That is, multiple third features can be fused, and the fused features can be input into the PU classifier to be trained. The weight of each third feature in the multiple third features is adjusted, and the parameters of the PU classifier to be trained are adjusted so that the classification result of the PU classifier satisfies the proportion information of the positive sample data in the second data. That is, according to the proportion information of the positive sample data in the second data, the first weight is determined to obtain the PU classifier.

[0153] The extended data is input into a feature extraction network, and multiple layers of the feature extraction network output multiple fourth features. The multiple fourth features output by the multiple layers of the feature extraction network are fused according to the first weight. The multiple fourth features correspond one to one with the multiple layers of the feature extraction network.

[0154] The correspondence between the third feature and the layer of the feature extraction network is the same as the correspondence between the fourth feature and the layer of the feature extraction network. In other words, the outputs of the multiple layers of the feature extraction network correspond one-to-one to the multiple weight values ​​in the first weight.

[0155] There are many ways to integrate.

[0156] Combination is a fusion method. When combining, the features to be combined can be added directly or weighted. Weighted addition means adding after multiplying by a certain coefficient, i.e., the weight value. In other words, combining can be used to linearly combine channel dimensions (channel wise).

[0157] The multiple features outputted by the multiple layers of the feature extraction network can be added together. For example, the multiple features outputted by the multiple layers of the feature extraction network can be directly added together, or the multiple features outputted by the multiple layers of the feature extraction network can be added together according to certain weights. T1 and T2 represent the features outputted by the two layers of the feature extraction network, respectively. The fused features can be represented by T3, T3 = a × T1 + b × T2, where a and b are the coefficients by which T1 and T2 are multiplied when calculating T3, i.e., weight values, a≠0, and b≠0.

[0158] Concatenation and channel fusion are another fusion method. Using the concatenate & channel fusion method, the dimensions of the features to be fused can be directly concatenated, or they can be concatenated after being multiplied by a certain coefficient, i.e., a weight value.

[0159] The feature extraction network is taken as the first neural network model as an example for explanation. A multi-feature network with an attention mechanism can be added to the first neural network model to obtain a third neural network model. The multi-feature network with an attention mechanism uses the attention mechanism to select multiple features output by different layers of the first neural network model. The labeled data and the unlabeled data are input into the third neural network model, and the positive sample data is obtained using the PU learning algorithm. Figure 7 A feature fusion method is shown.

[0160] Through a multi-feature network with an attention mechanism, the features extracted from different layers of the first neural network model are selected to improve the accuracy of classifying unclassified data.

[0161] In the training process of the PU classifier, the parameters of the multi-feature network and the PU classifier can be adjusted to obtain a more accurate PU classifier, thereby accurately selecting the extended data from the third data. In other words, the parameters of the multi-feature network and the PU classifier can be adjusted in the process of semi-supervised learning.

[0162] In step S304, the server trains a second neural network model using a KD method according to the extended data.

[0163] The first neural network model is used as the teacher network model of the KD method, and the second neural network model is used as the student network model of the KD method. The KD method can also be called the KD algorithm.

[0164] The server can also train the second neural network model using the knowledge distillation KD method based on the extended data and the training data of the first neural network uploaded by the user.

[0165] The training data of the first neural network is used as the training data for training the second neural network model, so as to increase the amount of training data and improve the accuracy of the second neural network model obtained by training.

[0166] The extended data may include multiple categories that the first neural network model can process. The amount of data in each category may vary greatly.

[0167] The extended data is input into the first neural network model and the second neural network model, and the parameters of the second neural network model are adjusted according to the loss function of the KD method so that the loss function satisfies the preset conditions. For example, the parameters of the second neural network model are adjusted so that the loss function of the KD method is minimized. The adjusted second neural network model is the result of compressing the first neural network model.

[0168] The first neural network divides the training data into N categories of data, where N is a positive integer, and the second weight of each data in the first category of data in the N categories of data in the loss function of the first neural network model and the original neural network model is negatively correlated with the number of the first category of data.

[0169] If the second weight of each data in the loss function of the KD method is equal, the following situation may occur: there is little data in a certain category corresponding to the classification result of the first neural network model, and for the data of this category, the output difference between the first neural network model and the trained second neural network model is large, but the loss function meets the preset requirements. In this case, the trained second neural network model cannot effectively process the data of this category, and the classification is inaccurate, that is, the compression result of the first neural network model is not accurate.

[0170] When compressing the first neural network model according to the extended data, the number of data of each category in the positive sample data in the labeled data and the unlabeled data can be considered. The loss function is adjusted so that the second weight of each type of extended data in the loss function is negatively correlated with the number of different types of data in the third data. By adjusting the loss function of the KD method, the compressed neural network model, i.e., the trained second neural network model, can be similar to the first neural network model for each category of data, thereby achieving a better compression effect on the first neural network model.

[0171] The extended data is input into the first neural network model to classify the extended data to obtain multiple categories of extended data and a second weight of each category of extended data in the multiple categories of extended data. The server minimizes the loss function of the KD algorithm to obtain a trained second neural network model.

[0172] The server obtains a trained second neural network model using a knowledge distillation KD method based on the third data and the first neural network model, wherein a loss function of the KD method is the sum of the products of the training errors of each type of extended data in the multiple types of extended data and the second weights of the each type of extended data.

[0173] The training error of each type of extended data can be understood as the sum of the training errors of each data in this type of extended data.

[0174] According to the number of positive sample data in the unlabeled data in the categories that the first neural network model can handle, the second weight corresponding to the data of each class in the loss function of the KD method is adjusted. When the positive sample data is unevenly distributed in different categories, the compressed neural network model can obtain better classification results for the data of each class.

[0175] Since the first neural network model may not accurately classify the data, there are errors in the classification results of the data by the first neural network model. Therefore, the second weights of different classes in the loss function can be randomly perturbed. The parameters of the second neural network model are adjusted so that the loss function meets the preset conditions under different perturbations. The adjusted second neural network model is the compressed neural network model.

[0176] The extended data are input into the first neural network model and the second neural network model, the first neural network divides the third data into N categories of extended data, N is a positive integer, and the second weights of each category of extended data in the loss function of the KD method are negatively correlated with the number of each category of extended data; the second weights corresponding to each category of extended data in the loss function of the KD method are randomly perturbed; and the parameters of the second neural network model are adjusted so that the loss function meets preset conditions under different perturbations to obtain the adjusted second neural network model.

[0177] The second weights of each type of extended data include multiple perturbed weights obtained by randomly perturbing the initial weights of the various types of extended data, and the loss function of the KD method includes multiple loss functions corresponding to the multiple perturbed weights. The initial weights of the various types of extended data are negatively correlated with the number of the various types of extended data. Minimize the maximum value of the multiple loss functions to obtain the trained second neural network model. The trained second neural network model minimizes the maximum value of the multiple loss functions. For the determination and perturbation of the second weight in the loss function, please refer to Figure 8 Description.

[0178] By randomly perturbing the initial weights in the loss function of the KD method, the perturbed weights randomly fluctuate around a value that is negatively correlated with the number of different types of data in the labeled data and the positive sample data, and adjusting the parameters of the original neural network model so that the loss function meets the preset conditions under multiple perturbed weights. The adjusted original neural network model that minimizes the maximum value of the multiple loss functions is used as the compressed second neural network model, which reduces the impact of the error in the classification result of the first neural network model on the accuracy of the compressed neural network model, and improves the accuracy of the classification result of the compressed neural network model on the data.

[0179] Through steps S301-S304, the PU classifier is used to classify the unlabeled data stored in the server according to the training data of the first neural network uploaded by the user, and the extended data that can be processed by the first neural network model can be obtained from the unlabeled data stored in the cloud, that is, data with similar properties and distribution to the training data of the first neural network model. According to the extended data, the compression of the neural network model can be achieved, thereby reducing the requirements of the neural network model compression on the number of labeled data, while ensuring the compression accuracy of the neural network model, reducing the amount of data transmitted, and improving the user experience.

[0180] Figure 5 Schematic diagram of a neural network model compression method provided in an embodiment of the present application, which is described by taking a neural network model for classifying images as an example.

[0181] The user device sends the first data and the first neural network model to the cloud service device. The cloud service device may also be referred to as a cloud server or server. Sending the first neural network model may also be understood as sending the parameters of the first neural network model. The cloud service device may use the first neural network model as a teacher network model to compress the first neural network model. The first neural network model is used to process positive sample data. The first neural network model is capable of determining the category of each image in the first data. The first data may be part of the original training data of the first neural network model. The first data may be labeled data, and the first data may be positive sample data. The first data may include at least one image in each category in the original training data of the first neural network model. The first neural network model is used to classify the input data into at least one of N categories, where N is a positive integer. The first data includes data of each of the N categories.

[0182] The cloud service device uses the first neural network model as a teacher network model according to the first data and the first neural network model sent by the user device, as well as the cloud data stored by the cloud service device, and compresses the first neural network model to obtain a compressed second neural network model. The cloud data is data stored in the cloud, including at least one image.

[0183] The cloud service device trains a PU classifier corresponding to the first neural network model according to the first data uploaded by the user device and the cloud data.

[0184] The cloud service device may compress the first neural network according to the positive sample data in the cloud data. The cloud service device uses the PU classifier to determine the positive sample data in the cloud data.

[0185] The cloud service device may use the KD method to compress the first neural network model. The cloud service device may use the first neural network model as a teacher network model and use the KD method to train the student network model. The cloud service device may use the trained student network model as a compressed second neural network model. The student network model before training may also be referred to as an original neural network model or an original model.

[0186] The cloud service device inputs the positive sample data in the cloud data into the first neural network model and the student network model, and determines the loss function L of the KD method according to the output of the first neural network model and the student network model. KD The loss function L of the KD method is KD It can be expressed as

[0187]

[0188] Where n is the number of positive sample data in the cloud data, and The outputs of the first neural network model and the student network model corresponding to the i-th data in the positive sample data in the cloud data respectively; is the training loss of the i-th data in the training data, that is and The cross entropy loss between .

[0189] The output of the first neural network model may range from negative infinity to positive infinity. The output value range of the first neural network model and the student network model can be adjusted in the same way to reduce the bit width of the output result, thereby reducing the amount of calculation for calculating the cross entropy loss. For example, the output of the first neural network model and the student network model can be standardized by performing a softmax transformation. Standardization is to adjust the output result between 0 and 1. Referring to "Distilling the Knowledge in a Neural Network" (Hinton G, Vinyals O, Dean J. Computer Science, 2015, 14 (7): 38-39.), the output of the first neural network model and the student network model can be divided by the same temperature parameter and transformed by the normalized exponential function softmax to obtain a softened probability, and the loss function of the KD method is calculated based on the two softened probability distributions. Through the temperature parameter, the distribution of the output result between 0 and 1 can be adjusted.

[0190] The cloud service device can adjust the student network model to minimize the loss function L of the loss function KD method. KD , thus obtaining the adjusted student network model. The adjusted student network model is the second neural network model.

[0191] The cloud service device can send the compressed second neural network model to the user device.

[0192] The user device processes the image according to the compressed neural network model, thereby reducing processing time, reducing resource usage, and improving processing efficiency.

[0193] The first neural network model classifies the positive sample data in the cloud data. Among the multiple classes of the classification results, the number of images in one class may be small, while the number of images in other classes is large. For the loss function of the minimized KD method, if the number of data in one class among the multiple classes of the classification results is small, the output difference between the second neural network model and the first neural network model for this class of data is large, then the second neural network model cannot effectively process this class of data, and the accuracy of the neural network compression is low. See Figure 8 , the loss function of the KD method can be adjusted to obtain a robust knowledge distillation method.

[0194] Figure 6 It is a schematic diagram of a method for expanding positive sample data provided in an embodiment of the present application.

[0195] The images belonging to the category of the original training data are positive sample data. The images not belonging to the category of the original training data are negative sample data. The positive sample data can be used to compress the first neural network model to obtain the second neural network model. The first neural network model is the teacher network model, and the second neural network model is the student network model. Compressing the first neural network model to obtain the second neural network model can be understood as training the second neural network model. Therefore, the first neural network model can also be called a pre-trained network model.

[0196] The cloud service device obtains first data. The first data includes at least one image, and each of the at least one image is positive sample data. The first data may include images of each class of all classes to which all images used to train the first neural network model belong. The first data may be part of the original training data of the first neural network model.

[0197] The cloud service device expands the positive sample data based on the first data and the first neural network model, and the second data, wherein the second data is a large amount of unlabeled data on the cloud, and the second data includes a plurality of images.

[0198] Through the PU learning algorithm, semi-supervised learning is performed based on the first data, the second data, and the proportion information of the positive sample data in the second data, and a PU classifier is generated through this learning process to label a large amount of unlabeled data. The training data of the positive-unlabeled (PU) classifier includes the first data, the second data, and the proportion information of the positive sample data in the second data. The first data is the positive sample data, and the second data is the unlabeled data. Bayesian decision theory uses the misjudgment loss to select the optimal category classification when the relevant probability is known.

[0199] The first data and the second data are input into the first neural network model. The first neural network model extracts features from each image in the first data and the second data, and the PU classifier to be trained classifies the image according to the features extracted by the first neural network model, and classifies the input image into positive sample data or negative sample data.

[0200] The parameters of the PU classifier can be adjusted according to the proportion information of the positive sample data in the second data to ensure the accuracy of the output result. The loss function of the PU learning algorithm can be determined according to the proportion information of the positive sample data in the second data. In other words, the parameters of the PU classifier can be adjusted according to the loss function of the PU learning algorithm.

[0201] The label of each image in the first data is positive sample data, and it is assumed that the label of each image in the unlabeled data is negative sample data.

[0202] xi is the training sample of PU classifier, Where X represents x i The set of training samples, R d Indicates the image representation method. i For x i The corresponding label, for example, y i ∈Y={-1,1}, where Y represents the set of labels, “+1” represents positive sample data, and “-1” represents negative sample data. i ∈T, T represents the set of training samples, T can be expressed as:

[0203]

[0204] Among them, L is the labeled data set, that is, the first data; U is the unlabeled data set, that is, the second data; n l is the number of first data, n u is the number of second data. The first data are all positive sample data, with the label “+1”. The label of the unlabeled data in the second data can be expressed as y u ,y u ∈Y={-1,1}, that is, y u is the true label of the unlabeled data.

[0205] Define the decision function f and the discriminant function F. The decision function f is used to represent the input image x i The output z of the PU classifier i The relationship between f:x i →z i According to the output z of the PU classifier i , we can determine the image x i The corresponding label y i , for example, y i With z i The corresponding relationship can be that when z i >0 o'clock i =1, when z i ≤0 when y i = -1. The discriminant function F is used to represent the input image x i With label y i The relationship between F:x i →y i .

[0206] For the traditional binary classification problem, considering the ratio of positive sample data to negative sample data in the training data, the loss function can be expressed as the expectation (mean) of the training loss of the training data. The expectation can also be called the mathematical expectation, which is the probability of each possible result in the experiment multiplied by the sum of its results. The output of the binary classifier can be adjusted according to the following loss function:

[0207]

[0208] in, It represents the loss caused by the traditional binary classifier classifying the positive sample data in the training data as negative sample data. represents the loss caused by the traditional binary classifier classifying negative sample data in the training data as positive sample data, π p Represents the prior probability of positive sample data, π n Represents the prior probability of negative sample data, π p With π n Has the following relationship:

[0209] π p +π n =1

[0210] In traditional binary classification, if the training data is randomly selected from natural data, the proportion of positive sample data in the training data can be used to represent the prior probability of positive sample data, and the proportion of negative sample data can be used to represent the prior probability of negative sample data. In other words, π p and π n can be a statistical probability. Of course, π p and π n It can also be obtained by estimating the proportion of positive samples in the second data.

[0211] In the data input to the PU classifier, we assume that the second data, i.e., the unlabeled data, is randomly selected from natural data. Therefore, we can estimate the proportion of positive sample data in the unlabeled data. This proportion can also be called the prior probability of positive sample data, denoted by π p However, since there is no labeled negative sample data in the input data of the PU classifier, it is impossible to directly obtain R pn (f) The second term in the expression.

[0212] The PU classifier may also classify the image according to the features extracted from the image. The possibility that the PU classifier classifies the positive sample data in the second data as negative sample data is the same as the possibility that the PU classifier classifies the positive sample data in the first data as negative sample data. The same may also be approximately the same.

[0213] The first data are all positive sample data. The label of the image in the first data determined by the output of the PU is compared with the label "+1" to determine the loss caused by the classification result of the first data by the PU classifier. The label "+1" is the label of the positive sample data.

[0214] therefore, It can be expressed as:

[0215]

[0216] Where l is the loss function, l(F(x),+1) is used to determine the loss caused by the label of a positive sample data obtained by the PU classifier; E p is a risk function used to determine the loss caused by the overall classification result of the first data by the PU classifier, E p It may be the sum of the losses caused by the label of each data in the first data obtained by the PU classifier.

[0217] Define R x (f) is the loss corresponding to the second data, i.e., the unlabeled data. Assume that the unlabeled data is negative sample data. Then the PU classifier classifies the images in the unlabeled data as positive sample data, and there is a loss. R x (f) represents the loss of the PU classifier when it classifies the image in the unlabeled data as a positive sample data. The label of the unlabeled data determined based on the output of the PU classifier is compared with the label “-1” to determine the loss R caused by the PU classifier in classifying the unlabeled data. x (f). The label “-1” is the label of negative sample data.

[0218] R x (f) can be expressed as:

[0219] R x (f) = E x [l(F(x),-1)]

[0220] Among them, E x is the risk function, which is used to determine the loss caused by the overall classification result of the PU classifier on the unlabeled data. x It can be the sum of the losses caused by the labels of each image obtained by the PU classifier. p With E x The subscripts p and x only indicate the source of the loss. p With E x The calculation method is the same as E p With E x The expressions can be the same.

[0221] Unlabeled data includes positive sample data and negative sample data. Therefore, Rx (f) Includes the loss caused by classifying positive sample data in unlabeled data as positive sample data, and the loss caused by classifying negative sample data in unlabeled data as positive sample data. x (f) can be expressed as

[0222]

[0223] in,

[0224]

[0225] It represents the loss caused by the PU classifier classifying the positive sample data in the unlabeled data as the positive sample data. It represents the loss caused by the PU classifier classifying negative sample data in unlabeled data as positive sample data. Negative sample data only exists in unlabeled data, so the loss function of the PU algorithm is It can be expressed as:

[0226]

[0227] Considering the actual situation, the proportion of positive sample data in the actual classification result may be different from the proportion of positive sample data in the unlabeled data π p There is an error, may be less than 0, and right Adjust to ensure that it is used to represent The value of the item is greater than 0, and we get:

[0228]

[0229] According to the loss function of the PU algorithm The parameters of the PU classifier are adjusted to change the decision function f.

[0230] In the expression, the first term can be represented by the classification loss of the decision function f on the labeled data set L, and the second term can be represented by the classification loss of the decision function f on the unlabeled data set U and the classification loss of the decision function f on the labeled data set U. The first term can represent the classification loss of the decision function f on the labeled data set L, and the second term can represent the classification loss of the decision function f on the unlabeled data set U. The sum of the two terms is the overall loss of the decision function f on the set T of training samples.

[0231] Assuming that T represents the distribution of real data, minimizing the loss of the decision function f on T means minimizing the loss of the decision function f on real data, which means that the optimal decision function f has been learned.

[0232] By adjusting the parameters of the PU classifier, Minimum, through the PU learning algorithm, a PU classifier is obtained. According to the PU classifier, the extended data in the third data, that is, the positive sample data in the unlabeled data, can be determined, and the positive sample data can be extended.

[0233] The expanded data may be included as part of the compressed training data for the first neural network model.

[0234] Figure 7 It is a schematic diagram of a multi-feature fusion model with an attention mechanism provided in an embodiment of the present application.

[0235] The PU classifier classifies the data input to the first neural network model according to the features extracted by the first neural network model. The features extracted by the first neural network model may include the output of one or more layers of the first neural network model, such as the output of the last layer, or the result of fusion or other transformation of the outputs of multiple layers.

[0236] When training the PU classifier, the first data and the second data may be input into the first neural network model. The features extracted from the multiple layers of the first neural network model are input into the multi-feature model. The multi-feature model processes the features extracted from the multiple layers of the first neural network model to obtain transformed features. The transformed features are input into the PU classifier. The PU classifier processes the fused features to obtain a label of the image.

[0237] According to the output of the PU classifier and the proportion information of the positive sample data in the second data, calculate By adjusting the parameters of the PU classifier and the multi-feature model, the Thus, the training of the PU classifier is completed.

[0238] There are many ways to integrate.

[0239] Combination is a fusion method. When combining, the features to be combined can be added directly or weighted. Weighted addition means adding after multiplying by a certain coefficient, i.e., weight value. In other words, combining can be used to linearly combine channel dimensions (channel wise). The weight value multiplied by each bit in the feature can be the same or different.

[0240] Concatenation and channel fusion are another way of fusion. With concatenate & channel fusion, the dimensions of the features to be fused can be directly concatenated, or they can be concatenated after being multiplied by a certain coefficient, i.e., weight value. The weight value multiplied by each bit in the feature can be the same or different.

[0241] The features extracted from multiple layers can be adjusted through the attention mechanism, and the adjusted features can be used as the input of the PU classifier.

[0242] According to the label of each image in the set of training samples, the parameters of the fusion model and PU classifier are adjusted to make Minimum. Adjust the parameters of the fusion model and PU classifier, that is, adjust the decision parameter f.

[0243] The outputs of the multiple layers of the first neural network may be globally averaged pooled to obtain multiple feature values ​​corresponding to the outputs of the multiple layers. The multiple feature values ​​may be concatenated to obtain a feature descriptor o.

[0244] Pooling is also called undersampling or downsampling. It is mainly used for feature dimensionality reduction, compressing the number of data and parameters, reducing overfitting, and improving the fault tolerance of the model. Global average pooling does not take the average in the form of a window, but takes the average value in the output of a layer. That is, the output of a layer is converted to a value. The calculation result of global average pooling is the average value of each point in the output matrix of a layer.

[0245] Perform attention mechanism change on the feature descriptor o to obtain the corresponding feature o'.

[0246] For example, the weight parameter w of feature descriptor o can be expressed as:

[0247] w=Attention(o,W)=σ(W2δ(W1o))

[0248] Among them, δ and σ are nonlinear transformation functions, and W1 and W2 are two fully connected parameters. W1 and W2 are two parameter matrices obtained through machine learning, and linear changes are performed through W1 and W2. δ can be, for example, a rectified linear unit (ReLU).

[0249] By combining linear changes with nonlinear transformations, the attention mechanism is used to select features. For given input data, the network outputs different features between multiple layers, and these features represent the expression of the original data at different levels. For example, when inputting a picture of a car, the features output by the bottom-level network are relatively common basic features such as edge lines and contours, while the features output by the high-level network are highly relevant to the picture, such as wheels and license plates. Through the attention mechanism, it is possible to select which layers of features are more important to the output results.

[0250] For the jth eigenvalue o in the feature descriptor j , whose weight parameter is w j , the output of the multi-feature model is related to the eigenvalue o j The corresponding field o j 'for:

[0251] o j '=w j o j

[0252] Among them, w j The value of indicates the importance of the corresponding feature. j The larger the value, the more important the corresponding feature is.

[0253] Figure 8 It is a schematic diagram of a knowledge distillation method provided in an embodiment of the present application.

[0254] This knowledge distillation method can be called a robust knowledge distillation (RKD) method.

[0255] The third data is classified by the PU classifier to obtain extended data. The extended data has the problem of unbalanced data volume. The first neural network model classifies the extended data. The extended data is the data in the unlabeled data stored by the server that has similar attributes and distribution to the training data of the first neural network model. The extended data may include part of the unlabeled data on the cloud, and may also include the training data of the first neural network model uploaded by the user device. In the extended data, the number of images of a certain class may be very small, while the number of images of other classes is relatively large. For classes with a small number of images, the output of the second neural network model should be kept the same as that of the first neural network model as much as possible, so that the second neural network model can better learn the relevant characteristics of the first neural network model, that is, the second neural network model and the first neural network model can achieve closer functions.

[0256] For example, among the 5005 training images of the expanded positive sample data, for 3 of them, the output of the trained second neural network model is different from the classification result of the first neural network model, and for the other images, the output of the trained second neural network model is the same as the classification result of the first neural network model. Among the 5005 training images, 5 belong to the first category and 5000 belong to the second category. If all the 3 images belong to the first category, then the trained second neural network model has 60% of the results for the first category images different from the classification results of the first neural network model, and the related characteristics of the trained second neural network model and the first neural network model are quite different, and the classification accuracy of such images is low, that is, the compression accuracy of the neural network model is low. If all the 3 images belong to the second category, then the trained second neural network model has 0.06% of the results for the second category images different from the classification results of the first neural network model, and the related characteristics of the trained second neural network model and the first neural network model are very different, and the compression accuracy of the neural network model is high.

[0257] In order to solve the problem of inaccurate compression of the neural network model caused by the imbalance of positive sample data in different classes, the first neural network model can be used to classify the extended data. According to the classification results, the weight of the data in the loss function of the KD method can be adjusted so that the weight corresponding to each data in the smaller category is larger, that is, the weight of the kth category in the loss function of the KD method is It is negatively correlated with the number of different types of data in the labeled data and the positive sample data.

[0258] In addition, in a category with a large amount of data, a small increase or decrease in the amount of data has little effect on the compression accuracy of the neural network model. In a category with a small amount of data, a small increase or decrease in the amount of data may also have a great impact on the compression accuracy of the neural network model. Therefore, the weight of the data in the loss function of the KD method can be adjusted so that the change in the amount of data in the category with the smaller amount of data makes the weight The greater the change in the amount of data, the greater the change in the amount of data in the category with more data. The changes are minor.

[0259] You can define weights

[0260]

[0261] Among them, K is the number of categories obtained by the first neural network to classify the extended data, y k represents the number of images of the kth class in the first data of class K. It should be understood that the above weight The expression is only for example. It can also be determined based on other functions.

[0262] The weight of each class in the K classes forms a weight vector:

[0263]

[0264] The loss function of the KD method can be expressed as:

[0265]

[0266] Among them, L c represents the loss of the i-th data, n is the amount of input data,

[0267]

[0268] w i It represents the weight corresponding to the category to which the i-th data belongs.

[0269]

[0270] Among them, k is The position of the largest element in is the category to which the image belongs. i It can be determined based on the processing result of the first neural network model on the input image.

[0271] It can be a vector output by the first neural network model, where each bit in the vector corresponds to a category that the first neural network model can process, indicating the probability that the input image belongs to the category. Or, can be a softened probability distribution, i.e. It can be obtained by normalizing the vector output by the first neural network model, that is,

[0272]

[0273] Among them, z i is the vector output by the first neural network model, each bit in the vector corresponds to a category that the first neural network model can process, and T is the temperature parameter. i or softened probability distribution It can be determined that i or The class to which the corresponding input image belongs.

[0274] Different from the traditional knowledge distillation method, in the embodiment of the present application, the parameter w is added when calculating the loss function of the KD method. i, add corresponding weights to each training data to distinguish the importance of different data. The rarer the data, the greater its weight, indicating that it is more important. Once such rare data is misclassified, a large penalty will be imposed on the classifier. Correspondingly, the more data there is, the less important it is to the classifier.

[0275] It can be said that the classification result of the first neural network model for the input image may be different from the class to which the input image actually belongs. In other words, There is noise in . Therefore, the weights determined according to the classification results of the first neural network model It may not be the best result.

[0276] Weight Or the amount of data y k Random perturbations are performed to obtain multiple perturbed weights. The difference between the amount of data in each category of the classification result of the first neural network model for the extended data and the amount of data in each category to which the extended data actually belongs is small. Therefore, the weight The actual weight situation will not be too different. Set a preset range for y k The range of the number of adjustments is smaller than the preset range. For example, for each y k , the amount or proportion of increase or decrease is less than the preset value, or the weight The amount or ratio of increase or decrease is less than the preset value.

[0277] The perturbed weight vectors form a weight vector matrix:

[0278] W={w kd_1 ,w kd_2 ,...,w kd_n}

[0279] Among them, w kd_1 ,w kd_2 ,...,w kd_n are the weight vectors corresponding to N disturbance conditions respectively.

[0280] Based on this weight matrix, the parameters of the original neural network model are adjusted to train the second neural network model. By solving the following equation, the final optimized second neural network model N can be obtained. st :

[0281]

[0282] When the weight vector is w, the second neural network model N st The loss function between the output result of the first neural network model is:

[0283] LKD (N st ,w)

[0284] For N perturbations, that is, different w in N, the maximum value of the loss function can be expressed as:

[0285]

[0286] Adjust the parameters of the second neural network model to reduce the maximum value of the loss function under different disturbance conditions as much as possible, that is, minimize the maximum value of the loss function of the KD method. st As the compressed second neural network model.

[0287] For different disturbances, the maximum value of the loss function of the KD method is less than the preset value, that is, for different disturbances, there will be no significant difference in the image processing results of the second neural network model and the first neural network model. Therefore, through the above method, the influence of the noise caused by the expansion of positive sample data on the compression of the neural network model can be reduced, and the accuracy of the compressed neural network model can be improved.

[0288] Fig. 9 It is a schematic flow chart of a method for neural network compression provided in an embodiment of the present application.

[0289] The first server obtains first data uploaded by the user device. The first data is training data of the first neural network model. The first server can train a PU classifier based on the first data and the stored second data.

[0290] The second server selects the stored third data using the PU classifier to obtain the extended data.

[0291] The second server obtains the first neural network model uploaded by the user device, and trains the second neural network model using the KD method according to the extended data. The first neural network model is used as a teacher network model of the KD method to obtain a compressed teacher network model of the KD method.

[0292] The second server can send the second neural network model to the user device.

[0293] The first data is all or part of the training data of the first neural network model. The first data includes data of each category in the multiple categories output by the first neural network model. The first server can train the PU classifier according to the PU algorithm. The principle of the PU algorithm can be found in Figure 5 Description.

[0294] The data belonging to the category output by the first neural network model can be called positive sample data. The second data includes unlabeled data. The second data includes positive sample data and data other than positive sample data. The positive sample data has similar properties and distribution to the training data of the first neural network model.

[0295] The neural network model compression process can be performed by two dedicated servers. One of the two servers can be dedicated to centralized processing of PU classifier training, and the other server can be dedicated to the selection of positive sample data and the knowledge distillation process based on the acquisition of the neural network model and the corresponding PU classifier. Centralized processing of PU classifier training and knowledge distillation can increase the speed of neural network model compression.

[0296] The neural network model compression method provided in the embodiment of the present application was verified and tested on the datasets ImageNet, Flicker1M and the modified National Institute of Standards and Technology Database (MNIST).

[0297] Table 1 shows the test results obtained by classifying pictures in the ImageNet dataset.

[0298] The ResNet-34 network is the first neural network model uploaded by users. For each category that the ResNet-34 network can handle, n samples are randomly selected from the Canadian Institute for Advanced Research (CIFAR-10) dataset. l pictures, and use the pictures of all selected classes as the first data uploaded by the user, and use the ImageNet dataset as the unlabeled dataset on the cloud, and the ResNet-18 network as the student network model.

[0299] Table 1

[0300]

[0301]

[0302] Among them, the teacher network model is the uncompressed first neural network model, and KD is the classification result obtained by training on the complete CIFAR-10 using the knowledge distillation method. The manual classification method selects positive sample data from the ImageNet data set through manual classification, and sends the data to RKD to train the second neural network model. PU-s1 selects positive sample data through the PU method, and sends all the data to RKD to train the second neural network model. PU-s2 selects positive sample data through the PU method, and randomly selects the same number of training data as the original training set CIFAR-10, and sends it to RKD to train the second neural network model. The results show that the accuracy of the method provided in the embodiment of the present application for image classification is even higher than the method of manually selecting data.

[0303] Table 2 shows the test results obtained by classifying images in the Flicker1M dataset.

[0304] The ResNet-34 network is the first neural network model uploaded by users. For each category that the ResNet-34 network can handle, n are randomly selected in the ImageNet dataset. l pictures, and use the pictures of all selected classes as the first data uploaded by the user, and use the Flicker1M dataset as the unlabeled dataset on the cloud, and the ResNet-18 network as the student network model.

[0305] Table 2

[0306] method <![CDATA[n t ]]> Data Source Top-1 acc(%) Top-5acc(%) Teacher Network Model 1,281,167 Raw data 73.27 91.26 KD-all 1,281,167 Raw data 68.67 88.76 KD-500k 500,000 Raw data 63.90 85.88 PU-s1 690,978 PU data 61.92 86.00 PU-s1 500,000 PU data 61.21 85.33

[0307] Among them, the teacher network model is the uncompressed first neural network model, KD-all is the classification result obtained by training on the complete ImageNet dataset using the knowledge distillation method, and KD-500k is the classification result obtained by training on 500,000 data randomly selected from the ImageNet dataset using the knowledge distillation method. PU-s1 and PU-s2 are the same as above. Top-1acc(%) means taking the one with the largest probability in the predicted label vector as the prediction result, and if the result is the same as the true label, it is correct. Top-5acc(%) means taking the five with the largest probability in the predicted label vector as the prediction result, and if any one of the results is the same as the true label, it is correct. The results show that the positive sample data determined by the method provided in the embodiment of the present application has a higher top5 accuracy rate than the classification result of the second neural network model obtained using the training data in the original data set.

[0308] Table 3 shows the test results obtained by classifying images in the EMNIST dataset.

[0309] The convolutional neural network model LeNet-5 network is used as the teacher network model. 1, 2, 5, 10, and 20 images are randomly selected from each type of data that the teacher network model can process in the MNIST dataset to form the first data. The EMNIST dataset is used as the unlabeled data on the cloud. The channel layers of all layers of the LeNet-5 network are reduced by half as the second neural network model.

[0310] Table 3

[0311] method 1 2 5 10 20 PU-s1 98.5 98.7 98.7 98.8 98.9 PU-s1 98.3 98.5 98.5 98.6 98.6

[0312] The results show that as the number of images in each category in the first data increases, the accuracy of the neural network model compression increases. Even when the number of images in each category in the first data is small (only one image is used), the neural network model compression can achieve good results (accuracy higher than 98%).

[0313] Fig.10 800 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. The device 800 includes an acquisition module 801 and a processing module 802.

[0314] An acquisition module 801 is used to acquire a first neural network model and training data of the first neural network uploaded by a user device;

[0315] The processing module 802 is used to obtain a PU classifier by using a positive class and unlabeled PU learning algorithm according to the training data of the first neural network and the unlabeled data stored in the server.

[0316] The processing module 802 is also used to select extended data from the unlabeled data stored in the server using the PU classifier, where the extended data is data having similar attributes and distribution to the training data of the first neural network model.

[0317] The processing module 802 is also used to train a second neural network model using the knowledge distillation KD method based on the extended data, wherein the first neural network model serves as a teacher network model of the KD method, and the second neural network model serves as a student network model of the KD method.

[0318] Optionally, the processing module 802 is also used to obtain a PU classifier using a positive class and unlabeled PU learning algorithm based on the training data of the first neural network, the unlabeled data stored in the server and the proportion information, wherein the loss function of the PU learning algorithm is the expectation of the training loss of the training data of the first neural network and the unlabeled data stored in the server, and the proportion information is used to indicate the proportion of the extended data in the unlabeled data stored in the server, and the proportion information is used to calculate the expectation.

[0319] Optionally, the PU classifier is obtained based on the first feature and the proportion information, the first feature is obtained based on the fusion of multiple third features, the multiple third features are obtained by using the first neural network model to perform feature extraction on the training data of the first neural network and the unlabeled data stored in the server, and the multiple third features correspond one-to-one to multiple layers of the first neural network.

[0320] The processing module 802 is further configured to extract features from the unlabeled data stored in the server using the first neural network model to obtain a second feature;

[0321] The processing module 802 is further configured to input the second feature into the PU classifier to determine the extended data.

[0322] Optionally, the first feature is obtained by fusing multiple third features after a first weight adjustment, the first weight adjustment is based on the proportion information, the second feature is obtained by fusing multiple fourth features using the first weight, and the multiple fourth features correspond one-to-one to multiple layers of the first neural network.

[0323] Optionally, the training data of the first neural network model is part of the training data used to train the first neural network model.

[0324] Optionally, the portion of training data includes data of each category in a plurality of categories output by the first neural network.

[0325] Optionally, the processing module 802 is further configured to input the extended data into the first neural network model to classify the extended data to obtain multiple categories of extended data and a second weight of each category of the multiple categories of extended data.

[0326] The processing module 802 is also used to minimize the loss function of the KD method to obtain a trained second neural network model, wherein the loss function of the KD method is the sum of the products of the training errors of each type of extended data in the multiple types of extended data and the second weights of each type of extended data.

[0327] Optionally, the second weights of the various types of extended data include multiple perturbed weights obtained by randomly perturbing the initial weights of the various types of extended data, and the loss function of the KD method includes multiple loss functions corresponding to the multiple perturbed weights one by one, wherein the initial weights of the various types of extended data are negatively correlated with the amount of the various types of extended data,

[0328] The processing module 802 is also used to minimize the maximum value of the multiple loss functions to obtain a trained second neural network model.

[0329] Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. Fig.10 The electronic device 1000 shown (the device 1000 may be a computer device) includes a memory 1001, a processor 1002, a communication interface 1003 and a bus 1004. The memory 1001, the processor 1002 and the communication interface 1003 are connected to each other through the bus 1004.

[0330] The memory 1001 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps of the method for compressing the neural network model of the embodiment of the present application.

[0331] Processor 1002 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the functions required to be performed by the units in the device for compressing the neural network model of the embodiment of the present application, or to execute the method for compressing the neural network model of the method embodiment of the present application.

[0332] The processor 1002 may also be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the method for compressing the neural network model of the present application may be completed by an integrated logic circuit of hardware or software instructions in the processor 1002. The above-mentioned processor 1002 may also be a general-purpose processor, a digital signal processor (digital signal processing, DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in a decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1001, and the processor 1002 reads the information in the memory 1001, and combines its hardware to complete the functions required to be performed by the units included in the device for compressing the neural network model of the embodiment of the present application, or executes the method for compressing the neural network model of the method embodiment of the present application.

[0333] The communication interface 1003 uses a transceiver such as, but not limited to, a transceiver to implement communication between the device 1000 and other devices or a communication network. For example, the communication interface 1003 can be used to obtain one or more of the first data, the second data, the proportion information of the positive sample data in the second data, the parameters of the first neural network model, and the PU classifier.

[0334] The bus 1004 may include a path for transmitting information between various components of the device 1000 (eg, the memory 1001 , the processor 1002 , and the communication interface 1003 ).

[0335] An embodiment of the present application also provides a neural network model compression device, characterized in that it includes: at least one processor and a communication interface, wherein the communication interface is used for the neural network model compression device to exchange information with other devices, and when program instructions are executed in the at least one processor, the neural network model compression device executes the above method.

[0336] An embodiment of the present application further provides a computer-readable medium, characterized in that the computer-readable medium has program instructions, and when the program instructions are directly or indirectly executed, the method in the foregoing text is implemented.

[0337] An embodiment of the present application further provides a chip, characterized in that the chip includes at least one processor, and when program instructions are executed in the at least one processor, the method described above is implemented.

[0338] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0339] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0340] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0341] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0342] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0343] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0344] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for compressing a neural network model, characterized in that: include: The server obtains a first neural network model uploaded by a user device and training image data of the first neural network model; The server obtains a PU classifier using a positive class and unlabeled PU learning algorithm according to the training image data of the first neural network model and the unlabeled image data stored in the server; The server uses the PU classifier to select extended image data from the unlabeled image data stored in the server, wherein the extended image data is image data having similar attributes and distribution to the training image data of the first neural network model; The server trains a second neural network model using a knowledge distillation KD method based on the extended image data, wherein the first neural network model serves as a teacher network model of the KD method, and the second neural network model serves as a student network model of the KD method.

2. The method according to claim 1, characterized in that The server obtains a PU classifier using a positive class and unlabeled PU learning algorithm according to the training image data of the first neural network and the unlabeled image data stored in the server, including: The server obtains the PU classifier using a positive class and unlabeled PU learning algorithm based on the training image data of the first neural network, the unlabeled image data stored in the server, and the proportion information. The loss function of the PU learning algorithm is the expectation of the training loss of the training image data of the first neural network and the unlabeled image data stored in the server. The proportion information is used to indicate the proportion of the extended image data in the unlabeled image data stored in the server, and the proportion information is used to calculate the expectation.

3. The method according to claim 2, characterized in that The PU classifier is obtained based on the first feature and the proportion information, the first feature is obtained based on the fusion of multiple third features, the multiple third features are obtained by extracting features from the training image data of the first neural network and the unlabeled image data stored by the server using the first neural network model, and the multiple third features correspond one-to-one to multiple layers of the first neural network, The server uses the PU classifier to select extended image data from the unlabeled image data stored in the server, including: The server uses the first neural network model to extract features from the unlabeled image data stored in the server to obtain a second feature; The server inputs the second feature into the PU classifier to determine the extended image data.

4. The method according to claim 3, characterized in that The first feature is obtained by fusing multiple third features after a first weight adjustment, the first weight adjustment is based on the proportion information, the second feature is obtained by fusing multiple fourth features using the first weight, and the multiple fourth features correspond one-to-one to the multiple layers of the first neural network.

5. The method according to any one of claims 1 to 4, characterized in that The training image data of the first neural network model is part of the training image data used to train the first neural network model.

6. The method according to claim 5, characterized in that The portion of training image data includes image data of each category of a plurality of categories output by the first neural network.

7. The method according to any one of claims 1 to 4, characterized in that The server trains a second neural network model using a KD method according to the extended image data, including: The server inputs the extended image data into the first neural network model to classify the extended image data to obtain multiple categories of extended image data and a second weight of each category of the multiple categories of extended image data; The server minimizes the loss function of the KD method to obtain a trained second neural network model, wherein the loss function of the KD method is the sum of the products of the training errors of each type of extended image data in the multiple types of extended image data and the second weights of the each type of extended image data.

8. The method according to claim 7, characterized in that The second weights of the various types of extended image data include a plurality of perturbed weights obtained by randomly perturbing the initial weights of the various types of extended image data, the loss function of the KD method includes a plurality of loss functions corresponding to the plurality of perturbed weights, wherein the initial weights of the various types of extended image data are negatively correlated with the quantity of the various types of extended image data, The server minimizes the loss function of the KD method to obtain a trained second neural network model, including: the server minimizes the maximum value of the multiple loss functions to obtain the trained second neural network model.

9. A device for compressing a neural network model, characterized in that: include: An acquisition module, used to acquire a first neural network model and training image data of the first neural network uploaded by a user device; Processing modules for: Obtaining a PU classifier using a positive class and unlabeled PU learning algorithm based on the training image data of the first neural network and the unlabeled image data stored in the server; Using the PU classifier, selecting extended image data from the unlabeled image data stored in the server, the extended image data being image data having similar attributes and distribution to the training image data of the first neural network model; According to the extended image data, a second neural network model is trained using the knowledge distillation KD method, wherein the first neural network model serves as a teacher network model of the KD method, and the second neural network model serves as a student network model of the KD method.

10. The device according to claim 9, characterized in that The processing module is also used to obtain the PU classifier by using a positive class and unlabeled PU learning algorithm according to the training image data of the first neural network, the unlabeled image data stored in the server and the proportion information, wherein the loss function of the PU learning algorithm is the expectation of the training loss of the training image data of the first neural network and the unlabeled image data stored in the server, and the proportion information is used to indicate the proportion of the extended image data in the unlabeled image data stored in the server, and the proportion information is used to calculate the expectation.

11. The device according to claim 10, characterized in that The PU classifier is obtained based on the first feature and the proportion information, the first feature is obtained based on the fusion of multiple third features, the multiple third features are obtained by extracting features from the training image data of the first neural network and the unlabeled image data stored by the server using the first neural network model, and the multiple third features correspond one-to-one to multiple layers of the first neural network, The processing module is also used for: Using the first neural network model to perform feature extraction on the unlabeled image data stored in the server to obtain a second feature; The second feature is input into the PU classifier to determine the extended image data.

12. The device according to claim 11, characterized in that The first feature is obtained by fusing multiple third features after a first weight adjustment, the first weight adjustment is based on the proportion information, the second feature is obtained by fusing multiple fourth features using the first weight, and the multiple fourth features correspond one-to-one to multiple layers of the first neural network.

13. The device according to any one of claims 9 to 12, characterized in that The training image data of the first neural network model is part of the training image data used to train the first neural network model.

14. The device according to claim 13, characterized in that The portion of training image data includes image data of each category of a plurality of categories output by the first neural network.

15. The device according to any one of claims 9 to 12, characterized in that The processing module is also used for: Inputting the extended image data into the first neural network model to classify the extended image data to obtain multiple categories of extended image data and a second weight of each category of the multiple categories of extended image data; Minimize the loss function of the KD method to obtain a trained second neural network model, wherein the loss function of the KD method is the sum of the products of the training errors of each type of extended image data in the multiple types of extended image data and the second weights of the each type of extended image data.

16. The device according to claim 15, characterized in that The second weights of the various types of extended image data include a plurality of perturbed weights obtained by randomly perturbing the initial weights of the various types of extended image data, the loss function of the KD method includes a plurality of loss functions corresponding to the plurality of perturbed weights, wherein the initial weights of the various types of extended image data are negatively correlated with the quantity of the various types of extended image data, The processing module is also used to minimize the maximum value of the multiple loss functions to obtain a trained second neural network model.

17. A computer readable medium, characterized in that The computer-readable medium stores a program code for execution by a device, wherein the program code includes a program code for executing the method according to any one of claims 1 to 8.

18. A chip, characterized in that: The method comprises at least one processor, and when the program instructions are executed in the at least one processor, the at least one processor is configured to execute the method according to any one of claims 1 to 8.

Images